Personnel behavior recognition model training method, behavior recognition method and device

Abstract

The present invention provides a human behavior recognition model training method, a behavior recognition method, and a device, comprising: obtaining point cloud data generated by a millimeter-wave radar for a sensing area, and spatial motion data of a target object obtained based on the point cloud data; inputting the point cloud data frame by frame into a point cloud feature extraction module, and inputting the target object spatial motion data into a target feature extraction module all at once, extracting the spatial features of the point cloud data and the spatial features of the target object spatial motion data, and splicing them into a combined feature sequence; adding a corresponding behavior category vector to the combined feature sequence, adding it to a set position vector, and then inputting it into an encoder module of a time series feature extraction module to calculate an output matrix with time series features; obtaining the first dimension of the output matrix, and inputting it into a linear classification layer to obtain a behavior classification result. The present invention can achieve contactless and continuous human behavior recognition, enrich the types of identifiable types, and improve recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a personnel behavior recognition model training method, a behavior recognition method and a device. Background Art

[0002] Driven by the rapid development of deep learning and mobile computing technologies, numerous applications for monitoring people's daily behavior have emerged. Human behavior recognition plays a vital role in numerous fields. For example, in healthcare, it can be used to detect falls in rehabilitation centers or the homes of elderly people living alone. In daily monitoring, it can be used in places like banks, supermarkets, and campuses to monitor abnormal events and facilitate timely remedial action.

[0003] Traditional human behavior recognition is usually achieved using image technology or wearable devices. However, image-based human behavior recognition methods are affected by weather and lighting conditions, and users also face the risk of privacy leakage. Wearable device-based human behavior recognition methods can make users feel constrained and difficult to provide a comfortable user experience, and they also require regular charging and maintenance.

[0004] The recent rise of wireless signal sensing technology enables contactless, continuous human behavior recognition. Among wireless signals, millimeter-wave signals, with their advantages in fine-grained sensing, ranging, and speed measurement, are increasingly being used for human behavior recognition tasks. However, due to the irregularity, sparseness, and susceptibility of millimeter-wave point clouds to environmental noise, these point clouds can experience quality fluctuations, such as increased noise points or excessive point sparseness, which in turn affects the accuracy of human behavior recognition. Therefore, existing human behavior recognition technologies based on millimeter-wave sensing generally suffer from a limited number of identifiable behaviors and difficulty distinguishing between similar behaviors (e.g., walking and running, sitting and falling).

[0005] Therefore, there is an urgent need for a human behavior recognition method that can identify the types of human daily behaviors and accurately identify similar behaviors. Summary of the Invention

[0006] In view of this, the embodiments of the present invention provide a personnel behavior recognition model training method, a behavior recognition method and a device to eliminate or improve one or more defects existing in the existing technology, and solve the problems existing in the existing personnel behavior recognition technology that there are few types of identifiable behaviors and it is difficult to distinguish similar behaviors (such as walking and running, sitting and falling).

[0007] In one aspect, the present invention provides a method for training a human behavior recognition model, the method comprising the following steps:

[0008] Obtaining a training sample set, the training sample set comprising a plurality of samples, each sample comprising a set number of frames of continuous point cloud data generated by the millimeter-wave radar for a sensing area, and spatial motion data of a target object obtained based on the point cloud data, the spatial motion data of the target object comprising spatial position coordinates, velocity information, and acceleration information of the target object within the sensing area; and adding a behavior of the target object as a label to each sample;

[0009] Obtaining an initial neural network model, the initial neural network model including a spatial feature extraction module and a temporal feature extraction module, the spatial feature extraction module including a point cloud feature extraction module and a target feature extraction module, the temporal feature extraction module including an encoder module and a linear classification layer, the encoder module being composed of a plurality of stacked submodules, each of which including a multi-head self-attention layer and a multi-layer perceptron;

[0010] Input the point cloud data in a single sample frame by frame into the point cloud feature extraction module to extract the spatial features of the point cloud data, input the target object spatial motion data in a single sample into the target feature extraction module at one time to extract the spatial features of the target object spatial motion data, and splice the spatial features of the point cloud data with the spatial features of the target object spatial motion data to obtain a combined feature sequence of the single sample; add the corresponding behavior category vector to the combined feature sequence, and add it to the set position vector. The set position vector is used to learn the relative position of the spatial features of each frame and input it into the encoder module to calculate an output matrix with time series characteristics, obtain the first dimension data of the output matrix, and input it into the linear classification layer to output the behavior classification result;

[0011] The initial neural network model is trained using the training sample set to obtain a personnel behavior recognition model.

[0012] In some embodiments of the present invention, the point cloud feature extraction module includes a first spatial transformation network, a first multilayer perceptron, a second spatial transformation network, a second multilayer perceptron and a third multilayer perceptron connected in sequence; wherein, the output of the first spatial transformation network is multiplied by the input and then input into the first multilayer perceptron, and the output of the second spatial transformation network is multiplied by the input and then input into the second multilayer perceptron.

[0013] In some embodiments of the present invention, the point cloud feature extraction module further includes a maximum pooling layer, which aggregates the features of the point cloud data in various dimensions using a maximum pooling operation on the output of the third multi-layer perceptron.

[0014] In some embodiments of the present invention, the target feature extraction module includes multiple multi-layer perceptrons, and ReLu and BN operations are performed after processing by each multi-layer perceptron.

[0015] In some embodiments of the present invention, the target object spatial motion data is obtained using a tracking algorithm based on an extended Kalman filter, including:

[0016] Obtaining observation values ​​of state parameters of the target object detected by the millimeter-wave radar, where the state parameters include spatial position coordinates, velocity information, and acceleration information;

[0017] Using a state transfer equation to calculate a predicted value of the state parameter of the target object at a current moment based on the observed value of the state parameter of the target object at a previous moment and determine a predicted trajectory;

[0018] Obtain one or more points in the point cloud data whose distance from the predicted trajectory is less than a set value, and calculate an average value of the state parameters of each point as the measurement value of the target object at the current moment;

[0019] The predicted value and the measured value of the state parameter are corrected by using the Kalman gain to obtain an estimated value of the state parameter of the target object at the current moment, and the estimated value is used as the spatial motion data of the target object.

[0020] In some embodiments of the present invention, the encoder module is expressed as the following formula:

[0021]

[0022]

[0023] Among them, R m-1 Represents the output of the m-1th submodule, R m Indicates the output of executing the mth submodule, represents the input value of the multilayer perceptron in the mth submodule, M is the total number of submodules, MHSA(·) represents the processing of the multi-head self-attention layer, LN(·) represents the layer normalization processing, and MLP(·) represents the processing of the multilayer perceptron.

[0024] In some embodiments of the present invention, a Softmax layer is connected after the linear classification layer to perform a normalization operation on the classification results.

[0025] In another aspect, the present invention provides a method for identifying human behavior, the method comprising the following steps:

[0026] Obtain a set number of frames of continuous point cloud data generated by the millimeter-wave radar for the perception area, and the spatial motion data of the target object obtained based on the point cloud data, and input the point cloud data and the spatial motion data of the target object into the personnel behavior recognition model in the personnel behavior recognition model training method as described in any one of the above items to obtain the recognition result of the behavior of the active personnel in the perception area.

[0027] On the other hand, the present invention provides an electronic device comprising a processor and a memory, characterized in that computer instructions are stored in the memory, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of any of the methods mentioned above.

[0028] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of any of the methods mentioned above.

[0029] The beneficial effects of the present invention are at least:

[0030] The human behavior recognition model training method, behavior recognition method, and device described in the present invention include: obtaining point cloud data generated by a millimeter-wave radar for a sensing area, and spatial motion data of a target object obtained based on the point cloud data; inputting the point cloud data frame by frame into a point cloud feature extraction module, and inputting the target object spatial motion data into a target feature extraction module all at once; extracting the spatial features of the point cloud data and the spatial features of the target object spatial motion data, and concatenating them into a combined feature sequence. The combined feature sequence is then added to the corresponding behavior category vector and added to a set position vector. This is then input into an encoder module of a temporal feature extraction module to calculate an output matrix with temporal features. The first dimension of the output matrix is ​​then obtained and input into a linear classification layer to obtain a behavior classification result. Human behavior recognition is achieved while simultaneously transmitting and receiving signals by the millimeter-wave radar, enabling contactless and continuous data collection. The target object spatial motion data is extracted from the point cloud data and input into the human behavior recognition model as data sources, ensuring the richness and stability of the data sources, improving recognition accuracy, and enriching the types of identifiable data.

[0031] Furthermore, a point cloud feature extraction module is designed within the human behavior recognition model to extract point cloud features. This module utilizes a spatial transformation network to process point cloud data, ensuring spatial invariance. This module also includes a max pooling layer, which uses the max pooling operation as a symmetric function to preserve the disorder of the point cloud.

[0032] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0033] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0035] Figure 1 1 is a flowchart of the steps of a method for training a human behavior recognition model in one embodiment of the present invention.

[0036] Figure 2 The figure is a flowchart of the steps of a method for obtaining spatial motion data of a target object based on a tracking algorithm using an extended Kalman filter in one embodiment of the present invention.

[0037] Figure 3 Schematic diagram of a method for acquiring spatial motion data of a target object using a tracking algorithm based on an extended Kalman filter in one embodiment of the present invention.

[0038] Figure 4 This is a structural diagram of a human behavior recognition model in one embodiment of the present invention.

[0039] Figure 5 This is a structural diagram of a point cloud feature extraction module in one embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0041] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0042] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0043] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0044] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0045] It should be emphasized here that the step marks mentioned below do not limit the order of the steps, but it should be understood that the steps can be executed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be executed simultaneously.

[0046] In order to solve the problem that the existing human behavior recognition technology has few identifiable behavior types and is difficult to distinguish similar behaviors (such as walking and running, sitting and falling), the present invention provides a human behavior recognition model training method, such as Figure 1 As shown, the method includes the following steps S110 to S130, and in step S120, steps S121 to S122 are further included:

[0047] Step S110: Obtain a training sample set, which contains multiple samples. Each sample includes a set number of continuous point cloud data generated by the millimeter-wave radar for the perception area, and the target object's spatial motion data obtained based on the point cloud data. The target object's spatial motion data includes the spatial position coordinates, velocity information, and acceleration information of the target object in the perception area; add the behavior of the target object as a label to each sample.

[0048] Step S120: Obtain an initial neural network model, which includes a spatial feature extraction module and a temporal feature extraction module. The spatial feature extraction module includes a point cloud feature extraction module and a target feature extraction module. The temporal feature extraction module includes an encoder module and a linear classification layer. The encoder module is composed of multiple sub-modules stacked together, and each sub-module includes a multi-head self-attention layer and a multi-layer perceptron.

[0049] Step S121: input the point cloud data in a single sample into the point cloud feature extraction module frame by frame to extract the spatial features of the point cloud data, input the target object spatial motion data in a single sample into the target feature extraction module at one time to extract the spatial features of the target object spatial motion data, and splice the spatial features of the point cloud data with the spatial features of the target object spatial motion data to obtain a combined feature sequence of the single sample.

[0050] Step S122: Add the corresponding behavior category vector to the combined feature sequence, add it to the set position vector, input it into the encoder module, calculate the output matrix with time series characteristics, obtain the first dimension data of the output matrix, input it into the linear classification layer, and output the behavior classification result.

[0051] Step S130: Using the training sample set to train the initial neural network model to obtain a personnel behavior recognition model.

[0052] In step S110, millimeter-wave radar operates in the millimeter-wave band, typically between 30 GHz and 300 GHz, with a wavelength of 1 to 10 mm, somewhere between microwaves and centimeter waves. It combines the advantages of both microwave and photoelectric radars. Due to the short wavelength of millimeter waves in the electromagnetic spectrum, millimeter-wave radars offer advantages over centimeter-wave radars, such as compact size, ease of integration, high detection accuracy, and high spatial resolution. For example, the present invention utilizes a commercial millimeter-wave radar operating in the 60-64 GHz frequency band. This device boasts advantages such as compact size, simple deployment, and low cost, capable of detecting movements as small as a fraction of a millimeter.

[0053] Millimeter-wave radar transmits a frequency-modulated continuous wave (FMCW) into its sensing area, receives radar waves reflected from target objects, and acquires point cloud data within the sensing area. For example, a commercial FMCW millimeter-wave radar is mounted at a height of 2.5 meters and tilted downward at 15 degrees, with the millimeter-wave transmitting and receiving antennas facing the sensing area. The millimeter-wave radar outputs point cloud data at a rate of approximately 20 frames per second. Each point in the point cloud frame contains at least five dimensions of information: x-axis coordinate, y-axis coordinate, z-axis coordinate, velocity, and signal-to-noise ratio. Velocity can be understood as the target object's motion relative to the radar, calculated by calculating the frequency change of the radar wave returning to the receiving antenna based on the Doppler effect. The signal-to-noise ratio is the ratio of the radar wave signal strength returning to the receiving antenna to the signal strength of internal and external noise. Point cloud data relies on the accuracy of the reflected radar wave, so a higher signal-to-noise ratio indicates stronger radar detection performance.

[0054] In some embodiments, in step S110, the spatial motion data of the target object is obtained using a tracking algorithm based on an extended Kalman filter, such as Figure 2 As shown, it includes steps S111 to S114:

[0055] Step S111: Obtain observation values ​​of state parameters of the target object detected by the millimeter wave radar, where the state parameters include spatial position coordinates, velocity information, and acceleration information.

[0056] Step S112: using the state transfer equation to calculate the predicted value of the target object state parameter at the current moment according to the observed value of the target object state parameter at the previous moment and determine the predicted trajectory.

[0057] Step S113: Obtain one or more points in the point cloud data whose distance to the predicted trajectory is less than a set value, and calculate the average value of the state parameters of each point as the measurement value of the current target object.

[0058] Step S114: using the Kalman gain to correct the predicted value and the measured value of the state parameter to obtain the estimated value of the state parameter of the target object at the current moment, and using the estimated value as the spatial motion data of the target object.

[0059] In steps S111-S114, the target object's spatial motion data is obtained by correcting the predicted and measured values ​​using the dynamically updated Kalman gain, resulting in a more accurate estimate. The main method steps can be divided into four parts: state vector prediction, point cloud association, new target assignment, and parameter update. The state vector refers to the target object's spatial position coordinates (position coordinates in the x, y, and z axes), velocity vector, and acceleration vector. These physical quantities are detected by millimeter-wave radar. The predicted value of the target object at the current moment is calculated using the state transition equation and the state parameters of the target object at the previous moment, and its predicted trajectory is determined. Since multiple people, i.e., multiple target objects, may be present in the sensing area, each target object generates its own predicted value and predicted trajectory. Points in the current point cloud are associated with the predicted trajectory according to a set distance rule. Ideally, all points generated by a target object are associated with the predicted trajectory generated by the corresponding target object. The set distance can be Euclidean distance. When the actual distance between two points in the point cloud data dimensional space is less than the set distance in this embodiment, the point is specified to be associated with the predicted trajectory. Calculate the average spatial position of the points that can be associated with the predicted trajectory and use it as the measured value of the target object at the current moment. Use the Kalman gain to correct the predicted value and the measured value to obtain the estimated value of the target object at the current moment, and use the estimated value as the spatial motion data of the target object. For example, Figure 3 As shown in the figure, for the first target object and the second target object in the millimeter wave radar sensing area, the state parameters at the previous moment are recorded as G1(n-1) and G2(n-1) respectively. The dimension of the state parameter here is 1×9, and the specific dimension information includes: coordinates, velocity and acceleration in the x-axis direction, coordinates, velocity and acceleration in the y-axis direction, and coordinates, velocity and acceleration in the z-axis direction. Based on the extended Kalman filter, the state parameters of the two targets at the current moment are predicted according to the state parameters of the two targets at the previous moment, and the predicted value G of the state parameters of the first target object and the second target object is obtained. 1,apr (n-1) and G 2,apr (n-1). G1(n-1)→G1,apr (n-1) is used as the first predicted trajectory of the first target object, and G2(n-1)→G 2,apr (n-1) is used as the second predicted trajectory of the second target object. Calculate the Euclidean distances from all points in the point cloud data to the first predicted trajectory and the second predicted trajectory. For example, the Euclidean distances from point u5 to the first predicted trajectory and the second predicted trajectory are d s1 and d s2 In the figure, the points that are less than the set distance from the first predicted trajectory include u1, u4, u5, u8 and u9. Here, we can directly calculate the distance from each point to G 1,apr The Euclidean distance of (n-1) is used to calculate the average value of the state parameters of u1, u4, u5, u8 and u9. That is the measurement value of the first target object; the points with a distance less than the set distance from the second predicted trajectory include u2, u3, u5 and u7. Here, we can directly calculate the distance from each point to G 2,apr The Euclidean distance of (n-1) is used to calculate the average value of the state parameters of u2, u3, u5 and u7. (not shown in the figure), which is the measurement value of the second target object. Taking the first target object as an example, using To G 1,apr (n-1) is corrected to obtain G1(n), which is the estimated value of the first target object and serves as the target space motion data of the first target object.

[0060] Among them, the parameters involved in the extended Kalman filter, such as the noise covariance matrix, Kalman gain, etc., are updated at each moment and then participate in the prediction calculation at the next moment.

[0061] In some embodiments, there is a new target assignment step. After the point cloud is associated with the predicted trajectory, there are some points in the point cloud data that are not on the corresponding predicted trajectory, that is, the Euclidean distance to any predicted trajectory is greater than the set distance. A clustering algorithm is used to cluster these points that are not associated with any predicted trajectory, wherein the clustering algorithm adopts the DBSCAN clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, a density-based clustering method with noise), sets the distance radius to r, and divides the areas where the density of points in the point cloud data within the distance radius of r is greater than the set value into clusters, and defines the cluster as the largest set of points that meet the distance and density requirements. The purpose of using the DBSCAN clustering algorithm is to filter low-density areas in the point cloud data, extract high-density areas, and determine new candidate objects based on the high-density areas. The spatial position average value of each point in the corresponding cluster is calculated as the spatial position parameter of the candidate object. When the candidate target object appears in multiple consecutive frames and the number of appearances is greater than the preset value, it is considered that the candidate target object does exist in the perception area. The candidate target object is assigned identification information and participates in the above method as a new target object.

[0062] After obtaining the sample set, the behavior of the target object is added as a label for each sample. In this embodiment, the behaviors of the target object include: walking, running, jumping, standing, squatting, falling and bending over. Among them, the actions of walking, running, squatting and falling are similar. The millimeter-wave radar has high detection accuracy and high spatial resolution. At the same time, the point cloud data and the target object spatial data are used as data sources for extracting the features of the target object, ensuring the richness and stability of the data collected by the millimeter-wave radar and accurately identifying similar behaviors.

[0063] In step S120, Figure 4 As shown, the internal structure of the initial neural network model is limited.

[0064] In some embodiments, the point cloud feature extraction module includes a first spatial transformation network, a first multilayer perceptron, a second spatial transformation network, a second multilayer perceptron, and a third multilayer perceptron connected in sequence; wherein, the output of the first spatial transformation network is dot-multiplied with the input and then input into the first multilayer perceptron, and the output of the second spatial transformation network is dot-multiplied with the input and then input into the second multilayer perceptron.

[0065] like Figure 5 As shown in the figure, it is a structural diagram of the point cloud feature extraction module. According to the figure, the method of extracting spatial features of point cloud data can be obtained, including:

[0066] The point cloud data is input into the point cloud extraction module. For example, each frame of point cloud data is set as a matrix with a dimension of 48×5, and the main 48 points in each frame are extracted. Each point contains the 5 dimensional information mentioned above. Therefore, in step S121, the point cloud data is input into the point cloud extraction module in a frame-by-frame manner, that is, a matrix with a dimension of 48×5 is input each time, and the number of frames is set to 20 frames.

[0067] First, the point cloud data is trained through the first spatial transformer network (Saptial Transformer Network, STN) to learn an alignment network, generating a corresponding spatial transformation parameter matrix. This matrix is ​​then multiplied by the point cloud data, i.e., matrix multiplication is performed on the x-axis, y-axis, and z-axis coordinate dimensions of the point cloud data to achieve spatial alignment. Subsequently, the first multilayer perceptron is used to map the spatial features of the point cloud data from 5 dimensions to 16 dimensions. An alignment matrix is ​​then learned through the second spatial transformer network, and the alignment matrix is ​​multiplied by the 16-dimensional point cloud data to achieve spatial feature alignment. The second and third multilayer perceptrons are then used, respectively, to map the spatial feature dimensions of the point cloud data to 32 and 64 dimensions.

[0068] In some embodiments, the point cloud feature extraction module further includes a maximum pooling layer, which aggregates features of the point cloud data in various dimensions using a maximum pooling operation on the output of the third multi-layer perceptron.

[0069] After obtaining the point cloud data that has been upgraded to 64 dimensions, the maximum pooling operation is used as a symmetric function to output the spatial features of a frame of point cloud data with a dimension of 1×64. After the spatial features of 20 consecutive frames of point cloud data are extracted, the spatial features of point cloud data with a dimension of 20×64 are obtained.

[0070] Among them, in order to ensure the invariance of the point cloud data sequence, a symmetric network based on maximum pooling of symmetric functions is adopted. Regardless of the order of the input point cloud data frames, the maximum pooling layer will obtain the same result. Specifically, the point cloud data of the previous layer, which has been calibrated by the alignment matrix mapping, is extracted through two MLP layers to extract point cloud features, and then all point cloud data in the high-dimensional feature space are aggregated through the maximum pooling layer to obtain the final global features. At the same time, since the original point cloud data has obtained high-dimensional point cloud data through multiple MLP layers, the high-dimensional space is redundant for the expression of five-dimensional point cloud data. By integrating information through symmetric operations, information loss can be reduced.

[0071] In some embodiments, ignoring the alignment operation of the spatial transformation network, the output of the point cloud feature extraction module is numerically expressed as follows:

[0072]

[0073] Where P = {p i}, i = 1, 2, ..., n, P is a point set containing n points; max(·) means taking the maximum value among multiple specified values; mlp(·) represents the processing of the multilayer perceptron.

[0074] The Transformer neural network is used in the temporal feature extraction module. Currently, commonly used neural networks include RNN (Recurrent Neural Network) and CNN (Convolutional Neural Network). Existing methods for human behavior recognition mostly use methods based on RNN or its variants (such as LSTM (Long Short-Term Memory) artificial neural networks). When extracting temporal features from continuous point cloud data sequences, traditional RNN-based networks can only predict the next moment based on the results of the previous moment, which is not parallelizable, resulting in slow extraction speed and low efficiency. Transformer neural networks, on the other hand, use a self-attention mechanism to achieve fast parallelization. In addition, Transformers can also increase network depth, unlike CNNs, which can only add models to two or three layers. Therefore, Transformer neural networks can obtain global information, thereby improving model accuracy.

[0075] The Transformer neural network model has an encoder-decoder structure. The encoder consists of multiple identical submodules, each of which has two sublayers: a multi-head self-attention layer and a multi-layer perceptron (MLP layer). The two sublayers are separated in the model: the multi-head self-attention layer only performs aggregation, and the MLP layer only performs transformation. The MLP layer is a feedforward artificial neural network model used to map a set of input vectors to a set of output vectors. The MLP can be viewed as a directed graph consisting of multiple node layers, each fully connected to the next layer. Except for the input node, each node is a neuron with a nonlinear activation function.

[0076] In some embodiments, the encoder module in the temporal feature extraction module is composed of two stacked sub-modules, each of which includes a multi-head self-attention layer and an MLP layer, where the multi-head sub-attention layer contains 8 heads, and the model dimension of each multi-head self-attention is 16 dimensions.

[0077] In some embodiments, the target feature extraction module includes multiple multi-layer perceptrons, and ReLu and BN operations are performed after each multi-layer perceptron is processed.

[0078] In step S121, since the target object spatial motion data obtained based on the point cloud data is one-dimensional data with a dimension of 1×9, 20 frames of target object spatial motion data are input into the target feature extraction module at one time. In some embodiments, after two MLP layers, the spatial features of the target object spatial motion data are mapped from 9 dimensions to 32 dimensions and 64 dimensions, and the spatial features of the target object spatial motion data with a dimension of 20×64 are output.

[0079] Since the spatial features of the point cloud data and the target object spatial motion data are extracted separately, after the spatial features are extracted, the spatial features of the point cloud data with a dimension of 20×64 are combined and spliced ​​with the spatial features of the target object spatial motion data with a dimension of 20×64 to obtain a combined feature sequence with a dimension of 20×128.

[0080] In step S122, a corresponding behavior category vector is added to the combined feature sequence, which is used to learn the corresponding personnel behavior through the extracted features in the temporal feature module. The combined feature sequence is added to the set position vector, wherein the set position vector is used to learn the relative position of the spatial features of each frame. Since the Transformer neural network is referenced in the temporal feature module, the Transformer neural network does not have the iterative operation of the RNN network, but uses a pure self-attention mechanism to capture the connection between the input data and processes the input data in parallel. Therefore, the self-attention mechanism cannot capture the order of the input combined feature sequence. The combined feature sequence is added to the set position vector. While the temporal feature module is extracting temporal features, the position vector is set to learn the relative position of the combined feature sequence of each frame, helping the temporal feature extraction module to determine the position of the combined feature sequence of each frame. The added position vector must follow the recognition rules learned by the personnel behavior recognition model.

[0081] In some embodiments, a position vector is set for the combined feature sequence, and sin and cos functions of different frequencies may be used for position encoding, as shown in formulas (2) and (3):

[0082]

[0083]

[0084] Among them, PE (positional encoding) represents position encoding; pos represents the absolute position of the spatial feature in the combined feature sequence, pos = 0, 1, 2...; d model Indicates the dimension of the combined feature sequence. Combined with the above, here d model=128; 2i and 2i+1 represent parity, i represents the dimension in the combined feature sequence, i∈[1,128]; sin(·) represents the processing of the sine function; cos(·) represents the processing of the cosine function.

[0085] The combined feature sequence of the behavior category vector and the position vector is input into the encoder module of the temporal feature module, and is calculated by two submodules. Each submodule must first pass through the self-attention layer and then the MLP layer. Therefore, the output of the encoder module is expressed as follows:

[0086]

[0087]

[0088] Among them, R m-1 Represents the output of the m-1th submodule, R m Indicates the output of executing the mth submodule, represents the input value of the multilayer perceptron in the mth submodule, M is the total number of submodules, MHSA(·) represents the processing of the multi-head self-attention layer, LN(·) represents the layer normalization processing, and MLP(·) represents the processing of the multilayer perceptron.

[0089] In some embodiments, after the combined feature sequence passes through the self-attention layer and the MLP layer, it also passes through the residual module. The data is residually connected in the residual module. As the depth of the personnel behavior recognition model increases, it may cause gradient dissipation or gradient explosion. The use of residual connection can better optimize the model.

[0090] In some embodiments, a Softmax layer is connected after the linear classification layer to perform a normalization operation on the classification results. Specifically, the Softmax layer maps each behavior category output by the linear classification layer to the interval (0, 1), which is equivalent to normalizing the behavior category into a probability distribution, and the sum of the probabilities of each behavior category is 1, and the behavior category with the largest probability value is selected as the final behavior category. For example, after processing by the Softmax layer, the probability values ​​corresponding to walking, running, jumping, standing, squatting, falling and bending over are [0.4, 0.2, 0.1, 0.1, 0.1, 0.05, 0.05] respectively. By comparing the probability values, it is finally determined that the current person's behavior is walking.

[0091] The present invention also provides a method for identifying human behavior, which comprises the following steps:

[0092] Obtain a set number of frames of continuous point cloud data generated by the millimeter-wave radar for the perception area, as well as the spatial motion data of the target object obtained based on the point cloud data, and input the point cloud data and the spatial motion data of the target object into the personnel behavior recognition model in any of the personnel behavior recognition model training methods mentioned above to obtain the recognition results of the activities of the personnel in the perception area.

[0093] Corresponding to the above method, the present invention also provides a device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0094] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0095] In summary, the present invention provides a training method for a human behavior recognition model, a behavior recognition method, and an apparatus, comprising: obtaining point cloud data generated by a millimeter-wave radar for a sensing area, and spatial motion data of a target object obtained based on the point cloud data; inputting the point cloud data frame by frame into a point cloud feature extraction module, and inputting the target object spatial motion data into a target feature extraction module all at once; extracting the spatial features of the point cloud data and the spatial features of the target object spatial motion data, and concatenating them into a combined feature sequence. Adding the corresponding behavior category vector to the combined feature sequence, adding it to a set position vector, and then inputting it into an encoder module of a temporal feature extraction module to calculate an output matrix with temporal features. The first dimension of the output matrix is ​​obtained and input into a linear classification layer to obtain a behavior classification result. Human behavior recognition is achieved while simultaneously transmitting and receiving signals by the millimeter-wave radar, enabling contactless and continuous data collection. The target object spatial motion data is extracted from the point cloud data and inputted into the human behavior recognition model as data sources, ensuring the richness and stability of the data sources, improving recognition accuracy, and enriching the types of identifiable data.

[0096] Furthermore, a point cloud feature extraction module is designed within the human behavior recognition model to extract point cloud features. This module utilizes a spatial transformation network to process point cloud data, ensuring spatial invariance. This module also includes a max pooling layer, which uses the max pooling operation as a symmetric function to preserve the disorder of the point cloud.

[0097] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0098] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0099] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0100] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for training a human behavior recognition model, characterized in that: The method comprises the following steps: Obtaining a training sample set, the training sample set comprising a plurality of samples, each sample comprising a set number of frames of continuous point cloud data generated by the millimeter-wave radar for a sensing area, and spatial motion data of a target object obtained based on the point cloud data, the spatial motion data of the target object comprising spatial position coordinates, velocity information, and acceleration information of the target object within the sensing area; and adding a behavior of the target object as a label to each sample; Obtain an initial neural network model, the initial neural network model includes a spatial feature extraction module and a temporal feature extraction module, the spatial feature extraction module includes a point cloud feature extraction module and a target feature extraction module, the point cloud feature extraction module includes a first spatial transformation network, a first multilayer perceptron, a second spatial transformation network, a second multilayer perceptron, a third multilayer perceptron and a maximum pooling layer connected in sequence; wherein, the output of the first spatial transformation network is input to the first multilayer perceptron after performing a dot product with the input, and the output of the second spatial transformation network is input to the second multilayer perceptron after performing a dot product with the input; the maximum pooling layer aggregates the features of the point cloud data in various dimensions using a maximum pooling operation on the output of the third multilayer perceptron; the temporal feature extraction module includes an encoder module and a linear classification layer, the encoder module is stacked by multiple submodules, each submodule includes a multi-head self-attention layer and a multilayer perceptron; Input the point cloud data in a single sample frame by frame into the point cloud feature extraction module to extract the spatial features of the point cloud data, input the target object spatial motion data in a single sample into the target feature extraction module at one time to extract the spatial features of the target object spatial motion data, and splice the spatial features of the point cloud data with the spatial features of the target object spatial motion data to obtain a combined feature sequence of the single sample; add the corresponding behavior category vector to the combined feature sequence, and add it to the set position vector. The set position vector is used to learn the relative position of the spatial features of each frame and input it into the encoder module to calculate an output matrix with time series characteristics, obtain the first dimension data of the output matrix, and input it into the linear classification layer to output the behavior classification result; The initial neural network model is trained using the training sample set to obtain a personnel behavior recognition model.

2. A method for training a human behavior recognition model according to claim 1, characterized in that: The target feature extraction module includes multiple multi-layer perceptrons, and ReLu and BN operations are performed after each multi-layer perceptron is processed.

3. The method for training a human behavior recognition model according to claim 1, wherein: The target object spatial motion data is obtained using a tracking algorithm based on an extended Kalman filter, including: Obtaining observation values ​​of state parameters of the target object detected by the millimeter-wave radar, where the state parameters include spatial position coordinates, velocity information, and acceleration information; Using a state transfer equation to calculate a predicted value of the state parameter of the target object at a current moment based on the observed value of the state parameter of the target object at a previous moment and determine a predicted trajectory; Obtain one or more points in the point cloud data whose distance from the predicted trajectory is less than a set value, and calculate an average value of the state parameters of each point as the measurement value of the target object at the current moment; The predicted value and the measured value of the state parameter are corrected by using the Kalman gain to obtain an estimated value of the state parameter of the target object at the current moment, and the estimated value is used as the spatial motion data of the target object.

4. The method for training a human behavior recognition model according to claim 1, wherein: The encoder module is expressed as follows: Among them, R m-1 represents the output of the m-1th submodule, R m Indicates the output of executing the mth submodule, represents the input value of the multilayer perceptron in the mth submodule, M is the total number of submodules, MHSA(·) represents the processing of the multi-head self-attention layer, LN(·) represents the layer normalization processing, and MLP(·) represents the processing of the multilayer perceptron.

5. The method for training a human behavior recognition model according to claim 1, wherein: A Softmax layer is connected after the linear classification layer to normalize the classification results.

6. A method for identifying human behavior, characterized in that: The method comprises the following steps: Obtain a set number of frames of continuous point cloud data generated by the millimeter-wave radar for the perception area, and the spatial motion data of the target object obtained based on the point cloud data, and input the point cloud data and the spatial motion data of the target object into the personnel behavior recognition model in the personnel behavior recognition model training method according to any one of claims 1 to 5 to obtain the recognition result of the behavior of the active personnel in the perception area.

7. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images