Activity conversion detection and identification method for continuous actions

Abstract

The invention relates to the technical field of action recognition, in particular to an activity conversion detection and recognition method for continuous actions, which comprises the following steps: mapping radar signal data points to a 3D spatial domain, and performing spatial normalization; spatial feature vector extraction and dimensionality reduction are carried out through a convolution layer, and dimensionality reduction is further carried out on spatial feature vectors through two full connection layers; performing transition feature fragment generation on the spatial feature vector subjected to further dimension reduction; and inputting the transition feature fragment into a conversion and recognition network to obtain whether the transition feature fragment is subjected to activity conversion or not. The problem that existing activity conversion detection and recognition are not high in accuracy is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of motion recognition, and in particular to a method for detecting and recognizing activity transitions of continuous motions. Background Art

[0002] Human daily activity monitoring is of special significance in smart health; millimeter wave-based human activity recognition has attracted widespread attention and made remarkable progress; however, most of these solutions work on data that has been manually segmented so that each segment contains only a single activity, which is impractical when sensors continuously generate data containing a series of activities.

[0003] High-frequency (e.g., millimeter-wave) radars have been introduced in recent years for HAR sensing; however, HAR solutions based on millimeter-wave radars are limited in research objects, i.e., each input millimeter-wave data is generated by a single activity, and their task is only to classify the data into a specific activity; in practice, millimeter-wave radars continuously generate data from sensed human activities; i.e., the sensed data contains a series of activities, not a single activity; therefore, detecting the transition from one activity to another in the sensed data is necessary and must occur before identifying each activity.

[0004] The research on continuous motion recognition based on optical character recognition of FMCW radar has a good accuracy rate in the recognition of single motion, but in the recognition of continuous motion, only the recognition accuracy rate in time has achieved a certain effect;

[0005] The literature's continuous human motion recognition method based on radar sensors uses long short-term memory networks as classifiers to classify continuous human motions, but the average recognition accuracy is not high. Summary of the invention

[0006] In view of the shortcomings of the existing methods, the present invention solves the problem of low accuracy in existing activity transition detection and recognition.

[0007] The technical solution adopted by the present invention is: a method for detecting and identifying activity transitions of continuous actions comprises the following steps:

[0008] Step 1: Map the radar signal data points to the 3D spatial domain and perform spatial normalization;

[0009] As a preferred embodiment of the present invention, the radar signal data includes: walking, jumping, jumping jacks, squatting and punching.

[0010] As a preferred embodiment of the present invention, step one specifically includes:

[0011] Step 11: Consider the radar working space as a three-dimensional rectangular prism, map all data points in each frame into the three-dimensional space according to the coordinates, and then divide the space into several sub-prisms on average;

[0012] Step 12: If a data point in a certain frame falls into a certain sub-prism, its corresponding index value in the three-dimensional matrix is ​​marked as "1", otherwise it is marked as "0";

[0013] Step 13: Represent all data points of a specific frame as a frame matrix.

[0014] Step 2: Extract and reduce the spatial feature vector through the convolution layer, and then further reduce the spatial feature vector through two fully connected layers;

[0015] Step 3: Generate transition feature fragments for the spatial feature vector with further dimension reduction;

[0016] As a preferred embodiment of the present invention, step three specifically includes:

[0017] Randomly select two active segments, take the ending L1 frame of the first segment and the starting L2 frame of the second segment, and concatenate L1 and L2 to obtain the transition feature segment.

[0018] As a preferred implementation of the present invention, the length of the transition feature segment is 120 frames.

[0019] Step 4: Input the transition feature segment into the conversion and recognition network to obtain whether the transition feature segment has an activity conversion;

[0020] As a preferred embodiment of the present invention, step 4 specifically includes:

[0021] Step 41, inputting the transition feature fragment into the position embedding layer;

[0022] Step 42: Input the output features of the position embedding layer into the fully connected layer and the CNN network. The CNN network includes a first convolution layer with a convolution kernel size of 3×3, a second convolution layer with a convolution kernel size of 5×5, and a 2×1 pooling layer.

[0023] Step 43: Input the output features of the CNN network into the LSTM network, and then input them into two softmax layers to obtain two activity recognition vectors;

[0024] Step 44: perform a dot product on the two activity recognition vectors, calculate the probability of activity transition, compare the probability of activity transition with a threshold, and determine whether activity transition occurs in the transition feature segment.

[0025] As a preferred embodiment of the present invention, step 44 specifically includes:

[0026] When the probability of activity transition occurring is greater than a threshold, it is determined that activity transition occurs; otherwise, it is determined that activity transition does not occur.

[0027] As a preferred embodiment of the present invention, a continuous action activity transition detection and identification system includes: a memory for storing instructions executable by a processor; a processor for executing the instructions to implement a continuous action activity transition detection and identification method.

[0028] As a preferred embodiment of the present invention, a computer readable medium stores computer program codes, and when the computer program codes are executed by a processor, a method for detecting and identifying activity transitions of continuous actions is implemented.

[0029] Beneficial effects of the present invention:

[0030] 1. Use spatial normalization to map data points to a smaller subspace, reducing computational complexity while retaining enough information for subsequent feature extraction and activity recognition; normalized data is easier to extract features;

[0031] 2. The design of the present invention includes a position embedding layer, a full connection, a CNN network and a LSTM conversion and recognition network, which distinguishes two different activities by analyzing the relationship between two segments of activity data; realizes activity conversion detection, and detects the previous action when the conversion occurs, which has certain practical significance for ensuring personnel safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the method for detecting and identifying the activity transition of continuous actions of the present invention;

[0033] Figure 2 It is a schematic diagram of generating a transition feature segment of the present invention;

[0034] Figure 3 It is a network block diagram of activity transition detection and activity recognition. DETAILED DESCRIPTION

[0035] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0036] like Figure 1 As shown, a method for detecting and identifying activity transitions of continuous actions comprises the following steps:

[0037] Step 1: Map the radar signal data points to the 3D spatial domain and perform spatial normalization;

[0038] Specifically include:

[0039] Step 11. In order to normalize the active position, the active position refers to the relative position of the target captured by the radar in the radar coordinate system. The radar working space is regarded as a three-dimensional rectangular prism. All data points in each frame are mapped to the three-dimensional space according to the coordinates. Then, this space is evenly divided into 32*32*16 smaller sub-prisms, which are represented by a 3D matrix.

[0040] Step 12: If a data point falls into one of the sub-prisms in a certain frame, its corresponding index value in the three-dimensional matrix is ​​marked as "1", otherwise it is marked as "0"; therefore, a point in the developed transition activity data set is defined as (x, y, z, V); wherein x and y∈[0, 31], z∈[0, 15], to represent the index of the sub-prism in the three-dimensional space, V takes "0" or "1" to indicate whether a data point falls into the sub-prism in a specific frame;

[0041] Step 13, all data points of a specific frame are represented as a 32*32*16 frame matrix with binary representation, where a specific frame refers to a set of data points captured at a single time point or cycle during the radar data acquisition process; in order to capture sufficient information of a specific activity for transition detection or activity recognition, where a specific activity refers to a specific type of human activity monitored and analyzed by the radar, and sufficient information means that in order to accurately detect activity transitions or identify specific activities, sufficient data points need to be captured and analyzed, and these data points contain the characteristics of the activity, such as motion patterns, speeds, accelerations, etc.; the frame matrices of data points are organized into segments, where one segment contains 120 frame matrices, equivalent to 4 seconds, and one segment has 120*32*32*16 elements.

[0042] Spatial normalization ensures that the location information of data points remains consistent in different experiments or tests, which is convenient for analysis and comparison. By mapping data points to smaller subspaces, the complexity of calculations can be reduced while retaining sufficient information for subsequent feature extraction and activity recognition. Normalized data is easier to extract features from.

[0043] Step 2: Feature extraction;

[0044] In the dataset, each fragment is represented by 120*32*32*16 elements in the spatial domain; however, this data representation will cause the size of the entire new dataset to exceed 2TB, which will put pressure on the calculation; another bad effect is that each frame has very sparse data points distributed in the spatial domain; so a CNN network is used to extract features from the data, which can compress the data and convert it into a feature domain with a smaller dimension: (120, 512). The result of this extraction is the feature fragment.

[0045] like Figure 1First, 120 frames of point cloud data continuously captured by the radar sensor are received, and each frame of data is represented as a 32x32x16 three-dimensional spatial grid; these data are first passed through a convolutional layer to extract spatial features and reduce the data dimension from (120, 32, 32, 16, 1) to (120, 2, 4, 4, 64);

[0046] These features are then further compressed through two fully connected layers, reducing the data dimension from (120, 2048) to a more compact representation (120, 512);

[0047] The compressed feature vector can capture sufficient information for subsequent activity transition detection or activity recognition tasks.

[0048] like Figure 2 ,Step 3, transition feature fragment generation;

[0049] Step three specifically includes:

[0050] Randomly select two activity segments, take the ending L1 frame of the first segment and the starting L2 frame of the second segment, and finally concatenate these two parts into a new segment, so that it is still 120 frames long to ensure that the newly generated segment has the same size as the original segment; if the selected segments are from the same activity, it means that no activity transition occurs, and the dataset composed of these new segments is called the transition activity dataset;

[0051] It is important to determine the values ​​of L1 and L2. In order for the model to recognize what the activities in the two segments are, both activities must be of sufficient length in the transition segment, so the length of L1 is between [20, 100] and the segment length is 120 frames after adding L2.

[0052] In order to ensure data diversity, the generated data consists of half with and half without conversions. That is, one segment of the current action and one segment of the other four actions are taken to form four segments with activity conversions; four segments are randomly taken from the current action to form four segments without activity conversions. In the final data set, half of the data is with action conversions and the other half is without action conversions.

[0053] For example, if the first one is a fist-bump, a segment is extracted from each of the other four actions and combined with the current fist-bump action to form data with active conversion, and then four segments are extracted from the current fist-bump action to form data without active conversion. In this way, there are four data segments with conversion and four data segments without conversion.

[0054] like Figure 3 ,Step 4: Build an activity transition detection and activity recognition network model, referred to as the transition and recognition network;

[0055] Step 41: Input the segments in the transition activity dataset into the conversion and recognition network, and the feature segments pass through the position embedding layer (Trainable Position Embedding), add the temporal position information to the frames of the transition segments, and then pass through the fully connected layer to reduce the network size;

[0056] Step 42: The data after passing through the fully connected layer is input into the CNN network. The local pattern is learned in the input features through the convolution operation, and the feature representations at different levels are extracted through multiple convolution kernels to improve the expression ability of the model. Two convolution layers are used in CNN to process the data. The convolution kernel size of the first convolution layer is 3×3, which extracts low-level feature representations. The size of the second convolution kernel is 5×5, which extracts higher-level features. After that, the output of the convolution layer is downsampled through a 2×1 pooling layer, thereby reducing the feature dimension and enhancing the computational efficiency of the model.

[0057] Step 43, the output feature map size of CNN is 60*100, and then the LSTM network is used to identify the sequence information; the hidden layer size of LSTM is set to 10, and only the output of the last time step is returned, so that the output of the LSTM module is a vector of 10 elements; its output is then divided into two parts, and the two parts are passed to the softmax layer for activity recognition.

[0058] If the two activities are different, then an activity transition should be detected and the two parts should be considered as two different types of activities.

[0059] Step 44, perform dot product dp on the two activity recognition vectors to calculate the correlation between the specific frames of the two parts of the current segment. The higher the correlation, the greater the probability that the activities within the two parts are the same in the absence of activity conversion; therefore, the probability of activity conversion is defined as: 1-dp; then compare it with the threshold, in this embodiment, the threshold is set to 80% to determine whether activity conversion has occurred; if activity conversion has occurred, activity 2 is inferred as a new activity, and the action of activity 1 is also output.

[0060] For example, if two activity vectors are [0.7, 0.1, 0.1, 0.05, 0.05] and [0.1, 0.6, 0.15, 0.1, 0.05], their inner product result is 0.1525, which means that there is a low correlation between the two vectors. The 1-dp is 0.8475, which is greater than the set threshold of 80%. Therefore, the model will determine that an activity conversion has occurred in the data segment.

[0061] The output is an 11-bit binary vector, consisting of three parts. The first five bits are the action recognition of activity 1; the sixth bit is activity transition detection. If an activity transition occurs, the sixth bit is 1 otherwise it is 0; the last five bits are the action recognition of activity 2, that is, the type of the current action; the first five bits and the last five bits correspond to the five specific activity types of each activity, in the order of walking, jumping, jumping jacks, squatting and punching. The position corresponding to the recognized action is 1, and the remaining bits are 0; when no activity transition is detected, the action type of activity 1 is not recognized, that is, the first five bits are all 0.

[0062] Experimental process:

[0063] The dataset uses the MMActivity dataset collected by Mani Srivastava's team at the University of California, Los Angeles. The TI IWR1443BOOST radar is mounted on a 1.3-meter-high tripod. The data from the radar is sent to a laptop via USB through ROS (Robot Operating System) messages; it is developed and described in the ROS node running on the laptop; in order to record and store data, another ROS package rosbag is used; finally, these rosbags are converted to .txt files; after that, these files are used to create a voxel representation of the point cloud.

[0064] Data was collected from two users who performed five different activities in front of the radar: walking, jumping, jumping jacks, squatting, and punching. For subjects performing the same activities, data was collected in continuous periods of about 20 seconds, with some data files exceeding 20 seconds, for a total of 93 minutes of data. The details of the data set are shown in Table 1:

[0065] Table 1 Details of the dataset

[0066]

[0067] In order to verify the effectiveness of the present invention, it is compared with several neural network architectures for transition detection and activity recognition;

[0068] Table 2 Comparison of model performance

[0069]

[0070] The first three methods in Table 2 are several commonly used neural network architectures for transition detection and activity recognition. The fourth method is the method of the present invention. It can be seen that the accuracy of the present invention is significantly better than the existing methods.

[0071] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A method for detecting and identifying activity transitions of continuous actions, characterized in that: The following steps are involved: Step 1: Map the radar signal data points to the 3D spatial domain and perform spatial normalization; Step 2: Extract and reduce the spatial feature vector through the convolution layer, and then further reduce the dimension of the spatial feature vector through two fully connected layers; Step 3: Generate transition feature fragments for the spatial feature vector with further dimension reduction; Step 4: Input the transition feature fragment into the conversion and recognition network to obtain whether the transition feature fragment has an active conversion.

2. The method for detecting and identifying activity transitions of continuous actions according to claim 1, characterized in that: Step 1 specifically includes: Step 11: Consider the radar working space as a three-dimensional rectangular prism, map all data points in each frame into the three-dimensional space according to the coordinates, and then divide the three-dimensional space into several sub-prisms on average; Step 12: If a data point in a certain frame falls into a certain sub-prism, its corresponding index value in the three-dimensional matrix is ​​marked as "1", otherwise it is marked as "0"; Step 13: Represent all data points of a specific frame as a frame matrix.

3. The method for detecting and identifying activity transitions of continuous actions according to claim 1, characterized in that: Step three specifically includes: Randomly select two active segments, take the ending L1 frame of the first segment and the starting L2 frame of the second segment, and concatenate L1 and L2 to obtain the transition feature segment.

4. The method for detecting and identifying activity transitions of continuous actions according to claim 3, characterized in that: The length of the transition feature segment is 120 frames.

5. The method for detecting and identifying activity transitions of continuous actions according to claim 1, characterized in that: Step 4 specifically includes: Step 41, inputting the transition feature fragment into the position embedding layer; Step 42: Input the output features of the position embedding layer into the fully connected layer and the CNN network. The CNN network includes a first convolution layer with a convolution kernel size of 3×3, a second convolution layer with a convolution kernel size of 5×5, and a 2×1 pooling layer. Step 43: Input the output features of the CNN network into the LSTM network, and then input them into two softmax layers to obtain two activity recognition vectors; Step 44: perform a dot product on the two activity recognition vectors, calculate the probability of activity transition, compare the probability of activity transition with a threshold, and determine whether activity transition occurs in the transition feature segment.

6. The method for detecting and identifying activity transitions of continuous actions according to claim 5, characterized in that: Step 44 specifically includes: when the probability of activity transition occurring is greater than a threshold, activity transition occurs; otherwise, activity transition does not occur.

7. The method for detecting and identifying activity transitions of continuous actions according to claim 1, characterized in that: Radar signal data includes: walking, jumping, jumping jacks, squatting and punching.

8. A continuous motion activity transition detection and recognition system, characterized in that: include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the method for detecting and identifying activity transitions of continuous actions as described in any one of claims 1 to 7.

9. A computer readable medium storing a computer program code, characterized in that: When the computer program code is executed by a processor, the method for detecting and identifying activity transitions of continuous actions as described in any one of claims 1 to 7 is implemented.

Images