A multi-person real-time action recognition method based on millimeter wave radar

Abstract

The application provides a multi-person real-time action recognition method based on a millimeter wave radar, multi-person action recognition is performed by receiving reflected millimeter wave signals in a recognition environment and processing real-time collected point cloud data and Doppler data, the number of detected objects between two consecutive frames is detected to eliminate transient ghosting, and the appearance of a new object is effectively perceived, then the ghosting mirror characteristic is used to eliminate the ghosting of the newly appeared object for a plurality of consecutive frames, so that consecutive ghosting is eliminated, the Doppler data of a plurality of detected objects is segmented based on a continuous sequence of actions, local segmentation makes the data field more focused on the action, and the position change of the action with time is more intuitively reflected, so as to be beneficial to action recognition, and the real speed of the detected object is calculated according to the speed index value with the largest intensity, so that the speed dimension in the Doppler data is compressed, the number of parameters of the neural network is reduced, and the calculation complexity is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to action recognition technology, and in particular to multi-person action recognition technology based on millimeter wave radar. BACKGROUND

[0002] As an important part of ubiquitous computing and human-computer interaction, action recognition has been widely used with the development of communication technology and artificial intelligence. By monitoring people's daily activities, human-computer interaction can be achieved, or health and safety can be ensured. The current mature monitoring methods include video detection and wearable detection. However, the video detection method requires good light conditions, no occlusion, and has the risk of privacy exposure; and the wearable detection requires the monitored target to wear a specific device at all times, which brings many inconveniences.

[0003] As a kind of wireless signal, millimeter wave radar signal can realize action recognition in the case of insufficient light or occlusion, and can better protect user privacy. Millimeter wave radar is deployed in the user's surrounding environment, and the user does not need to carry any device, and can realize action recognition without contact. The current implementation scheme for action recognition based on millimeter wave radar collected point cloud data and Doppler data is to first transmit a frequency-modulated continuous wave by the millimeter wave radar, then receive the reflected signal of the target object, reflect the position information of the target object through the point cloud data, reflect the speed information of the target object through the Doppler data, and finally calculate the action information of the target object.

[0004] In the process of action detection by millimeter wave radar, due to the influence of multipath environment, the phenomenon of false target (referred to as ghost) may occur, that is, two pieces of point cloud appear for the same target object, as shown in Figure 1 , which leads to errors in the number of people detection and action recognition. In addition, due to the time continuity of action, the Doppler data collected by the millimeter wave radar is three-dimensional data containing time, distance and speed, and the data is relatively complex. When a neural network is used for action classification, a large number of neural network parameters are required, which increases the time complexity of training and detection, and is not conducive to real-time action recognition. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a method capable of eliminating ghost in the process of action detection by millimeter wave radar and suitable for real-time multi-person action recognition.

[0006] The technical solution adopted by the present application to solve the above problems is a multi-person real-time action recognition method based on millimeter wave radar. A millimeter wave radar is deployed in an action recognition environment to transmit a frequency-modulated continuous millimeter wave signal, receive reflected millimeter wave data in the recognition environment, and process real-time collected point cloud data and Doppler data to perform multi-person action recognition. The specific processing process is as follows:

[0007] Step 1, clustering the point cloud data to form a plurality of point clusters, each point cluster representing a detection object; calculating the center point of each point cluster, one point cluster corresponding to one detection object, and the center point of one point cluster as the position of the detection object;

[0008] Step 2, eliminating ghost based on point cloud data:

[0009] Eliminating ghost based on point cloud from the second frame data; defining four set variables: current detection person CP, last frame detection person LP, current capture person CC and last frame capture person LC; the detection person is a detection object that has not been subjected to point cloud ghost elimination, and the capture person is a detection object that has completed point cloud ghost elimination; CP, LP, CC and LC are represented by a sequence, each element in the sequence corresponding to an information vector of a detection object, the information vector including coordinate information and energy intensity of the detection object; only CP can be obtained in the first frame, and CC is set to CP;

[0010] 2-1) According to the number of detection objects Num in the continuous two frames of point cloud data, the elimination of transient ghost in the continuous two frames of point cloud data is completed:

[0011] 1) Num(CP) = Num(LP): using Hungarian algorithm based on CP for one-to-one matching to obtain the matching number sequence Index = Hungarian(CP, LP), then updating CC using CP, CC = CP, and then entering step 3;

[0012] 2) Num(CP) > Num(LP): using Hungarian algorithm based on LP for one-to-one matching to obtain Index = Hungarian(LP, CP), then updating CC using the part of CP corresponding to the matching of LP, and then entering step 3;

[0013] 3) Num(CP) < Num(LP): introducing the last frame capture person LC to compare LC and CP, and then dividing into the following three cases:

[0014] (a) Num(LC) = Num(CP): using Hungarian algorithm based on CP for one-to-one matching to obtain Index = Hungarian(CP, LP), then updating CC using CP, and then entering step 3;

[0015] (b) Num(LC) > Num(CP): then using Hungarian algorithm based on CP for one-to-one matching to obtain Index = Hungarian(CP, LP), then updating CC using CP, and finding the number set Index last of the detection objects that leave through the difference set between the output of the last frame Index left: Index left = Index last - Index; then stop updating Index left The corresponding detected object enters step 3;

[0016] (c) Num(LC) < Num(CP): Use the Hungarian algorithm for one-to-one matching based on CP to obtain Index = Hungarian(CP, LP), then update CC using CP, and find the set of numbers Index of newly emerged detected objects through the difference set between the output of the current frame Index and the output of the previous frame Index last of the newly emerged detected objects new : Index new = Index - Index last , then enter step 2-2);

[0017] 2-2) Eliminate persistent ghosts in point cloud data for multiple consecutive frames:

[0018] For each newly emerged detected object number new ∈ Index new , calculate the position change M of this newly emerged detected object through the position (x last , y last , z last ) of the detected object in the previous frame and the position (x, y, z) of the detected object in the current frame new :

[0019] M new = (x last - x, y last - y, z last - z)

[0020] Then compare the position change M of the newly emerged detected object new new with the position changes M of all existing detected objects i i , where i represents the number of existing detected objects, i ∈ Index - Index new ; When there is only 1 dimension with opposite numerical signs in the three dimensions of M new and M i , and the energy intensity of the newly emerged detected object new is less than the energy intensity of the existing object i, then delete the newly emerged detected object new in CC, otherwise, keep CC and enter step 3;

[0021] Step 3: Using the currently captured person CC as a reference, read the position information of all detected objects; then, using the position of each detected object as a reference, restrict the detected object to a space within the set index length, and separate the local three-dimensional Doppler data of the detected object from the global Doppler data;

[0022] Step 4: Compress the local Doppler data. The compressed local Doppler data is used as the feature of the detected object for subsequent action recognition model training and real-time multi-person action recognition.

[0023] Furthermore, this invention provides a novel three-dimensional data compression method:

[0024] In step 3, based on the position of each detected object, the detected object is restricted to a spatial range with a set index length. Local three-dimensional Doppler data of the detected object is then extracted from the global Doppler data. Specifically:

[0025] For each frame of global Doppler data S r,v S r,v The first element of the subscript represents the distance index value, the second element represents the velocity index value, and r and v represent the maximum index values ​​of distance and velocity, respectively; the local Doppler S of the detected object i. i Cut into:

[0026] S i =S r,v [0:v,R i -12:R i +13]

[0027] Where A:B represents the range from A to B; R i This indicates the location information of the detected object i. Based on this location information, the global Doppler data is segmented to obtain the local Doppler data of object i, that is, to extract a rectangle with a distance index length of 25.

[0028] In step 4, the local Doppler data is converted into local three-dimensional Doppler data before compression. in The superscript indicates the time dimension, where t is the current time.

[0029] D t,r,v Each two-dimensional matrix The format representing the i-th frame is:

[0030]

[0031] Two-dimensional matrix The first element of the subscript represents the distance index, and the second element represents the velocity index; a two-dimensional matrix. corresponds to a speed index value, each speed index value corresponds to a real speed; the two-dimensional matrix corresponds to a distance index value, each distance index value corresponds to a real distance, and each element represents the possibility of speed at the corresponding distance, the greater the value of the element, the greater the possibility of the corresponding speed; r represents the maximum value of the distance index, and v represents the maximum value of the speed index;

[0032] compress the speed dimension, calculate the real speed v1, v2, …, v r , the two-dimensional matrix is compressed into a one-dimensional vector that is

[0033]

[0034] represents the instantaneous speed at each distance index;

[0035] compress all frames in the action sequence D t,r,v , so as to compress D t,r,v into compressed local Doppler data D t,r :

[0036]

[0037] Specifically, the action recognition model is trained. The action recognition model includes a feature extractor based on Resnet and a classifier based on a fully connected neural network. The compressed two-dimensional local Doppler data D t,r and its positive and negative sub-matrices and are taken as three channels and put into the feature extractor and the classifier for training to establish an action feature extraction model and an action classification model.

[0038] Real-time multi-person action recognition. The real-time collected point cloud data and Doppler data are processed according to the foregoing method to obtain two-dimensional local Doppler three-channel data of each detection object, which is input into the trained feature extractor and classifier to complete multi-person action recognition.

[0039] The present application extracts local compressed Doppler data from radar data to identify the actions of multiple people according to the different effects of different actions of experimenters in the radar sensing area on the millimeter wave radar received signal. During signal reception, due to the influence of multipath environment, ghosting may occur in the received point cloud data. The present application eliminates temporary ghosting by detecting the number of objects between two consecutive frames and effectively senses the appearance of new objects. Then, through the mirror characteristics of ghosting, the present application eliminates the continuous ghosting of the newly appeared objects. The present application divides the Doppler data of multiple detected objects on the basis of the continuous sequence of actions. In order to reduce the number of parameters of the neural network and reduce the computational complexity, the present application calculates the real speed of the detected object according to the speed index value with the strongest intensity, thereby compressing the speed dimension in the Doppler data. Finally, the compressed local two-dimensional Doppler data is put into the pre-trained model Resnet to extract features and use a fully connected neural network to classify actions, thereby obtaining the multiple person action recognition result.

[0040] The present application has the beneficial effects that real-time action recognition of multiple detected objects can be achieved; the ghosting problem existing in the point cloud data is eliminated through the ghosting elimination method, thereby connecting the action sequences of each detected object together; in order to reduce the number of parameters of the neural network when using Doppler data, the three-dimensional Doppler data is locally cut according to the action sequence of each detected object, and the speed dimension is compressed, so that the number of neural network parameters is reduced by 75%, the training time is reduced by nearly 50%, and compared with three-dimensional convolution, two-dimensional convolution is easier to train. Local cutting makes the data field more focused on the action, and more intuitively reflects the position change of the action with time, thereby facilitating action recognition. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 Ghosting in point cloud;

[0042] Figure 2 Schematic diagram of compressing three-dimensional Doppler data to two-dimensional Doppler data;

[0043] Figure 3 Overall flow of multiple person action recognition method. DETAILED DESCRIPTION

[0044] The application deploys a millimeter wave radar in the identification environment, which continuously transmits frequency-modulated continuous wave and receives reflected signals. In order to obtain corresponding Doppler data and point cloud data, the experiment uses a TI IWR1443 boost radar board, which transmits at a frequency of 76-81GHz, has a 4GHz continuous linear frequency modulation signal, includes three transmitting antennas and four receiving antennas with built-in PLL and A2D converters. By using the matching program of TI IWR1443 boost, point cloud data and Doppler data can be obtained from the radar board. The specific implementation steps are as shown in Figure 3

[0045] Step 1: Deploy a millimeter wave radar in the perception area, so that the millimeter wave radar can cover the entire perception area, and the millimeter wave radar continuously transmits frequency-modulated continuous wave and receives reflected signals.

[0046] Step 2: Collect multi-person action data in the perception area. Select several different positions, and each person performs several actions at each position, such as pushing, pulling, and drawing letters with hands, and each action is performed several times. The collection time of each action is 4 seconds, and the number of swept frames obtained per second is 15.

[0047] The point cloud data of the collected action samples is time series data, and each frame of point cloud data P includes the basic information of each point, that is,

[0048] P=(p1,p2,…,p n )

[0049] Where n represents the number of points in the point cloud, and the maximum number of points that can be detected by the TI IWR1443 boost radar board is 64. i p i i represents the data of the i-th point in the point cloud, which is a six-dimensional vector:

[0050]

[0051] Where r represents distance, v represents velocity, s represents intensity, and (x, y, z) represents the position coordinates of the point.

[0052] The Doppler data D t,r,v of the collected action samples can be represented as

[0053]

[0054] Where t is the time index, and a frame of Doppler data is a two-dimensional matrix of r·v, that is,

[0055]

[0056] Where the two-dimensional matrix​​ The first element of the subscript represents the distance index, and the second element represents the velocity index; a two-dimensional matrix. Each column corresponds to a speed index value, and each speed index value corresponds to an actual speed; a two-dimensional matrix. Each row corresponds to a distance index value, and each distance index value corresponds to a real distance. Each element represents the probability of speed at that distance; the larger the value of the element, the greater the probability of speed. The subscript 'r' indicates the maximum distance index value, and the subscript 'v' indicates the maximum speed index value. In the radar, v is 16 and r is 128.

[0057] Step 3: Cluster each frame of point cloud data P using the DBSCAN algorithm. The distance calculation in the DBSCAN algorithm is defined as follows:

[0058]

[0059] Among them, (x a ,y a ,z a ) and (x b ,y b ,z b The coordinates of the two points are shown below. The maximum radius R of the DBSCAN algorithm is set to 150, and the minimum number of points is set to 3. After clustering the point cloud, DBSCAN forms multiple point clusters, each representing a detection object. The center point of each point cluster is calculated; one point cluster corresponds to one detection object, and the center point of a point cluster is used as the position of that detection object, thus obtaining multiple detection objects and their positions in each frame. Detection objects are represented by corresponding numbers.

[0060] Step 4: Ghosting removal based on point cloud begins from frame 2. Define four set variables: Current Person (CP), Last Person (LP), Current Capture (CC), and Last Capture (LC). The Detected Person refers to the object that has not yet undergone ghosting removal, while the Captured Person refers to the object that has undergone ghosting removal. CP, LP, CC, and LC are all represented by sequences. Each element in the sequence corresponds to an information vector list of the detected object, which includes the object's coordinates and energy intensity. Only CP is available in the first frame; CC is set to CP, and ghosting removal begins in frame 2.

[0061] Ghost elimination uses to Hungarian algorithm Hungarian. Hungarian is a kind of existing combination optimization algorithm for solving task allocation problem in polynomial time, and a maximum matching is solved on the concept of bipartite graph, that is, after receiving two groups of data as input, the matching result of the second group of data with the first group of data is output as the first group of data, for example, Hungarian(CP, LP) in Hungarian(CP, LP) input CP is CP1, CP2, CP3, LP is LP1, LP2, LP3, and the matching pair of the two groups of data is CP1-LP3, CP2-LP2, CP3-LP1, then Hungarian(CP, LP) outputs LP3, LP2, LP1.

[0062] Comprise the following two steps:

[0063] Step 4-1: Eliminate the short ghost in the continuous two frames of point cloud.

[0064] According to the number of detection objects of the continuous two frames, it is divided into the following three cases, Num represents the number:

[0065] 1) Num(CP) = Num(LP): using Hungarian algorithm Hungarian to one-to-one match the detection objects of the two frames based on CP, obtaining Index = Hungarian(CP, LP), Index is the number sequence after matching, indicating the correspondence between the number of the current frame object and the last frame object, then updating CC using CP, CC = CP, and then entering step 5;

[0066] 2) Num(CP) > Num(LP): it is indicated that new detection objects or ghosts appear, but since the ghost appears continuously for a short time, the new detection objects are not considered, and Hungarian is used to one-to-one match based on the detection object with less number LP, Index = Hungarian(LP, CP), then CC is updated using the part of CP corresponding to the matching of LP, and then entering step 5;

[0067] 3) Num(CP) < Num(LP): in this case, the last frame captured person LC needs to be introduced, and LC and CP are compared, which is divided into the following three cases:

[0068] (1) Num(LC) = Num(CP): it is considered that the new object detected in the last frame is a ghost, because the object does not appear in the current frame, which meets the characteristics of short-term appearance of ghost. Short-term appearance means that it only appears for 1 frame. Using Hungarian to one-to-one match based on CP, Index = Hungarian(CP, LP), then updating CC using CP, and entering step 5;

[0069] (2) Num(LC) > Num(CP): This situation occurs due to the departure of the detection object, i.e. the current number of detection objects is less than the number of captured objects in the last frame. One-to-one matching is performed using Hungarian with CP as the reference, Index = Hungarian(CP, LP), and then CC is updated using CP. The number set Index of the departed detection objects is found by the difference set between the output of the last frame Index last and the output of the current frame Index left , i.e.

[0070] Index left = Index last - Index

[0071] After that, the update of Index left is stopped, and the corresponding detection object enters step 5.

[0072] (3) Num(LC) < Num(CP): This situation occurs due to the appearance of a new detection object or a ghost that lasts for several frames. The ghost that lasts for several frames is a ghost that appears continuously for 2 frames or more. One-to-one matching is performed using Hungarian with CP as the reference, Index = Hungarian(CP, LP), and then CC is updated using CP. The number set Index of the newly appeared detection objects is found by the difference set between the output of the current frame Index last and the output of the last frame Index new :

[0073] Index new = Index - Index last

[0074] After that, step 4-2 is entered.

[0075] Step 4-2: Eliminate the ghost that lasts for several continuous frames in the point cloud. Define a variable M representing the position change of the detection object in the last two continuous frames to obtain the moving direction of the detection object. (x last , y last , z last ) is the position of the detection object in the last frame, and (x, y, z) is the position of the detection object in the current frame. For the new detection object number new e Index new , the position change M new of each newly appeared detection object is calculated:

[0076] M new = (x last - x, y last - y, z last - z)

[0077] The position change M new of the new detection object new is compared with the position change M i of all existing detection objects i, where i represents the number of the existing detection object, i∈Index-Index new . Due to the ghost mirror characteristic, the new object new is determined as a continuous ghost only when and only when M new and M i have one positive and one negative difference in three dimensions, i.e. M new and M i have opposite signs in one dimension and the same sign in the other two dimensions, and the energy intensity of the new object new is less than that of the existing object i. In this case, the new object new is deleted in the CC. Otherwise, the new object new is considered as a new detection object, the CC is maintained, and continuous detection is performed.

[0078] Step 5: Based on the current capture character CC, the position information of all detection objects is read. Then, based on the position of each detection object, the detection object is limited in a space range with an index length of 25, and the local three-dimensional Doppler data of the detection object is separated from the global Doppler data. In the embodiment, the detection object is limited in a certain space range. The three dimensions are time-distance-velocity respectively. Let R i represent the position information of the detection object i. Based on the position information, the global Doppler data is cut to obtain the local Doppler data of the object i. For each frame of global Doppler data S r,v , the local three-dimensional Doppler cutting of the object i is as follows:

[0079] S i = S r,v [0:v,R i -12:R i +13]

[0080] That is, a rectangle with a distance index length of 25 is intercepted as the local Doppler of the detection object i. By intercepting each frame of global Doppler in the time sequence, the Doppler time sequence of the action of the detection object i is obtained. The local three-dimensional Doppler data of all detection objects is intercepted.

[0081] Step 6: Compression of local Doppler data. After the local cutting in step 5, the local three-dimensional Doppler data corresponding to the detection object is obtained

[0082]

[0083] wherein The superscript indicates the time dimension, t is the current time, the first element of the subscript indicates the distance index value, the second element of the subscript indicates the speed index value, r and v respectively represent the maximum index value of distance and speed;

[0084] D t,r,v Each two-dimensional matrix in the middle The format of the i-th frame is represented as:

[0085]

[0086] In order to reduce the data complexity and compress the speed dimension, the real speed v1, v2, …, v of the detection object at each distance index is calculated according to the maximum intensity. r The two-dimensional matrix is compressed into a one-dimensional vector That is

[0087]

[0088] Indicates the instantaneous speed at each distance index. All frames in the action sequence D t,r,v are compressed, so that D t,r,v is compressed into D t,r , that is The time axis is added to obtain complete multi-frame action data:

[0089]

[0090] Thus, the three-dimensional Doppler data D t,r,v is compressed into two-dimensional Doppler data D t,r , as shown in Figure 2 The compression in the speed dimension proposed by the present application is different from the existing micro-Doppler compression method, which accumulates the distance dimension for compression and ignores the spatial nature of the action, only retaining the frequency information (periodicity of the action) of the action. The compression method proposed by the present application can reflect the change of the action in space over time, and provides a basis for better completing action recognition.

[0091] Step 7: Action recognition model training. The action recognition model includes a feature extractor based on a Resnet pre-training network and a classifier based on a fully connected neural network. The input data is feature extracted using Resnet, and then the features are classified using a fully connected neural network to obtain the multi-person action recognition result.

[0092] Step 7-1: Construct three-channel input data. According to the positive and negative nature of the speed, two two-dimensional sub-matrices and are separated from D t,r , where The medium speed values are all less than 0, and the values greater than 0 are filled with 0; The medium speed values are all greater than 0, and the values less than 0 are filled with 0. The three matrices are input into Resnet as three-channel data, where the time dimension t = 60 and the distance dimension r = 25.

[0093] Step 7-2: Feature extraction using Resnet pre-training network to obtain action feature f. The input data dimension is 3x60x25, and the selected Resnet model has 18 layers of neural network. The output feature f has a dimension of 1x1000.

[0094] Step 7-3: Classification of extracted feature f using a fully connected neural network classifier. The classifier is a two-layer fully connected neural network, the first layer has an input dimension of 1000 and an output dimension of 64, and the second layer has an input dimension of 64 and an output dimension of the number of detected actions. When training the model, the learning rate is set to 2x10 -4 , and the loss function is defined as cross-entropy.

[0095] Step 8: Real-time multi-person action recognition. After the action recognition model is trained, the real-time collected point cloud data and Doppler data are clustered, ghosting is eliminated, local Doppler data is segmented and compressed, and three-channel action samples of each detected object are generated. The samples are input into the Resnet feature extractor for feature extraction, and then the classifier is used for action classification to obtain the multi-person action recognition result.

Claims

1. A method for multi-person real-time action recognition based on millimeter wave radar, characterized in that, In the action recognition environment, a millimeter wave radar is deployed to emit a frequency-modulated continuous millimeter wave signal, receive the reflected millimeter wave signal in the recognition environment, and process the real-time collected point cloud data and Doppler data to perform multi-person action recognition. The specific recognition process is as follows: Step 1: Cluster the point cloud data to form multiple point clusters, each point cluster representing a detected object; calculate the center point of each point cluster, and a point cluster corresponds to a detected object, and the center point of a point cluster is taken as the position of the detected object; Step 2: Eliminate ghosting based on point cloud data: Starting from the second frame of data, eliminate ghosting based on point cloud; define four set variables: current detected person CP, last frame detected person LP, current captured person CC, and last frame captured person LC; the detected person is a detected object that has not yet undergone point cloud ghosting elimination, and the captured person is a detected object that has completed point cloud ghosting elimination; CP, LP, CC, and LC are all represented by sequences, and each element in the sequence corresponds to an information vector of a detected object, which includes coordinate information and energy intensity of the detected object; only CP can be obtained in the first frame, and CC is set to CP; 2-1) According to the number of detected objects Num in two consecutive frames, eliminate the short ghosting in the two consecutive frames of point cloud data: 1) Num(CP)=Num(LP): Use Hungarian algorithm to match one by one based on CP, obtain the matched index sequence Index=Hungarian(CP,LP), then update CC using CP, CC=CP, and then go to step 3; 2) Num(CP)>Num(LP): Use Hungarian algorithm to match one by one based on LP, obtain Index=Hungarian(LP,CP), then update CC using the part of CP that has corresponding matching with LP, and then go to step 3; 3) Num(CP)<Num(LP): Introduce the last frame captured person LC to compare LC and CP, and then divide into the following three cases: (a) Num(LC)=Num(CP): Use Hungarian algorithm to match one by one based on CP, obtain Index=Hungarian(CP,LP), then update CC using CP, and then go to step 3; (b) Num(LC) > Num(CP): then use Hungarian algorithm based on CP to find one-to-one matching, get Index = Hungarian(CP, LP), then update CC using CP, find the set of Index of the leaving detection objects by the difference set between the output of the last frame Index and the output of the current frame Index last left : Index left = Index last - Index; then stop updating Index left the corresponding detection objects, go to step 3;​ (c) Num(LC) < Num(CP): One-to-one matching is performed using the Hungarian algorithm based on CP to obtain Index = Hungarian(CP, LP), and then the CC is updated using CP, and the number set Index of newly appearing detection objects is found by the difference set between the output of the current frame Index and the output of the last frame Index last new new last , and then step 2-2) is entered.​​​ 2-2) Eliminate the persistent ghosting in the continuous multiple frames of point cloud data: For each newly appearing detection object number new e Index new , the position change M new of the new detection object is calculated from the position (x last ,y last ,z last ) of the detection object in the previous frame and the position (x,y,z) of the detection object in the current frame: M new = (x last -x, y last -y, z last -z) Then change the position M of the newly detected object new. new The positional changes M of all existing detected objects i i The comparison is performed, where i represents the existing object number, i∈Index-Index. new When M new and M i If there is only one dimension with opposite numerical signs among the three dimensions, and the energy intensity of the newly appearing detection object new is less than the energy intensity of the existing object i, then delete the newly appearing detection object new in CC; otherwise, keep CC and proceed to step 3. Step 3: Take the current captured person CC as the reference to read the position information of all detected objects; then take the position of each detected object as the reference to limit the detected object within a space range with a set index length, and separate the local three-dimensional Doppler data of the detected object from the global Doppler data; Step 4: Compress the local Doppler data, and the compressed local Doppler data is used as the feature of the detected object for subsequent action recognition model training and real-time multi-person action recognition.

2. The method of claim 1, wherein, In step 3, the position of each detected object is taken as the reference to limit the detected object within a space range with a set index length, and the local three-dimensional Doppler data of the detected object is separated from the global Doppler data, which is specifically: For each frame of global Doppler data S r,v where the first element of the subscript of S r,v represents the range index value, the second element of the subscript represents the velocity index value, and r and v represent the maximum index values for range and velocity, respectively; the local Doppler S i is cut into: S i = S r，v [0:v, R i -12: R i +13] wherein A:B denotes a range from A to B; R i represents the position information of the detection object i, according to which the global Doppler data is cut to obtain the local Doppler data of the object i, that is, a rectangle with a distance index length of 25 is intercepted.

3. The method of claim 2, wherein, The local Doppler data in step 4 is local three-dimensional Doppler data before compression wherein the superscript of t denotes the time dimension, and t is the current time D t,r,v each of the two-dimensional matrices The format representing the i-th frame is: Two-dimensional matrix The first element of the subscript represents the index value of distance, and the second element of the subscript represents the index value of speed; the column of the two-dimensional matrix corresponds to one speed index value, and each speed index value corresponds to one real speed; the row of the two-dimensional matrix corresponds to one distance index value, and each distance index value corresponds to one real distance, and each element represents the possibility of speed on the corresponding distance, and the greater the numerical value of the element, the greater the possibility of the corresponding speed; r represents the maximum value of distance index, and v represents the maximum value of speed index. The compression speed dimension, according to the maximum strength calculation, the real speed v1, v2, …, v of the detection object on each distance index r The two-dimensional matrix Compressed into a one-dimensional vector That is denotes the instantaneous velocity at each distance index; to the action sequence D t,r,v all frames are compressed, so that D t,r,v is compressed into compressed local Doppler data D t,r :

4. The method of claim 3, wherein, v is 16, and r is 128.

Images