Millimeter wave multi-person action recognition method based on multi-task learning

Abstract

The invention provides a millimeter wave multi-person action recognition method based on multi-task learning, and the method comprises the steps: collecting action data through a millimeter wave radar, and obtaining multi-person data through clustering segmentation point cloud data. The segmented data are preprocessed to generate a point cloud trajectory and a compressed Doppler map, time sequence and space feature information is obtained through feature extraction, and action feature representation is obtained after fusion. And further performing feature extraction and weighted filtering on the action features through a multi-task feature extraction structure and a gating mechanism in the multi-task learning model. The multi-task deep learning model performs shared representation and independent representation on different task data, so that three tasks of action mode discrimination, independent action recognition and interactive action recognition can be realized in the same model at the same time; therefore, single-person action recognition, multi-person independent action recognition and multi-person interactive action recognition under the condition that the number of people and the action mode are unknown in advance are achieved, the accuracy of millimeter wave multi-person action recognition can be improved, and the recognition real-time performance of the millimeter wave multi-person action recognition can be guaranteed.

Description

Technical Field

[0001] The present invention relates to a multi-person action recognition technology, and in particular to a millimeter wave multi-person action recognition technology based on multi-task deep learning. Background Art

[0002] With the development of the Internet of Things and artificial intelligence, motion recognition based on wireless signals and deep learning has become possible and has been widely used in the fields of intelligent monitoring, human-computer interaction, etc. In particular, wireless sensing technology based on millimeter-wave radar can achieve high-precision recognition due to the high frequency and high bandwidth of millimeter waves. However, most existing motion recognition technologies are aimed at single-person scenarios and are difficult to correctly identify in multi-person scenarios. The few methods for multi-person motion recognition are almost all based on single-task learning models or traditional signal processing methods. When faced with complex and changing environments, especially when multiple people perform different actions at the same time, the recognition accuracy drops significantly. Summary of the invention

[0003] The technical problem to be solved by the present invention is to propose a multi-task learning model based on a single millimeter-wave radar to simultaneously process multiple related tasks, and a method for simultaneously realizing single-person action recognition, multi-person independent action recognition and multi-person interactive action recognition based on the same model under the condition of unknown number of people and action patterns.

[0004] The technical solution adopted by the present invention to solve the above-mentioned problem is a millimeter wave multi-person action recognition method based on multi-task learning, comprising the steps of:

[0005] Data collection steps: receiving millimeter-wave radar reflection signals in the monitoring environment and converting them into point cloud data;

[0006] Point cloud denoising step: The point cloud data is first denoised using the density-based clustering algorithm DBSCAN to remove the point cloud data not included in the cluster, and the point cloud data denoised by DBSCAN is output to the trajectory segment and compressed Doppler map output step;

[0007] Point cloud clustering step: K-Means clustering is performed on the denoised point cloud data whose action mode is judged as multi-person independent action, and the point cloud data of each cluster obtained by K-Means clustering is output to the trajectory segment and compressed Doppler map output step respectively;

[0008] The output steps of trajectory segments and compressed Doppler images are as follows: the input point cloud data is matched with associated data points to form a number of trajectories, and then the length and number of the trajectories are unified to form trajectory segments; at the same time, the corresponding position information is obtained according to the input point cloud data, and the three-dimensional range Doppler data of the millimeter-wave radar reflection signal is segmented according to the position information to obtain local three-dimensional range Doppler data, and then the local three-dimensional range Doppler data is compressed in the velocity dimension to obtain a two-dimensional compressed Doppler image;

[0009] Feature extraction and fusion steps: extract features from the trajectory segments and the two-dimensional compressed Doppler images respectively and fuse them to obtain motion features;

[0010] Task feature extraction step: the task feature extraction part of the multi-task learning model that has completed training receives the input action features; the shared feature extraction layer extracts the common information shared by all tasks from the action features and outputs the shared features to the gating mechanism of each task feature extraction layer; the task feature extraction layer includes the action mode discrimination task feature extraction layer, the independent action recognition task feature extraction layer and the interactive action recognition task feature extraction layer; the task feature extraction layer extracts the action mode discrimination task features, the independent action recognition task features and the interactive action recognition task features from the action features and outputs them to the corresponding gating mechanism; the gating mechanism performs weighted filtering on the received shared features and the corresponding task features and outputs them to the corresponding task classifier in the action recognition part of the multi-task learning model;

[0011] Action recognition step: The action pattern classifier of the action recognition part of the multi-task learning model receives the weighted filtered features output by the action pattern discrimination task gating mechanism, the independent action classifier receives the weighted filtered features output by the independent action recognition task gating mechanism, and the interactive action classifier receives the weighted filtered features output by the interactive action recognition task gating mechanism; when the action pattern output by the action pattern classifier is a single-person action, the action category output by the independent action classifier is directly used as the final action recognition result; when the action pattern output by the action pattern classifier is a multi-person interactive action, the action category output by the interactive action classifier is directly used as the final action recognition result; when the action pattern output by the action pattern classifier is a multi-person independent action, the point cloud clustering step is performed again.

[0012] The present invention proposes a multi-person real-time action recognition method based on a multi-task deep learning model. The millimeter wave radar is used to collect action data, and multi-person data is obtained by clustering and segmenting the point cloud data. The segmented data is preprocessed to generate point cloud trajectories and compressed Doppler maps, and the temporal and spatial feature information is obtained by feature extraction, and the action feature representation is obtained after fusion. The action features are further extracted and weighted filtered by the multi-task feature extraction structure and gating mechanism in the multi-task learning model, and then multiple task classifiers are used to realize action recognition in different scenarios.

[0013] Compared with traditional single-task learning, multi-task learning models can handle multiple related tasks at the same time, thereby improving the accuracy, robustness and real-time performance of the model. The multi-task deep learning model proposed in the present invention can simultaneously realize the three tasks of action mode discrimination, independent action recognition and interactive action recognition in the same model by sharing and independently representing different task data, thereby realizing single-person action recognition, multi-person independent action recognition and multi-person interactive action recognition under the condition of unknown number of people and action modes in advance, which can not only improve the accuracy of millimeter-wave multi-person action recognition, but also ensure the efficiency of the recognition process.

[0014] The beneficial effects of the present invention are as follows: based on a single millimeter-wave radar, it is possible to realize the recognition of independent actions and interactive actions in multi-person scenarios. In the case of unknown number of people and action patterns, based on the same deep learning model, it is possible to automatically distinguish action patterns based on input samples, and identify single-person action categories, multi-person independent action categories, and multi-person interactive action categories, solving the industry's problem of wireless multi-person recognition. The accuracy of action recognition is ensured by designing effective data preprocessing methods and adopting advanced feature extraction models. By designing a multi-task feature extraction model structure, distinguishing between shared feature extraction and task feature extraction, and adopting a gating mechanism, it is possible to dynamically select a task feature extraction model and adjust the feature flow input to the task feature extraction model, realize information sharing and task balance, and ultimately achieve accurate multi-person action recognition in different scenarios.

[0015] In the verification experiment, the accuracy of single-person action recognition can reach 99%, the accuracy of two-person independent action recognition can reach 91%, and the accuracy of two-person interactive action recognition can reach 99%. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A system diagram for implementing the method.

[0017] Figure 2 Flowchart for multi-person action recognition. DETAILED DESCRIPTION

[0018] In order to clearly describe the present application, the following terms of the action modes are explained:

[0019] Multiple independent actions: Multiple people perform the same or different actions separately, without overlapping or touching each other, such as two people drawing circles separately, or one person waving and another drawing circles, etc. Multiple people refers to two or more people;

[0020] Multi-person interactive action: Multiple people perform an action together, and their bodies overlap or touch each other during the interaction, such as hugging, shaking hands, etc.

[0021] Single person independent action: One person performs an action, such as one person waving and another person drawing circles with their arms.

[0022] The present invention needs to be carried out in an environment covered by millimeter waves, so a millimeter wave radar is deployed in the monitoring environment, and the millimeter wave radar is equipped with multiple transceiver antennas. The specific implementation steps are as follows: Figure 2 As shown:

[0023] Step 1: The millimeter wave radar transmits a frequency modulated continuous wave (FMCW) signal and receives the reflected signal.

[0024] Step 2: Collect millimeter wave data during the target object's action as action samples, including action samples of a single person's action, multiple people's independent action, and multiple people's interactive action.

[0025] Step 3: The point cloud data of the action sample is first denoised using a density-based clustering algorithm DBSCAN (Density-Based Spatial Clustering of Applications with Noise) and then proceeds to Steps 5 and 6. The point cloud data after denoising in this step may be the point cloud data of a single person or the point cloud data of multiple people. The specific steps are as follows:

[0026] Step 3-1: Calculate the distance between neighboring points. First, enrich the multi-frame time series point cloud data, and calculate the distance between each point and its neighboring points for the enriched point cloud data;

[0027] Step 3-2: Draw a k-distance graph. Sort the distances of each point to its kth nearest neighbor and draw a graph of these distances. The k value is consistent with the minimum number of samples minPts parameter in the DBSCAN algorithm.

[0028] Step 3-3: Determine DBSCAN parameters. Analyze the k-distance graph and select the inflection point as the neighborhood radius ε parameter of the DBSCAN algorithm.

[0029] Step 3-4: DBSCAN denoising. Use the k value and the neighborhood radius ε parameter to perform DBSCAN clustering on the point cloud data. Points not included in any cluster are marked as noise points and removed. The point cloud data after DBSCAN clustering denoising is output to steps 5 and 6.

[0030] Step 4: Point cloud clustering. After the denoised point cloud data that is identified as multi-person independent actions by the action mode classifier is clustered using the K-Means algorithm, the point cloud data of each cluster is output as the point cloud data of a single person to steps 5 and 6 respectively. After K-Means clustering, each cluster represents the point cloud data of a single person.

[0031] Step 5: Extract the trajectory segment (PTS) of the action sample. Apply the Hungarian algorithm to the input point cloud data to match and associate the data points of adjacent frames to form multiple trajectories, then unify the length and number of trajectories to form trajectory segments and output them to step 7. The specific steps are as follows:

[0032] Step 5-1: Sort all input point cloud data by time frame.

[0033] Step 5-2: Feature point matching and connection. For each data point in a frame, the Hungarian algorithm is used to match and connect the corresponding points in the next frame:

[0034] T i,t =Hungarian(Point i,t ,Point i,t+1 )

[0035] Among them, T i,t represents the i-th track of the t-th frame, Hungarian represents the Hungarian algorithm, Point i,t Represents the data point of the ith trajectory of the tth frame, Point i,t+1 Represents the data point of the ith trajectory in the t+1th frame. In actual scenes, the number of data points in each frame of the point cloud is usually different, so the constructed trajectory is usually multiple trajectory segments.

[0036] Step 5-3: Unify the trajectory length. Since the number of points in each frame of the point cloud is different, the length of the constructed trajectory segments is different, so the length of the trajectory segments needs to be unified to L. If the length of the trajectory segment is not equal to L, the spline interpolation method is used to reconstruct the trajectory segment:

[0037] T i =Interpolate(T i , L)

[0038] Among them, T i represents the i-th trajectory, Interpolate represents the spline interpolation function, and L represents the reconstruction length of the spline interpolation function.

[0039] Step 5-4: Unify the number of trajectories. Multiple trajectories contain redundant information, so only representative trajectories need to be selected to reduce computational complexity. Unify the number of trajectories to n. If the actual number of trajectories exceeds n, filter them according to their importance, remove unimportant trajectories, and keep only important trajectories. If the actual number of trajectories is less than n, copy the trajectories with high importance. The method of filtering by importance is:

[0040]

[0041] Among them, Importance is the importance calculation function, which is defined as the current trajectory Trajectory i The inverse of the average cosine similarity with other trajectories is used to measure the difference between the current trajectory and other trajectories. i If the importance is less than the preset threshold, the track is removed, otherwise it is retained.

[0042] The specific calculation is as follows:

[0043] Given a trajectory T i and trajectory set {T 1 , T 2 , ..., T n}, cosine similarity cos(θ ij ) is defined as:

[0044]

[0045] Among them, T i ·T j Represents the trajectory T i and trajectory T j The dot product of |T i | and |T j |represent the trajectory T i and trajectory T j The L1 norm of the trajectory T i The average cosine similarity MeanSim with other trajectories is calculated as:

[0046]

[0047] Importace(T i ) is defined as the inverse of the mean cosine similarity:

[0048]

[0049] Step 6: Extract the two-dimensional compressed Doppler map CDM (Compressed Doppler Map) of the action sample. For the three-dimensional range Doppler data of the action sample, refer to a position coordinate corresponding to the input point cloud data, and use the coordinate position as the coordinate position of a target object. The target object here can be a single person or a combination of multiple people. Segment the local three-dimensional range Doppler data of each target object. Compress the three-dimensional range Doppler data of each target object in its velocity dimension, generate a two-dimensional compressed Doppler map CDM, and then output it to step 7. The specific steps are as follows:

[0050] Step 6-1: 3D range Doppler image segmentation. Obtain the location information R of the target object i based on the input point cloud data i , based on this, the local 3D Doppler data of the target object is cut out from the 3D range Doppler data:

[0051]

[0052]

[0053] Among them, S j，r，v represents the three-dimensional Doppler data of the jth frame, r is the distance index, v is the velocity index; R i is the location information of the target object i; Crop represents the cropping function; represents the local three-dimensional Doppler data of the target object i in the three-dimensional Doppler data of the j-th frame; is the value representing the target object i in the j-th frame of the three-dimensional Doppler data when the distance index is r and the velocity index is v in the local three-dimensional Doppler data; Represents the 3D Doppler data of target object i in frames 1 to t.

[0054] Step 6-2: Compression of local 3D range Doppler data. Compress the local 3D Doppler data in the velocity dimension and calculate its instantaneous velocity at each range index:

[0055]

[0056] Among them, v r is the instantaneous velocity at the distance index r, is the two-dimensional Doppler data of the target object i in the three-dimensional Doppler data of the j-th frame after the velocity dimension is compressed;

[0057] Compress all frames. Compress to Obtain a 2D Compressed Doppler CDM:

[0058]

[0059] in, represents the compressed 2D Doppler data of the target object i in the velocity dimension from the 1st to the tth frame;

[0060] Step 6-3: The positive and negative values ​​in are extracted separately to form and and Together, they serve as three-channel data for a convolution-based feature extractor.

[0061] Step 7: Feature extraction and fusion. The Transformer model is used to extract features from the point cloud trajectory segment PTS, and the self-supervised convolutional network model ConvNeXt is used to extract features from the three-channel data of the compressed Doppler map CDM. The two are then fused through the cross-attention mechanism as the final action features. The specific steps are as follows:

[0062] Step 7-1: For the point cloud trajectory segment PTS, the Transformer model is used in combination with the attention mechanism to extract the temporal features in the action as PTS features.

[0063] Step 7-2: For the two-dimensional compressed Doppler map CDM, select ConvNeXt as the feature extractor. Through its deep convolution operation, it effectively captures the spatial features and complex structures in CDM to obtain CDM features, such as Figure 1 shown.

[0064] Step 7-3: After feature extraction, the cross-attention mechanism is used to fuse PTS features and CDM features. By learning the complementarity between different data sources, the optimal combination of features is achieved. During fusion, the CDM feature is used as the query vector Q, and the PTS feature is used as the key K and value V. After extracting their useful information in the attention mechanism, they are spliced ​​to obtain the fused feature. The fusion formula is:

[0065]

[0066] Among them, f cdm is the CDM feature, f ts is the PTS feature, W q is the query weight matrix, W k is the key weight matrix, W v is the value weight matrix, b q is the query bias, b k is the key bias, b v is the value bias, W f is the fusion weight, b f is the fusion bias, softmax is the Softmax activation function, T Indicates transpose.

[0067] Step 8: Construct the task feature extraction part of the multi-task learning model, such as Figure 1 The model adopts a multi-task feature extraction structure and a gating mechanism to ensure the model's efficient processing of features and information balance between tasks. The multi-task feature extraction structure includes a shared feature extraction layer, an action mode discrimination task feature extraction layer, an independent action recognition task feature extraction layer, and an interactive action recognition task feature extraction layer, as follows:

[0068] Step 8-1: Multi-task feature extraction model structure. The model first processes the fused features through a shared task feature extraction layer. The shared feature extraction layer includes 4 feature extraction networks, and the structure of each feature extraction network is a multi-layer feedforward neural network. The shared task feature extraction layer focuses on capturing common information shared in all tasks to form shared features. Corresponding to each specific task of action mode discrimination, independent action recognition, and interactive action recognition, the model extracts task features through three task feature extraction layers, namely, the action mode discrimination task feature extraction layer, the independent action recognition task feature extraction layer, and the interactive action recognition task feature extraction layer. Each task feature extraction layer contains 3 feature extraction networks, and the structure of each feature extraction network is also a multi-layer feedforward neural network. The task feature extraction layer is designed for specific tasks to ensure that task features can be extracted from input features.

[0069] Step 8-2: Gating mechanism. The gating mechanism performs weighted filtering on the information output by each task feature extraction layer, and then outputs it to the corresponding task classifier to implement the corresponding task. The predecessor of the gating mechanism is the task feature extraction layer, which includes a shared feature extraction layer and three task feature extraction layers. Each task classifier obtains features from the shared task feature extraction layer and its own task feature extraction layer for classification. The shared feature extraction and task feature extraction are combined through the gating mechanism for selective fusion. The gating mechanism includes a gating network, whose structure is a single-layer feedforward neural network with an activation function of Softmax. The output of the gating network for task p can be expressed as:

[0070] g p (x) = w p (x)S p (x)

[0071] Where x represents the input feature, w p (x) is the weight function of task p, which can be calculated through linear transformation and Softmax layer:

[0072] w p (x) = softmax(W p x)

[0073] in, is a parameter matrix, m s Represents the number of shared feature extraction networks, m p represents the number of feature extraction networks for task p, and d represents the input feature S p (x) dimension. S p (x) is composed of the shared task feature vector E s (1), E s (2), ..., E s (m s ) and the task feature vector E of task p p (1), E p (2), ..., E p (m p )composition:

[0074] S p (x) = [E p (1), E p (2), ..., E p (m p ), E s (1), E s (2), ..., E s (m s )] T

[0075] In this way, the input fusion features are passed through the shared feature extraction layer and the task feature extraction layer to obtain weighted filtered features, which are then output to the task classifier, allowing it to focus on task-related features and make full use of shared features.

[0076] Step 8-3: Action recognition. After the multi-task feature extraction model and gating network in the multi-task model, the predicted output y of task p p for:

[0077] y p (x)=Classifier p (g p (x))

[0078] Among them, Classifier p Predict a function for task p.

[0079] In order to classify different tasks, the present invention designs three task classifiers, namely, action mode discriminator, independent action classifier and interactive action classifier, which are used to construct the action recognition part of the multi-task learning model. The weighted filtered features after passing through the gated network are fed into the three task classifiers, such as Figure 1 As shown in the figure, according to the action mode classification results output by the action mode discriminator, there are three cases:

[0080] (1) If the action mode output by the action mode discriminator is a single-person action, the action classification result output by the independent action classifier is directly used;

[0081] (2) If the action mode output by the action mode discriminator is a multi-person interactive action, the action classification result output by the interactive action classifier is directly used;

[0082] (3) If the action mode output by the action mode discriminator is an independent action of multiple people, the input is returned to the point cloud data of multiple single people segmented in step 4, and then the trajectory segments and compressed Doppler maps are obtained based on the point cloud data of the single person, and then feature extraction and fusion are performed. The fused features are output to the multi-task learning model again to complete multiple single-person action recognition.

[0083] Step 9: Loss function construction. During model training, the multi-task combination loss function is defined by constructing a task mask to optimize the performance of the multi-task learning model and feature extractor. The specific steps are as follows:

[0084] Step 9-1: Determine the task mask. During model training, the task mask is used to specify the valid task corresponding to each sample, which is used to mask samples that are not applicable to a specific task, so that only the loss of the relevant task is calculated. The task mask is defined as:

[0085] mask = [action mode, independent action, interactive action]

[0086] If the corresponding bit meets the requirement, it is set to 1, otherwise it is set to 0. There are three cases:

[0087] (1) For single-person action samples, the mask is marked as [1, 1, 0], indicating that only the loss of the action mode discrimination task and the independent action recognition task is calculated, and the loss of the interactive action recognition task is not calculated.

[0088] (2) For interactive action samples, the mask is marked as [1, 0, 1], which means that only the loss of the action mode discrimination task and the interactive action recognition task is calculated, and the loss of the independent action recognition task is not calculated.

[0089] (3) For multi-person independent action samples, the mask is marked as [1, 0, 0], indicating that only the loss of the action mode discrimination task is calculated. Since it needs to be further segmented and re-identified, the loss of the independent action recognition task and the interactive action recognition task is not calculated.

[0090] Step 9-2: Define the loss function. Use the combined loss function Loss, which contains the classification loss of three tasks. The classification loss of each task is calculated by cross entropy. The loss function L for each task p is p , using its mask mask p After weighting, the combined loss function is:

[0091]

[0092] where prediction p and truth p They represent the model’s predicted output and true label for task p, respectively.

[0093] Step 10: During real-time action recognition, the millimeter wave signal of the unknown number of people and action mode is input. After preprocessing, the point cloud trajectory segment PTS and the compressed Doppler map CDM of the target object are obtained. After feature extraction and fusion of the trajectory segment and the compressed Doppler map, the fused action features are respectively input into the three classifiers of the multi-task learning model. If the action mode output by the action mode classifier in the multi-task learning model is a single-person action, the action category output by the independent action classifier is directly used as the final action recognition result; if the action mode output by the action mode classifier is a multi-person interactive action, the action category output by the interactive action classifier is directly used as the final action recognition result; if the action mode output by the action mode classifier is a multi-person independent action, then return to step 4 to segment the original point cloud data to form multiple single-person point cloud data, and then extract and fuse the trajectory segments and compressed Doppler maps of these single-person point cloud data respectively, and then input them into the multi-task learning model again as single-person action data to obtain the multi-person independent action recognition result.

Claims

1. A millimeter wave multi-person action recognition method based on multi-task learning, characterized in that: The following steps are involved: Data collection steps: receiving millimeter-wave radar reflection signals in the monitoring environment and converting them into point cloud data; Point cloud denoising step: The point cloud data is first denoised using the density-based clustering algorithm DBSCAN to remove the point cloud data not included in the cluster, and the point cloud data denoised by DBSCAN is output to the trajectory segment and compressed Doppler map output step; Point cloud clustering step: K-Means clustering is performed on the denoised point cloud data whose action mode is judged as multi-person independent action, and the point cloud data of each cluster obtained by K-Means clustering is output to the trajectory segment and compressed Doppler map output step respectively; The output steps of trajectory segments and compressed Doppler images are as follows: the input point cloud data is matched with associated data points to form a number of trajectories, and then the length and number of the trajectories are unified to form trajectory segments; at the same time, the corresponding position information is obtained according to the input point cloud data, and the three-dimensional range Doppler data of the millimeter-wave radar reflection signal is segmented according to the position information to obtain local three-dimensional range Doppler data, and then the local three-dimensional range Doppler data is compressed in the velocity dimension to obtain a two-dimensional compressed Doppler image; Feature extraction and fusion steps: extract features from the trajectory segments and the two-dimensional compressed Doppler images respectively and fuse them to obtain motion features; Task feature extraction step: the task feature extraction part of the multi-task learning model that has completed training receives the input action features; the shared feature extraction layer extracts the common information shared by all tasks from the action features and outputs the shared features to the gating mechanism of each task feature extraction layer; the task feature extraction layer includes the action mode discrimination task feature extraction layer, the independent action recognition task feature extraction layer and the interactive action recognition task feature extraction layer; The task feature extraction layer extracts action mode discrimination task features, independent action recognition task features and interactive action recognition task features from the action features and outputs them to the corresponding gating mechanism; the gating mechanism performs weighted filtering on the received shared features and corresponding task features and outputs them to the corresponding task classifier in the action recognition part of the multi-task learning model; Action recognition step: The action pattern classifier of the action recognition part of the multi-task learning model receives the weighted filtered features output by the action pattern discrimination task gating mechanism, the independent action classifier receives the weighted filtered features output by the independent action recognition task gating mechanism, and the interactive action classifier receives the weighted filtered features output by the interactive action recognition task gating mechanism; when the action pattern output by the action pattern classifier is a single-person action, the action category output by the independent action classifier is directly used as the final action recognition result; when the action pattern output by the action pattern classifier is a multi-person interactive action, the action category output by the interactive action classifier is directly used as the final action recognition result; when the action pattern output by the action pattern classifier is a multi-person independent action, the point cloud clustering step is performed again.

2. The method according to claim 1, characterized in that: The millimeter wave radar reflection signal is a frequency modulated continuous wave FMCW signal.

3. The method according to claim 1, characterized in that: In the step of outputting trajectory segments and compressed Doppler images, the Hungarian algorithm is used to match the associated data points of the input point cloud data to form several initial trajectory segments, and then the length and number of the trajectory segments are unified to form trajectory segments; the selection or copying of trajectory segments is carried out according to their importance.

4. The method according to claim 1, characterized in that: In the feature extraction and fusion step, the Transformer model is used to extract features from the trajectory segments, and the ConvNeXt model is used to extract features from the two-dimensional compressed Doppler image.

5. The method according to claim 1, characterized in that: In the feature extraction and fusion steps, the cross-attention mechanism is used for feature fusion; during fusion, the two-dimensional compressed Doppler feature is used as the query vector Q, and the trajectory segment feature is used as the key K and value V.

6. The method according to claim 1, characterized in that: The shared feature extraction layer includes four feature extraction networks, and the structure of each feature extraction network is a multi-layer feedforward neural network.

7. The method according to claim 1, characterized in that: The three task feature extraction layers, namely, the action mode discrimination task feature extraction layer, the independent action recognition task feature extraction layer and the interactive action recognition task feature extraction layer, respectively contain three feature extraction networks, and the structure of each feature extraction network is a multi-layer feedforward neural network.

8. The method according to claim 1, characterized in that: Each gating mechanism uses a single-layer feed-forward neural network with Softmax as the activation function.

9. The method according to claim 1, characterized in that: The two-dimensional compressed Doppler image is three-channel data, one channel is the original two-dimensional Doppler data compressed by the velocity dimension, and the other two channels are respectively the positive part and the negative part extracted from the original two-dimensional Doppler data compressed by the velocity dimension.

10. The method according to claim 1, characterized in that: During model training, the multi-task combination loss function is defined by constructing a task mask; the task mask is defined as: mask = [action mode, independent action, interactive action], and the corresponding bit is set to 1 if it meets the requirements and set to 0 if it does not meet the requirements.

Images