A millimeter-wave multi-person action recognition method based on multi-task learning

Abstract

The present invention provides a millimeter-wave multi-person action recognition method based on multi-task learning, which uses millimeter-wave radar to collect action data and obtains multi-person data by clustering and segmenting point cloud data. The segmented data is preprocessed to generate point cloud trajectories and compressed Doppler maps, and temporal and spatial feature information is obtained through feature extraction, which is then fused to obtain action feature representation. The action features are further extracted and weighted filtered through the multi-task feature extraction structure and gating mechanism in the multi-task learning model. The multi-task deep learning model can simultaneously realize the three tasks of action mode discrimination, independent action recognition and interactive action recognition in the same model by sharing and independently representing different task data, thereby realizing single-person action recognition, multi-person independent action recognition, and multi-person interactive action recognition when the number of people and action patterns are unknown in advance, which can not only improve the accuracy of millimeter-wave multi-person action recognition, but also ensure its real-time recognition.

Description

Technical Field

[0001] The present invention relates to a multi-person action recognition technology, and in particular to a millimeter-wave multi-person action recognition technology based on multi-task deep learning. Background Art

[0002] With the development of the Internet of Things (IoT) and artificial intelligence (AI), motion recognition based on wireless signals and deep learning has become possible and is finding widespread application in areas such as intelligent surveillance and human-computer interaction. In particular, wireless sensing technology based on millimeter-wave radar, due to the high frequency and bandwidth of millimeter-waves, enables high-precision recognition. However, existing motion recognition technologies are mostly designed for single-person scenarios and struggle to accurately identify multiple people. The few methods that do target multi-person motion recognition are almost all based on single-task learning models or traditional signal processing methods. Recognition accuracy declines significantly in complex and changing environments, especially when multiple people perform different actions simultaneously. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to propose a multi-task learning model based on a single millimeter-wave radar to simultaneously process multiple related tasks, and a method for simultaneously realizing single-person action recognition, multi-person independent action recognition and multi-person interactive action recognition based on the same model when the number of people and action patterns are unknown in advance.

[0004] The technical solution adopted by the present invention to solve the above problems is a millimeter wave multi-person action recognition method based on multi-task learning, comprising the following steps:

[0005] Data collection steps: Receive millimeter-wave radar reflection signals in the monitoring environment and convert them into point cloud data;

[0006] Point cloud denoising step: The point cloud data is first denoised using the density-based clustering algorithm DBSCAN to remove point cloud data not included in the cluster. The point cloud data obtained by DBSCAN denoising is output to the trajectory segment and compressed Doppler map output step;

[0007] Point cloud clustering step: K-Means clustering is performed on the denoised point cloud data whose motion mode is determined to be multiple independent motions. The point cloud data of each cluster obtained by K-Means clustering is output to the trajectory segment and compressed Doppler map output steps respectively;

[0008] The following steps are used to output trajectory segments and compressed Doppler maps: The input point cloud data is matched with associated data points to form several trajectories, and the length and number of the trajectories are then unified to form trajectory segments. Furthermore, the corresponding position information is obtained based on the input point cloud data. The three-dimensional range Doppler data of the millimeter-wave radar reflection signal is segmented according to the position information to obtain local three-dimensional range Doppler data. This local three-dimensional range Doppler data is then compressed in the velocity dimension to obtain a two-dimensional compressed Doppler map.

[0009] Feature extraction and fusion steps: extract features from trajectory segments and two-dimensional compressed Doppler images respectively and fuse them to obtain motion features;

[0010] Task feature extraction step: The task feature extraction part of the trained multi-task learning model receives the input action features; the shared feature extraction layer extracts the common information shared by all tasks from the action features and outputs the shared features to the gating mechanism of each task feature extraction layer; the task feature extraction layer includes the action mode discrimination task feature extraction layer, the independent action recognition task feature extraction layer and the interactive action recognition task feature extraction layer; the task feature extraction layer extracts the action mode discrimination task features, the independent action recognition task features and the interactive action recognition task features from the action features and outputs them to the corresponding gating mechanism; the gating mechanism performs weighted filtering on the received shared features and corresponding task features and outputs them to the corresponding task classifier in the action recognition part of the multi-task learning model;

[0011] Action recognition step: The action pattern classifier of the action recognition part of the multi-task learning model receives the weighted filtered features output by the action pattern discrimination task gating mechanism, the independent action classifier receives the weighted filtered features output by the independent action recognition task gating mechanism, and the interactive action classifier receives the weighted filtered features output by the interactive action recognition task gating mechanism; when the action pattern output by the action pattern classifier is a single-person action, the action category output by the independent action classifier is directly used as the final action recognition result; when the action pattern output by the action pattern classifier is a multi-person interactive action, the action category output by the interactive action classifier is directly used as the final action recognition result; when the action pattern output by the action pattern classifier is a multi-person independent action, the point cloud clustering step is repeated.

[0012] This paper proposes a multi-person real-time action recognition method based on a multi-task deep learning model. Motion data is collected using millimeter-wave radar, and multi-person data is obtained through clustering and segmentation of point cloud data. The segmented data undergoes preprocessing to generate point cloud trajectories and compressed Doppler maps. Feature extraction is used to obtain temporal and spatial feature information, which is then fused to form action feature representations. The action features are further extracted and weighted filtered using the multi-task feature extraction architecture and gating mechanism within the multi-task learning model. Multiple task classifiers are then used to implement action recognition in different scenarios.

[0013] Compared to traditional single-task learning, multi-task learning models can handle multiple related tasks simultaneously, thereby improving the model's accuracy, robustness, and real-time performance. The multi-task deep learning model proposed in this paper can simultaneously implement three tasks in the same model: action pattern discrimination, independent action recognition, and interactive action recognition, by sharing and independently representing different task data. This allows for single-person action recognition, multi-person independent action recognition, and multi-person interactive action recognition, even when the number of people and action patterns are unknown in advance. This improves the accuracy of millimeter-wave multi-person action recognition while ensuring efficiency during the recognition process.

[0014] The beneficial effects of the present invention are as follows: based on a single millimeter-wave radar, it is possible to realize the recognition of independent actions and interactive actions in multi-person scenarios. In the case of unknown number of people and action patterns, based on the same deep learning model, it is possible to automatically perform action pattern discrimination based on input samples, and identify single-person action categories, multi-person independent action categories, and multi-person interactive action categories, solving the industry's difficult problem of wireless multi-person recognition. The accuracy of action recognition is ensured by designing an effective data preprocessing method and adopting an advanced feature extraction model. By designing a multi-task feature extraction model structure, distinguishing between shared feature extraction and task feature extraction, and adopting a gating mechanism, it is possible to dynamically select a task feature extraction model and adjust the feature flow input to the task feature extraction model, realize information sharing and task balance, and ultimately achieve accurate multi-person action recognition in different scenarios.

[0015] In the verification experiment, the accuracy of single-person action recognition can reach 99%, the accuracy of two-person independent action recognition can reach 91%, and the accuracy of two-person interactive action recognition can reach 99%. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A system diagram for implementing the method.

[0017] Figure 2 Flowchart for multi-person action recognition. DETAILED DESCRIPTION

[0018] To clearly describe the present application, the following terms for the action modes are explained:

[0019] Multiple independent actions: Multiple people perform the same or different actions separately, without overlapping or touching each other, such as two people drawing circles, or one person waving while another draws circles. Multiple people refers to two or more people.

[0020] Multi-person interaction: When multiple people perform an action together, their bodies overlap or touch during the interaction, such as hugging or handshaking.

[0021] Single-person independent action: One person performs an action, such as one person waving his hands or one person drawing circles with his arms.

[0022] The present invention needs to be carried out in an environment covered by millimeter waves, so a millimeter wave radar is deployed in the monitoring environment, and the millimeter wave radar is equipped with multiple transmitting and receiving antennas. The specific implementation steps are as follows: Figure 2 As shown:

[0023] Step 1: The millimeter-wave radar transmits a frequency modulated continuous wave (FMCW) signal and receives the reflected signal.

[0024] Step 2: Collect millimeter wave data during the target object's motion as motion samples, including motion samples of a single person, multiple people's independent motion, and multiple people's interactive motion.

[0025] Step 3: De-noise the point cloud data of the action sample using a density-based clustering algorithm, DBSCAN (Density-Based Spatial Clustering of Applications with Noise), before proceeding to Steps 5 and 6. The denoised point cloud data in this step can be single-person point cloud data or multi-person point cloud data. The specific steps are as follows:

[0026] Step 3-1: Calculate the distance between neighboring points. First, enrich the multi-frame time series point cloud data, and calculate the distance between each point and its neighboring points in the enriched point cloud data;

[0027] Step 3-2: Plot the k-distance graph. Sort the distance from each point to its k-th nearest neighbor and plot these distances. The value of k corresponds to the minPts parameter in the DBSCAN algorithm.

[0028] Step 3-3: Determine DBSCAN parameters. Analyze the k-distance graph and select the inflection point as the neighborhood radius ε parameter of the DBSCAN algorithm.

[0029] Steps 3-4: DBSCAN denoising. Using the k value and the neighborhood radius ε parameter, perform DBSCAN clustering on the point cloud data. Points not included in any cluster are marked as noise and removed. The point cloud data after DBSCAN clustering denoising is output to steps 5 and 6.

[0030] Step 4: Point cloud clustering. After using the K-Means algorithm to cluster the denoised point cloud data identified by the action pattern classifier as multiple independent actions, the point cloud data from each cluster is treated as the point cloud data of a single person and output to steps 5 and 6, respectively. After K-Means clustering, each cluster represents the point cloud data of a single person.

[0031] Step 5: Extract the point trajectory segment (PTS) of the action sample. Apply the Hungarian algorithm to the input point cloud data to match and associate adjacent frame data points to form multiple trajectories. Then unify the length and number of trajectories to form trajectory segments and output them to step 7. The specific steps are as follows:

[0032] Step 5-1: Sort all input point cloud data by time frame.

[0033] Step 5-2: Feature point matching and connection. For each data point in a frame, the Hungarian algorithm is applied to match and connect the corresponding points in the next frame:

[0034] T i,t =Hungarian(Point i,t ,Point i,t+1 )

[0035] Among them, T i,t represents the i-th track of the t-th frame, Hungarian represents the Hungarian algorithm, Point i,t Represents the data point of the i-th trajectory of the t-th frame, Point i,t+1 represents the data point of the i-th trajectory in the t+1-th frame. In real scenes, the number of data points in each frame of the point cloud is usually different, so the constructed trajectory is usually composed of multiple trajectory segments.

[0036] Step 5-3: Unify the trajectory length. Since the number of points in each frame of the point cloud is different, the length of the constructed trajectory segments is different, so the length of the trajectory segments needs to be unified to L. If the trajectory segment length is not equal to L, the spline interpolation method is used to reconstruct the trajectory segment:

[0037] T i =Interpolate(T i , L)

[0038] Among them, T i represents the i-th trajectory, Interpolate represents the spline interpolation function, and L represents the reconstruction length of the spline interpolation function.

[0039] Step 5-4: Unify the number of trajectories. There is redundant information in multiple trajectories, so only representative trajectories need to be selected to reduce the computational complexity. The number of trajectories is unified to n. If the actual number of trajectories exceeds n, then filter according to the importance of the trajectories, eliminate unimportant trajectories, and only retain important trajectories. If the actual number of trajectories is less than n, then copy the trajectories with high importance. The method for screening according to importance is as follows:

[0040]

[0041] Among them, Importance is the importance calculation function, defined as the reciprocal of the average cosine similarity between the current trajectory Trajectory i and other trajectories, and is used to measure the difference between the current trajectory and other trajectories. When the importance of the trajectory T i is less than the preset threshold, then eliminate this trajectory, otherwise retain this trajectory.

[0042] The specific calculation is as follows:

[0043] Given the trajectory T i and the trajectory set {T1, T2,..., T n}, the cosine similarity cos(θ ij ) is defined as:

[0044]

[0045] Among them, T i ·T j represents the dot product of the trajectory T i and the trajectory T j |T i | and |T j | respectively represent the L1 norms of the trajectory T i and the trajectory T j The average cosine similarity MeanSim between the trajectory T i and other trajectories is calculated as:

[0046]

[0047] [[ID=**57**]]The importance Importace(T i ) is defined as the reciprocal of the average cosine similarity:

[0048]

[0049] Step 6: Extract the two-dimensional compressed Doppler map (CDM) of the motion sample. For the three-dimensional range Doppler data of the motion sample, refer to a position coordinate corresponding to the input point cloud data and use the coordinate position as the coordinate position of a target object. The target object here can be a single person or a group of people. Segment the local three-dimensional range Doppler data of each target object. Compress the three-dimensional range Doppler data of each target object in its velocity dimension, generate a two-dimensional compressed Doppler map (CDM), and output it to step 7. The specific steps are as follows:

[0050] Step 6-1: 3D range Doppler image segmentation. Obtain the location information R of the target object i based on the input point cloud data. i , based on this, the local 3D Doppler data of the target object is cut out from the 3D range Doppler data:

[0051]

[0052]

[0053] Among them, S j，r，v Represents the three-dimensional Doppler data of the jth frame, r is the distance index, v is the velocity index; R i is the location information of the target object i; Crop represents the cropping function; represents the local 3D Doppler data of target object i in the 3D Doppler data of the j-th frame; is the value representing the target object i in the j-th frame 3D Doppler data when the distance index is r and the velocity index is v in the local 3D Doppler data; represents the 3D Doppler data of target object i in frames 1 to t.

[0054] Step 6-2: Compression of local 3D range Doppler data. Compress the local 3D Doppler data in the velocity dimension and calculate its instantaneous velocity at each range index:

[0055]

[0056] Among them, v r is the instantaneous velocity at the distance index r, The two-dimensional Doppler data of the target object i in the three-dimensional Doppler data of the j-th frame is compressed in the velocity dimension;

[0057] Compress all frames. Compressed to Obtain a two-dimensional compressed Doppler CDM:

[0058]

[0059] in, represents the compressed 2D Doppler data of the target object i in the velocity dimension from frame 1 to frame t;

[0060] Step 6-3: The positive and negative values ​​in the and and Together they serve as three-channel data for a convolution-based feature extractor.

[0061] Step 7: Feature extraction and fusion. The Transformer model is used to extract features from the point cloud trajectory segment PTS. The self-supervised convolutional network model ConvNeXt is used to extract features from the three-channel data of the compressed Doppler map CDM. The two are then fused using a cross-attention mechanism to form the final action features. The specific steps are as follows:

[0062] Step 7-1: For the point cloud trajectory segment PTS, the Transformer model is used in combination with the attention mechanism to extract the temporal features in the action as PTS features.

[0063] Step 7-2: For the two-dimensional compressed Doppler map CDM, select ConvNeXt as the feature extractor. Through its deep convolution operation, it effectively captures the spatial features and complex structures in the CDM to obtain CDM features, such as Figure 1 shown.

[0064] Step 7-3: After feature extraction, the cross-attention mechanism is used to fuse PTS features and CDM features. By learning the complementarity between different data sources, the optimal combination of features is achieved. During fusion, the CDM features are used as the query vector Q, and the PTS features are used as the key K and value V. After extracting their useful information in the attention mechanism, the fused features are obtained by splicing. The fusion formula is:

[0065]

[0066] Among them, f cdm is the CDM characteristic, f ts is the PTS feature, W q is the query weight matrix, W k is the key weight matrix, W v is the value weight matrix, b q is the query bias, b k is the key offset, b v is the value bias, W f is the fusion weight, b f is the fusion bias, softmax is the Softmax activation function, T Indicates transpose.

[0067] Step 8: Build the task feature extraction part of the multi-task learning model, such as Figure 1 The model uses a multi-task feature extraction structure and gating mechanism to ensure efficient feature processing and information balance between tasks. The multi-task feature extraction structure includes a shared feature extraction layer, an action pattern discrimination task feature extraction layer, an independent action recognition task feature extraction layer, and an interactive action recognition task feature extraction layer, as follows:

[0068] Step 8-1: Multi-task feature extraction model structure. The model first processes the fused features through a shared task feature extraction layer. The shared feature extraction layer includes 4 feature extraction networks, and the structure of each feature extraction network is a multi-layer feedforward neural network. The shared task feature extraction layer focuses on capturing common information shared by all tasks to form shared features. Corresponding to each specific task of action pattern discrimination, independent action recognition, and interactive action recognition, the model extracts task features through three task feature extraction layers, namely the action pattern discrimination task feature extraction layer, the independent action recognition task feature extraction layer, and the interactive action recognition task feature extraction layer. Each task feature extraction layer contains 3 feature extraction networks, and the structure of each feature extraction network is also a multi-layer feedforward neural network. The task feature extraction layer is designed for specific tasks to ensure that task features can be extracted from the input features.

[0069] Step 8-2: Gating mechanism. The gating mechanism performs weighted filtering on the information output by each task feature extraction layer, and then outputs it to the corresponding task classifier to implement the corresponding task. The predecessor of the gating mechanism is the task feature extraction layer, which includes a shared feature extraction layer and three task feature extraction layers. Each task classifier obtains features from the shared task feature extraction layer and its own task feature extraction layer for classification. The shared feature extraction and task feature extraction are combined through the gating mechanism for selective fusion. The gating mechanism includes a gating network, whose structure is a single-layer feedforward neural network with a Softmax activation function. The output of the gating network for task p can be expressed as:

[0070] g p (x) = w p (x)S p (x)

[0071] Where x represents the input feature, w p (x) is the weight function of task p, which can be calculated through linear transformation and Softmax layer:

[0072] w p (x)=softmax(W p x)

[0073] in, is a parameter matrix, m s Represents the number of shared feature extraction networks, m p Represents the number of feature extraction networks for task p, and d represents the input feature S p The dimension of (x). S p (x) is composed of the shared task feature vector E s (1), E s (2), ..., E s (m s ) and the task feature vector E of task p p (1), E p (2), ..., E p (m p )composition:

[0074] S p (x)=[E p (1), E p (2), ..., E p (m p ), E s (1), E s (2), ..., E s (m s )] T

[0075] In this way, the input fusion features are passed through the shared feature extraction layer and the task feature extraction layer to obtain weighted filtered features, which are then output to the task classifier, allowing it to focus on task-related features and make full use of shared features.

[0076] Step 8-3: Action recognition. After the multi-task feature extraction model and gating network in the multi-task model, the predicted output y of task p p for:

[0077] y p (x)=Classifier p (g p (x))

[0078] Among them, Classifier p Predict a function for task p.

[0079] In order to classify different tasks, this paper designs three task classifiers, namely action pattern discriminator, independent action classifier and interactive action classifier, to construct the action recognition part of the multi-task learning model. The weighted filtered features after passing through the gated network are fed into the three task classifiers, such as Figure 1 As shown in Figure 2, the action mode classification results output by the action mode discriminator are divided into three cases:

[0080] (1) If the action pattern output by the action pattern discriminator is a single-person action, the action classification result output by the independent action classifier is directly used;

[0081] (2) If the action pattern output by the action pattern discriminator is a multi-person interactive action, the action classification result output by the interactive action classifier is directly used;

[0082] (3) If the action mode output by the action mode discriminator is an independent action of multiple people, the input is returned to the point cloud data of multiple single people segmented in step 4, and then the trajectory segments and compressed Doppler maps are obtained based on the point cloud data of the single person, and feature extraction and fusion are performed. The fused features are output to the multi-task learning model again to complete multiple single-person action recognition.

[0083] Step 9: Loss function construction. During model training, the multi-task combination loss function is defined by constructing a task mask to optimize the performance of the multi-task learning model and feature extractor. The specific steps are as follows:

[0084] Step 9-1: Determine the task mask. During model training, a task mask is used to specify the valid task corresponding to each sample. It is used to mask samples that are not applicable to a specific task, so that only the loss of the relevant task is calculated. The task mask is defined as:

[0085] mask = [action mode, independent action, interactive action]

[0086] If the corresponding bit meets the requirements, it is set to 1; if not, it is set to 0. There are three cases:

[0087] (1) For single-person action samples, the mask is marked as [1, 1, 0], which means that only the loss of the action pattern discrimination task and the independent action recognition task is calculated, and the loss of the interactive action recognition task is not calculated.

[0088] (2) For interactive action samples, the mask is marked as [1, 0, 1], which means that only the loss of the action mode discrimination task and the interactive action recognition task is calculated, and the loss of the independent action recognition task is not calculated.

[0089] (3) For multi-person independent action samples, the mask is marked as [1, 0, 0], indicating that only the loss of the action pattern discrimination task is calculated. Since it needs to be further segmented and re-identified, the loss of the independent action recognition task and the interactive action recognition task is not calculated.

[0090] Step 9-2: Define the loss function. Use the combined loss function Loss, which contains the classification loss of the three tasks. The classification loss of each task is calculated by cross entropy. The loss function L for each task p is p , using its mask mask p After weighting, the combined loss function is:

[0091]

[0092] where prediction p and truth p Represent the model's predicted output and true label for task p, respectively.

[0093] Step 10: During real-time action recognition, the millimeter wave signal of an unknown number of people and motion patterns is input. After preprocessing, the target object's point cloud trajectory segments (PTS) and compressed Doppler maps (CDM) are obtained. After feature extraction and fusion of the trajectory segments and compressed Doppler maps, the fused motion features are respectively input into the three classifiers of the multi-task learning model. If the action pattern classifier in the multi-task learning model outputs a single-person action, the action category output by the independent action classifier is directly used as the final action recognition result. If the action pattern classifier outputs a multi-person interactive action, the action category output by the interactive action classifier is directly used as the final action recognition result. If the action pattern classifier outputs a multi-person independent action, the process returns to step 4 to segment the original point cloud data into multiple single-person point cloud data. These point cloud data trajectory segments and compressed Doppler maps are then subjected to feature extraction and fusion, and then re-input into the multi-task learning model as single-person action data to obtain the multi-person independent action recognition result.

Claims

1. A millimeter wave multi-person action recognition method based on multi-task learning, characterized in that: The following steps are involved: Data collection steps: Receive millimeter-wave radar reflection signals in the monitoring environment and convert them into point cloud data; Point cloud denoising step: The point cloud data is first denoised using the density-based clustering algorithm DBSCAN to remove point cloud data not included in the cluster. The point cloud data obtained by DBSCAN denoising is output to the trajectory segment and compressed Doppler map output step; Point cloud clustering step: K-Means clustering is performed on the denoised point cloud data whose motion mode is determined to be multiple independent motions. The point cloud data of each cluster obtained by K-Means clustering is output to the trajectory segment and compressed Doppler map output steps respectively; The following steps are used to output trajectory segments and compressed Doppler maps: The input point cloud data is matched with associated data points to form several trajectories, and the length and number of the trajectories are then unified to form trajectory segments. Furthermore, the corresponding position information is obtained based on the input point cloud data. The three-dimensional range Doppler data of the millimeter-wave radar reflection signal is segmented according to the position information to obtain local three-dimensional range Doppler data. This local three-dimensional range Doppler data is then compressed in the velocity dimension to obtain a two-dimensional compressed Doppler map. Feature extraction and fusion steps: extract features from trajectory segments and two-dimensional compressed Doppler images respectively and fuse them to obtain motion features; Task feature extraction step: The task feature extraction part of the trained multi-task learning model receives the input action features; the shared feature extraction layer extracts common information shared by all tasks from the action features and outputs the shared features to the gating mechanism of each task feature extraction layer; the task feature extraction layer includes the action mode discrimination task feature extraction layer, the independent action recognition task feature extraction layer, and the interactive action recognition task feature extraction layer; The task feature extraction layer extracts action pattern discrimination task features, independent action recognition task features, and interactive action recognition task features from the action features and outputs them to the corresponding gating mechanism; the gating mechanism performs weighted filtering on the received shared features and corresponding task features and outputs them to the corresponding task classifier in the action recognition part of the multi-task learning model; Action recognition step: The action pattern classifier of the action recognition part of the multi-task learning model receives the weighted filtered features output by the action pattern discrimination task gating mechanism, the independent action classifier receives the weighted filtered features output by the independent action recognition task gating mechanism, and the interactive action classifier receives the weighted filtered features output by the interactive action recognition task gating mechanism; when the action pattern output by the action pattern classifier is a single-person action, the action category output by the independent action classifier is directly used as the final action recognition result; when the action pattern output by the action pattern classifier is a multi-person interactive action, the action category output by the interactive action classifier is directly used as the final action recognition result; when the action pattern output by the action pattern classifier is a multi-person independent action, the point cloud clustering step is repeated.

2. The method according to claim 1, wherein: The millimeter wave radar reflection signal is a frequency modulated continuous wave (FMCW) signal.

3. The method according to claim 1, wherein: In the trajectory segment and compressed Doppler map output step, the Hungarian algorithm is used to match the associated data points of the input point cloud data to form several initial trajectory segments, and then the length and number of trajectory segments are unified to form trajectory segments; the trajectory segments are screened or copied according to their importance.

4. The method according to claim 1, wherein: In the feature extraction and fusion steps, the Transformer model is used to extract features from the trajectory segments, and the ConvNeXt model is used to extract features from the two-dimensional compressed Doppler image.

5. The method according to claim 1, wherein: In the feature extraction and fusion steps, a cross-attention mechanism is used for feature fusion. During fusion, the two-dimensional compressed Doppler feature is used as the query vector Q, and the trajectory segment feature is used as the key K and value V.

6. The method according to claim 1, wherein: The shared feature extraction layer includes four feature extraction networks, and the structure of each feature extraction network is a multi-layer feedforward neural network.

7. The method according to claim 1, wherein: The three task feature extraction layers, namely the action mode discrimination task feature extraction layer, the independent action recognition task feature extraction layer and the interactive action recognition task feature extraction layer, respectively contain three feature extraction networks, and the structure of each feature extraction network is a multi-layer feedforward neural network.

8. The method according to claim 1, wherein: Each gating mechanism uses a single-layer feedforward neural network with the activation function being Softmax.

9. The method according to claim 1, wherein: The two-dimensional compressed Doppler image is three-channel data, one channel is the original two-dimensional Doppler data compressed by the velocity dimension, and the other two channels are the positive part and the negative part extracted from the original two-dimensional Doppler data compressed by the velocity dimension.

10. The method according to claim 1, wherein: During model training, a multi-task combination loss function is defined by constructing a task mask; the task mask is defined as: mask = [action mode, independent action, interactive action], and the corresponding bit is set to 1 if it meets the requirements, and set to 0 if it does not meet the requirements.

Images