Three-dimensional human behavior recognition method based on point cloud data

Abstract

The present application provides a three-dimensional human behavior recognition method based on point cloud data, which uses a convolution layer to perform dynamic convolution operations on a multi-frame original point cloud sequence of a target object in a time period T to obtain a multi-frame point cloud sequence; uses a spatial feature operation layer to build local associations for each frame point cloud sequence, obtains a spatial feature descriptor D and divides it into g groups, and decomposes each frame point cloud sequence into N local areas and divides it into g groups, matching g groups of feature descriptors; the spatial feature operation layer determines the aggregation weight according to the feature descriptor corresponding to the local area, completes the spatial feature aggregation operation based on the weight to obtain a frame aggregated point cloud sequence, obtains multi-frame spatial features based on the multi-frame point cloud sequence and the multi-frame aggregated point cloud sequence, and performs inter-frame fusion of the multi-frame spatial features to determine the behavior of the target object in the time period T. The method enhances the accuracy and reliability of human action recognition.

Description

Technical Field

[0001] The present application relates to the technical field of human motion recognition, and in particular to a three-dimensional human motion recognition method based on point cloud data. Background Art

[0002] Human behavior recognition based on video data usually relies on visual information such as color, texture, and appearance, which makes it susceptible to factors such as lighting changes, occlusion, background complexity, and changes in perspective. In addition, behavior recognition based on video data may need to process higher data dimensions and may have difficulty capturing subtle changes in movements. Although behavior recognition based on skeletal data reduces the dependence on complex backgrounds to a certain extent and improves the recognition accuracy of movements, it also has its own limitations. For example, skeletal data may have difficulty handling occlusion problems of human body parts and may not be sensitive enough to subtle differences in complex movements. In addition, the acquisition of skeletal data may require specific hardware support, such as depth cameras or infrared sensors, which may limit its application in certain scenarios.

[0003] Point clouds contain rich spatial information, which provides important supplementary clues for human action recognition. However, existing human action recognition methods based on point cloud sequences mainly rely on complex local spatiotemporal encoding, which not only increases the amount of calculation but also leads to the loss of spatial information. In addition, the extraction of local area features is only achieved through maximum pooling, which does not fully utilize the relationship between regions, resulting in low accuracy of action recognition. Summary of the invention

[0004] Based on this, it is necessary to provide a three-dimensional human behavior recognition method based on point cloud data to address the above technical problems.

[0005] In a first aspect, the present application provides a three-dimensional human behavior recognition method based on point cloud data, the method comprising:

[0006] Obtain a multi-frame original point cloud sequence of the target object in a time period T, and use the point relationship to guide the convolution layer of the network model to perform dynamic convolution operations on the multi-frame original point cloud sequence to obtain a multi-frame point cloud sequence; wherein each frame of the point cloud sequence includes N spatial coordinate points, and each spatial coordinate point has K nearest neighboring spatial coordinate points;

[0007] For each frame of point cloud sequence, the spatial feature operation layer of the network model is guided by point relationship to build local associations for each frame of point cloud sequence to obtain the spatial feature descriptor. , and use the point relationship to guide the spatial feature operation layer of the network model to transform the spatial feature descriptor Divide into groups, and each group determines the importance of the feature descriptors of the group through the activation function to guide the aggregation operation of spatial features;

[0008] For each frame of point cloud sequence, the spatial feature operation layer of the network model is guided by point relationship to decompose each frame of point cloud sequence into N local regions, and the N local regions are divided into Group to match Group feature descriptors;

[0009] For each frame of point cloud sequence, the spatial feature operation layer of the network model guided by point relationship determines the aggregation weight of each local area according to the feature descriptor corresponding to each local area, and completes the spatial feature aggregation operation based on the aggregation weight of each local area to obtain the frame aggregation point cloud sequence of each frame of point cloud sequence;

[0010] The feature connection layer of the point relationship-guided network model is used to splice multi-frame point cloud sequences and multi-frame aggregated point cloud sequences to obtain multi-frame spatial features. The inter-frame motion coding layer of the point relationship-guided network model is used to perform inter-frame fusion of multi-frame spatial features to determine the behavior of the target object in the time period T.

[0011] In one embodiment, the spatial feature operation layer includes a spatial feature descriptor module; for each frame of the point cloud sequence, the spatial feature operation layer of the point relationship guidance network model constructs a local association for each frame of the point cloud sequence to obtain a spatial feature descriptor. ,include:

[0012] For each frame of point cloud sequence, the spatial feature descriptor module is used to build local associations for each frame of point cloud sequence based on the nearest neighbor algorithm to determine the local features of the human body;

[0013] For each frame of point cloud sequence, the spatial feature descriptor module is used to Perform the maximum pooling operation on the local features of the human body in the set to obtain the pooling result, and determine the spatial feature descriptor based on the pooling result .

[0014] In one embodiment, for each frame of the point cloud sequence, a spatial feature descriptor module is used to construct a local association for each frame of the point cloud sequence based on a nearest neighbor algorithm to determine the local features of the human body, including:

[0015] For each frame of point cloud sequence, the spatial feature descriptor module is used to determine the center point feature Its nearest k neighbor point features The difference between the edge features and the center point features is obtained. Perform stitching to determine the local features of the human body ;in, represents edge features, represents the center point feature, , , For the Frame point cloud sequence, is any one of the N points in each frame of the point cloud sequence; and / or,

[0016] For each frame of point cloud sequence, the spatial feature descriptor module is used to Perform the maximum pooling operation on the local features of the human body in the set to obtain the pooling result, and determine the spatial feature descriptor based on the pooling result ,include:

[0017] For each frame of point cloud sequence, the spatial feature descriptor module is used to The local features of the human body in the collection Perform the maximum pooling operation to obtain the pooling result , spatial feature descriptor , It refers to the updated feature vector of each channel in the t-th frame point cloud sequence. Indicates the length of the feature channel.

[0018] In one embodiment, the spatial feature operation layer includes a spatial feature descriptor module; the spatial feature descriptor module is used to convert the spatial feature descriptor into Divide into group, where the Importance between group feature descriptors:

[0019] ;

[0020] in , Represents the number of feature descriptors contained in each group, represents the sigmoid activation function, For the Important factors of group feature descriptors.

[0021] In one embodiment, the spatial feature operation layer includes a spatial feature aggregation module; for each frame of the point cloud sequence, the spatial feature operation layer of the network model is guided by the point relationship to decompose each frame of the point cloud sequence into N local areas, and the N local areas are divided into Group to match Group feature descriptors, including:

[0022] For each frame of point cloud sequence, the spatial feature aggregation module is used to select N points As the center point, N local regions are obtained, and the N local regions are divided along the channel dimension based on a reshaping function Group to match Group feature descriptors;

[0023] The reshape function is , , the local area is divided into Group .

[0024] In one embodiment, the spatial feature operation layer includes a spatial feature aggregation module; for each frame of the point cloud sequence, the spatial feature operation layer of the point relationship-guided network model determines the aggregation weight of each local area according to the feature descriptor corresponding to each local area, including:

[0025] For each frame of point cloud sequence, the spatial feature aggregation module is used to multiply each set of feature descriptors with its corresponding local area to determine the aggregation weight of each local area. ;

[0026] in, , each descriptor Multiply it with its corresponding local area to obtain the aggregation weights of different local areas.

[0027] In one embodiment, a spatial feature aggregation operation is performed based on the aggregation weight of each local area to obtain a frame aggregated point cloud sequence of each frame point cloud sequence, including:

[0028] The spatial feature aggregation module is used to perform primary aggregation operations based on the aggregation weights of each local area to obtain the primary aggregation results corresponding to each frame of the point cloud sequence;

[0029] Utilizing the spatial feature aggregation module, a regional shuffling method is used to perform a secondary aggregation operation based on the primary aggregation results to enhance the spatial diversity of the human body. The inverse reshaping function is used to transform the secondary aggregation operation results to obtain a frame-aggregated point cloud sequence for each frame of the point cloud sequence.

[0030] In one embodiment, the primary polymerization operation is performed as follows:

[0031] ;

[0032] is the primary aggregation result of any frame point cloud sequence, The output dimension is ,in and denote the number of regions, neighbors and groups respectively, Indicates the number of channels per group; and / or,

[0033] Using the spatial feature aggregation module, a secondary aggregation operation is performed based on the primary aggregation result using the regional shuffling method to enhance the spatial diversity of the human body, including: using the spatial feature aggregation module to aggregate the primary aggregation result Transpose , and flatten it to obtain a flattened result, and perform a secondary aggregation operation based on the flattened result.

[0034] In one embodiment, inter-frame fusion of multi-frame spatial features is performed using an inter-frame motion coding layer of a point relationship-guided network model, including:

[0035] Using the inter-frame motion coding layer, pooling is performed on the spatial features of each frame to obtain the feature vector corresponding to the spatial features of each frame;

[0036] Using the inter-frame motion coding layer, multiple feature vectors corresponding to the multi-frame spatial features are divided into a preset number of time partitions, and a maximum pooling operation is performed on the feature vectors in each time partition to obtain a preset number of spatiotemporal feature vectors;

[0037] Using the inter-frame motion coding layer, a preset number of spatiotemporal feature vectors are fused.

[0038] In one embodiment, a multi-frame original point cloud sequence of a target object in a time period T is obtained, and a convolution layer of a point relationship-guided network model is used to perform a dynamic convolution operation on the multi-frame original point cloud sequence to obtain a multi-frame point cloud sequence, including:

[0039] Obtain a multi-frame original point cloud sequence of the target object in a time period T, and use the point relationship to guide the first convolutional layer of the network model to downsample and increase the dimension of the multi-frame original point cloud sequence to obtain a multi-frame secondary point cloud sequence;

[0040] The second convolutional layer of the relationship-guided network model is used to downsample and increase the dimension of the multi-frame secondary point cloud sequence to obtain a multi-frame point cloud sequence.

[0041] In the above-mentioned 3D human behavior recognition method based on point cloud data, the convolution layer of the point relationship guidance network model is used to perform dynamic convolution operation on the multi-frame original point cloud sequence of the target object in the time period T to obtain a multi-frame point cloud sequence; wherein, for each frame of the point cloud sequence, the spatial feature operation layer of the point relationship guidance network model constructs a local association for each frame of the point cloud sequence, and the obtained spatial feature descriptor is converted into Divide into Group and decompose each frame point cloud sequence into N local regions, and divide the N local regions into Group to match A group of feature descriptors is formed, and each group of feature descriptors determines the importance of the group of feature descriptors through an activation function to guide the aggregation operation of spatial features; further, the spatial feature operation layer determines the aggregation weight of each local area according to the feature descriptors corresponding to each local area, and completes the spatial feature aggregation operation based on the aggregation weight of each local area to obtain a frame-aggregated point cloud sequence for each frame of point cloud sequence; the feature connection layer of the point relationship-guided network model is used to splice multi-frame point cloud sequences and multi-frame aggregated point cloud sequences to obtain multi-frame spatial features, and the inter-frame motion coding layer of the point relationship-guided network model is used to perform inter-frame fusion of multi-frame spatial features to determine the behavior of the target object in the time period T. The present application overcomes the disorder and irregularity of point clouds, extracts discriminative human body structure descriptors based on the spatial feature operation layer, and aggregates body regions by grouping and activating these descriptors, thereby enhancing the spatial features of the human body, establishing intrinsic connections between different regions while maintaining their respective characteristics, and enhancing the accuracy and reliability of human motion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of a method for three-dimensional human behavior recognition based on point cloud data in one embodiment;

[0043] Figure 2 A schematic diagram of a process of constructing a local association to obtain a spatial feature descriptor in one embodiment;

[0044] Figure 3 A schematic diagram of a process for determining local features of a human body in one embodiment;

[0045] Figure 4 A schematic diagram of a flow chart of a reshaping function in one embodiment;

[0046] Figure 5 A schematic diagram of a process of completing a spatial feature aggregation operation using an aggregation weight for each local area in one embodiment;

[0047] Figure 6 A schematic diagram of the operation of a spatial feature operation layer in a point-to-point relationship guided network model in one embodiment;

[0048] Figure 7 A schematic diagram of a process of regional shuffling in one embodiment;

[0049] Figure 8 It is a schematic diagram of a process of using the inter-frame motion coding layer of the point relationship guidance network model to perform inter-frame fusion on multi-frame spatial features in one embodiment;

[0050] Fig. 9 A schematic diagram of a process for determining a spatiotemporal feature vector in one embodiment;

[0051] Fig.10 A schematic diagram of a flow chart of downsampling and upscaling operations on a multi-frame original point cloud sequence in a convolutional layer in one embodiment;

[0052] Fig.11 A network architecture diagram of a node relationship guided network model in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] In one embodiment, Figure 1 As shown, a 3D human behavior recognition method based on point cloud data is provided, comprising the following steps:

[0055] Step 101: obtaining a multi-frame original point cloud sequence of the target object in a time period T, and using the point relationship to guide the convolution layer of the network model to perform a dynamic convolution operation on the multi-frame original point cloud sequence to obtain a multi-frame point cloud sequence; wherein each frame of the point cloud sequence includes N spatial coordinate points, and each spatial coordinate point has K nearest neighboring spatial coordinate points;

[0056] Specifically, a multi-frame point cloud sequence of the target object in a time period T is obtained. The target object can be captured by using a depth camera to generate a continuous point cloud sequence, that is, a multi-frame original point cloud sequence. The original point cloud sequence is subjected to a dynamic convolution operation. A multi-frame point cloud sequence can be obtained by sampling and changing the dimension. One or more convolution layers can be used to perform dynamic convolution operations in sequence. Each frame of the original point cloud sequence in the multi-frame original point cloud sequence includes N0 spatial coordinate points (hereinafter referred to as points), and each point can be represented by its coordinates (x, y, z) in three-dimensional space. After performing a dynamic convolution operation on the multi-frame original point cloud sequence, a multi-frame point cloud sequence is obtained, and each frame of the point cloud sequence contains N spatial coordinate points. For each spatial coordinate point in each frame, algorithms such as KD tree or ball tree can be used to obtain the K nearest neighbor spatial coordinate points.

[0057] Step 102: for each frame of point cloud sequence, the spatial feature operation layer of the point relationship guidance network model is used to construct a local association for each frame of point cloud sequence to obtain a spatial feature descriptor D, and the spatial feature descriptor D is divided into g groups by using the spatial feature operation layer of the point relationship guidance network model, and the importance of each group of feature descriptors is determined by an activation function to guide the aggregation operation of spatial features;

[0058] Among them, for each point in each frame of the point cloud sequence, the K nearest neighbor points corresponding to each point can be found using methods such as KD tree, ball query or nearest neighbor algorithm. Each point in each frame of the point cloud sequence has features associated with it, and the K nearest neighbor points corresponding to each point also have features corresponding to it. By analyzing the feature differences between the features of each point in each frame of the point cloud sequence and the features of the K nearest neighbor points corresponding to it, and combining these feature differences with the features of the point, a local feature descriptor is formed. An aggregation strategy (such as maximum pooling, average pooling or attention mechanism) is used to extract key information from the local feature descriptor to obtain the spatial feature descriptor D of each frame of the point cloud sequence.

[0059] The spatial feature descriptors D of N points in each frame of the point cloud sequence are divided into g groups evenly or according to specific rules, each group contains N / g descriptors. For each group of spatial feature descriptors, an activation function, such as sigmoid or ReLU function, is used to determine the importance of the group of feature descriptors. The importance obtained by the activation function also plays an important role in guiding the aggregation of spatial features.

[0060] Step 103: for each frame of point cloud sequence, use the point relationship to guide the spatial feature operation layer of the network model to decompose each frame of point cloud sequence into N local regions, and divide the N local regions into g groups along the channel dimension to match the g groups of feature descriptors;

[0061] Among them, for each frame of the point cloud sequence of N points, each point and its K nearest neighboring points constitute a local area, that is, each frame of the point cloud sequence is divided into N local areas. The feature descriptors of these N local areas are divided into g groups along the channel dimension. Then each group will contain feature descriptors of N / g local areas. For each group of local areas to be divided into g groups along the channel dimension, the purpose is to correspond to the g groups of feature descriptors.

[0062] Step 104: for each frame of the point cloud sequence, the spatial feature operation layer of the point relationship-guided network model determines the aggregation weight of each local area according to the feature descriptor corresponding to each local area, and completes the spatial feature aggregation operation based on the aggregation weight of each local area to obtain a frame aggregation point cloud sequence of each frame of the point cloud sequence;

[0063] Specifically, the spatial feature operation layer of the network model guided by the point relationship determines the aggregation weight of each local area according to the feature descriptor of each local area. The aggregation weight may refer to the multiplication of the feature descriptor corresponding to each local area with the corresponding local area to obtain the aggregation weight of each area. Weighted aggregation is performed according to these aggregation weights and the features of the corresponding local areas to complete the spatial feature aggregation operation, wherein weighted aggregation may refer to the multiplication of the aggregation weight of each area with the feature vector of each area, and the weighted summation of the results obtained in each area is performed to finally obtain the frame aggregation point cloud sequence of each frame point cloud sequence.

[0064] Step 105: Use the feature connection layer of the point relationship guidance network model to splice the multi-frame point cloud sequence and the multi-frame aggregated point cloud sequence to obtain multi-frame spatial features, and use the inter-frame motion coding layer of the point relationship guidance network model to perform inter-frame fusion of the multi-frame spatial features to determine the behavior of the target object in the time period T.

[0065] Specifically, the feature connection layer of the network model is guided by point relationships to effectively splice multi-frame point cloud sequences and multi-frame aggregated point cloud sequences to integrate them into unified multi-frame spatial features. The time period T can be divided into different blocks, and each frame of the point cloud sequence in different blocks is first subjected to inter-frame fusion processing, and finally the blocks are subjected to inter-frame fusion processing as a whole to obtain the behavior of the target object in the entire time period T. Through the inter-frame motion coding layer of the model, these multi-frame spatial features are deeply fused between frames, so as to accurately capture the behavior changes of the target object in the time period T, and finally determine the behavior pattern of the target object in the time period T, that is, a specific behavior.

[0066] In this embodiment, the point relationship guidance network model processes the multi-frame original point cloud sequence of the target object within the time period T, and finally determines the behavior of the target object. By constructing local associations and extracting spatial feature descriptors, the point relationship network model can capture the attributes and structural information of each local area in each frame of the point cloud sequence, and enhance the expression ability of spatial features. By using the activation function to determine the importance of the feature descriptor group, the point relationship network model can identify key spatial features, thereby improving the accuracy of recognition. The inter-frame coding layer fuses the multi-frame spatial features, so that the point relationship guidance network model can analyze complex actions with time. This embodiment can overcome the disorder and irregularity of the point cloud, extract discriminative human body structure descriptors based on the spatial feature operation layer, and aggregate the body regions by grouping and activating these descriptors, enhance the spatial characteristics of the human body, establish the intrinsic connection between different regions while maintaining their respective characteristics, and enhance the accuracy and reliability of human body action recognition.

[0067] In one embodiment, Figure 2As shown, the spatial feature operation layer includes a spatial feature descriptor module; for each frame of point cloud sequence, the spatial feature operation layer of the point relationship-guided network model constructs a local association for each frame of point cloud sequence to obtain a spatial feature descriptor, including the following steps:

[0068] Step 201: for each frame of the point cloud sequence, a spatial feature descriptor module is used to construct a local association for each frame of the point cloud sequence based on a nearest neighbor algorithm to determine a local feature of the human body;

[0069] Among them, for a multi-frame original point cloud sequence, each frame of the original point cloud sequence contains N0 points. The multi-frame original point cloud sequence is subjected to a dynamic convolution operation through the convolution layer of the point relationship network model to obtain a multi-frame point cloud sequence. For each frame of the obtained multi-frame point cloud sequence, there are N spatial coordinate points (hereinafter referred to as points). For each point, the nearest neighbor algorithm is used to obtain the K nearest neighbor points. For each point and its K nearest neighbor points, a local association is constructed, that is, the difference between the features of each point and the features of its K nearest neighbor points is calculated to obtain the local edge features, and the local edge features are spliced ​​with the features of each point, and the splicing calculation is performed to obtain the local features of the human body.

[0070] Step 202: For each frame of point cloud sequence, use the spatial feature descriptor module to Perform the maximum pooling operation on the local features of the human body in the set to obtain the pooling result, and determine the spatial feature descriptor based on the pooling result ;

[0071] Specifically, the local features of the human body calculated in step 201 are Perform the maximum pooling operation on the set of is generated by N points of size The maximum pooling result is obtained by combining the edges of The maximum eigenvalue of the local features of the human body in the set, these values ​​constitute the pooling result. Determine the spatial feature descriptor based on the pooling result of the local features of the human body D .

[0072] In this embodiment, a nearest neighbor algorithm is used to build local associations for each point in each frame of the point cloud sequence, which helps capture the geometric and spatial relationships around each point, thereby determining local features of the human body. A maximum pooling operation is performed within the local feature set to extract the most significant features in the feature set. This step effectively aggregates local features, reduces redundant information, and highlights the features that are most critical to behavior recognition. Combining local associations and maximum pooling feature descriptors provides more accurate behavior recognition, especially in complex or dynamic environments.

[0073] In one embodiment, Figure 3As shown, for each frame of point cloud sequence, a spatial feature descriptor module is used to construct a local association for each frame of point cloud sequence based on a nearest neighbor algorithm to determine the local features of the human body, including the following steps:

[0074] Step 301: For each frame of point cloud sequence, use the spatial feature descriptor module to determine the center point feature Its nearest k neighbor point features The difference between the edge features and the center point features is obtained. Perform stitching to determine the local features of the human body ;in, represents edge features, represents the center point feature, , , For the Frame point cloud sequence, is any one of the N points in each frame of the point cloud sequence; and / or,

[0075] Specifically, for the point cloud sequence in the tth frame, the spatial feature descriptor module is used to determine N center points. Each center point uses the nearest neighbor algorithm to obtain the K nearest neighbor spatial coordinate points. The feature of each center point is calculated Features of its K nearest neighbors The edge features are obtained by combining the edge features with the features of each center point. , ,in, For the Frame point cloud sequence, For any central point among the N points in each frame of the point cloud sequence, a new feature vector is formed, namely the local feature of the human body. , the local human feature contains the spatial relationship information of each center point and its K neighboring points.

[0076] Step 302: For each frame of point cloud sequence, use the spatial feature descriptor module to Perform the maximum pooling operation on the local features of the human body in the set to obtain the pooling result, and determine the spatial feature descriptor based on the pooling result ,include:

[0077] For each frame of point cloud sequence, the spatial feature descriptor module is used to The local features of the human body in the collection Perform the maximum pooling operation to obtain the pooling result , spatial feature descriptor , It refers to the updated feature vector of each channel in the t-th frame point cloud sequence. Indicates the length of the feature channel.

[0078] Specifically, for each center point in the t-th frame point cloud sequence, these center points constitute the local feature set of each frame point cloud sequence ,in is generated by N center points. The set of edges. In the set, the local features of the human body are Perform the maximum pooling operation, that is, perform the maximum pooling operation on each feature channel. That is, among the features of each center point and its K nearest neighbor features, select the maximum value for each feature channel. Obtain the pooling result corresponding to each center point , which is the most significant feature value of the center point and its nearest neighbor in each channel. These pooled result vectors constitute the spatial feature descriptor in the t-th frame point cloud sequence , It refers to the updated feature vector of each channel in the t-th frame point cloud sequence. Indicates the length of the feature channel.

[0079] In this embodiment, by calculating the difference between the features of each point and the features of its K nearest neighboring points, a more comprehensive and rich edge feature can be obtained. By splicing the edge features and the features of each point, and the maximum pooling operation, the most significant features of the local features can be extracted, which helps to improve the accuracy of human behavior recognition. Among them, the maximum pooling can reduce local noise, thereby enhancing the robustness of the features. At the same time, through the maximum pooling operation, the most representative features can be effectively screened out, the redundant information in the data can be reduced, and the processing efficiency can be improved.

[0080] In one embodiment, the spatial feature operation layer includes a spatial feature descriptor module; the spatial feature descriptor module is used to convert the spatial feature descriptor Divide into group, where the Importance between group feature descriptors:

[0081] ;

[0082] in , Represents the number of feature descriptors contained in each group, represents the sigmoid activation function, For the The important factors of the group feature descriptor can be understood as the important factors that can be expressed Importance between group feature descriptors.

[0083] Specifically, there are c spatial feature descriptors D, where c represents the length of the feature channel. The spatial feature descriptor module is used to divide the spatial feature descriptors D into g groups, that is, each group contains c / g number of feature descriptors, represented by m. The importance of the group is determined by the activation function, and the sigmoid function is used here. The sigmoid function can map the input value to the interval (0,1), indicating the importance of each group of feature descriptors. The value obtained by the activation function can be used to explain the relative importance of the group of features in the overall features.

[0084] In this embodiment, the spatial feature descriptors are divided into groups by spatial feature descriptors, and the importance of each group of features is determined using the sigmoid activation function. By evaluating the importance of each group of feature descriptors, the feature group that contributes most to behavior recognition can be identified, and feature selection can be achieved to a certain extent, reducing unnecessary computational burden. At the same time, importance helps the model pay more attention to features that are more critical to recognition, which can improve the robustness of the model to noise and outliers.

[0085] In one embodiment, the spatial feature operation layer includes a spatial feature aggregation module; for each frame of point cloud sequence, the spatial feature operation layer of the network model is guided by the point relationship to decompose each frame of point cloud sequence into N local areas, and the N local areas are divided into Group to match Group feature descriptors, including:

[0086] For each frame of point cloud sequence, the spatial feature aggregation module is used to select N points As the center point, N local regions are obtained, and based on a reshaping function, the N local regions are divided along the channel dimension into Group to match Group feature descriptors;

[0087] The reshape function is , , the local area is divided into Group .

[0088] Specifically, for each frame of the point cloud sequence, the spatial feature aggregation module is used to select N points As the center point, each center point is used to define a local area. For each center point, its corresponding local area is determined, that is, N local areas are obtained. Figure 4 The reshape function is shown below: , , the local area is divided into Group , then the divided g groups of local regions correspond to g groups of spatial feature descriptors. In the set, the number of channels is c, there are g local area groups in total, and each group contains m descriptors. It should be noted that Represents dimension.

[0089] In this embodiment, each frame of the point cloud sequence is processed by the spatial feature aggregation module, the point cloud sequence is decomposed into local regions, and the reshaping function is used to divide it into groups along the channel dimension to match the number of spatial feature descriptor groups. By decomposing the point cloud sequence into local regions and further grouping them, the unique features of each local region can be captured and expressed more carefully. Detailed grouping can provide richer spatial information, help improve subsequent recognition accuracy, and reduce redundant feature information, so that more attention is paid to useful features, thereby improving processing efficiency and recognition performance.

[0090] In one embodiment, the spatial feature operation layer includes a spatial feature aggregation module; for each frame of the point cloud sequence, the spatial feature operation layer of the point relationship-guided network model determines the aggregation weight of each local area according to the feature descriptor corresponding to each local area, including:

[0091] For each frame of point cloud sequence, the spatial feature aggregation module is used to multiply each set of feature descriptors with its corresponding local area to determine the aggregation weight of each local area. ;

[0092] in, , each descriptor Multiply it with its corresponding local area to obtain the aggregation weights of different local areas.

[0093] Specifically, for each frame of point cloud sequence, the spatial feature descriptor module is used to obtain the feature descriptor of each group. The spatial feature aggregation module is used to multiply each group of feature descriptors with its corresponding local area to determine the aggregation weight of each local area. ,in Through the above operations, the aggregation weight of each local area is obtained, which reflects the importance of each area in the overall point cloud sequence. The local area with higher importance will make a greater contribution to the final point cloud sequence features.

[0094] In this embodiment, each local area in each frame of the point cloud sequence is determined by the spatial feature aggregation module, and each set of feature descriptors is multiplied by its corresponding local area to obtain the aggregation weight. The aggregation model can pay more attention to the local areas that have an important impact on behavior recognition and strengthen the expression of regional features. The weighted features of important local areas can provide more accurate spatial and structural information, which helps to improve the recognition accuracy of behaviors. The point relationship guidance network model can dynamically adjust the feature aggregation strategy according to the importance of the local area, and enhance the adaptability of the point relationship guidance network model to different scenarios and behaviors.

[0095] In one embodiment, Figure 5 As shown, the spatial feature aggregation operation is completed based on the aggregation weight of each local area to obtain a frame aggregated point cloud sequence for each frame point cloud sequence, including the following steps:

[0096] Step 501: using the spatial feature aggregation module, performing a primary aggregation operation based on the aggregation weight of each local area, and obtaining a primary aggregation result corresponding to each frame of the point cloud sequence;

[0097] The aggregation weights of each local area in each frame of the point cloud sequence are obtained through the spatial feature operation layer. These weights reflect the importance of each area in the overall features. For each frame of the point cloud sequence, the spatial aggregation module is used to perform weighted aggregation on each local area, that is, the feature vector of each local area is multiplied by its corresponding aggregation weight. The results of all weighted aggregations are subjected to primary aggregation operations, where the primary aggregation operation can refer to adding all weighted aggregation results to obtain the primary aggregation results corresponding to each frame of the point cloud sequence.

[0098] Step 502: Utilizing the spatial feature aggregation module, a secondary aggregation operation is performed based on the primary aggregation result using a regional shuffling method to enhance the spatial diversity of the human body, and an inverse reshaping function is used to transform the secondary aggregation operation result to obtain a frame-aggregated point cloud sequence for each frame of the point cloud sequence.

[0099] Among them, the primary aggregation result of each frame point cloud sequence is obtained through step 501, and the primary aggregation result is a vector of comprehensive local area weighted features. In order to improve the diversity of human body space, the regional shuffling method is used to process the primary aggregation result. Among them, regional shuffling can refer to rearranging the features of the primary aggregation result, such as first transposing the rows and columns, and then flattening it, and the flattened result is used as the input of the secondary aggregation operation. Regional shuffling can rearrange the order of local areas, break the constraints of the original spatial results, and enhance the model's adaptability to spatial changes. After the secondary aggregation operation, the inverse reshaping function is used to convert the result of the secondary aggregation operation into a traditional regional form to obtain a frame-aggregated point cloud sequence for each frame of the point cloud sequence.

[0100] In this embodiment, the primary aggregation operation is performed based on the aggregation weight of each local area by utilizing the spatial feature aggregation module to obtain the primary aggregation result corresponding to each frame of the point cloud sequence. The primary aggregation operation can strengthen the most important features in the local area, so that the point relationship guides the network model to pay more attention to the features that have a significant impact on behavior recognition. Weighted aggregation can more accurately capture the key information of the local area, thereby improving the accuracy of behavior recognition. Regional shuffling breaks the point cloud spatial feature results, increases the spatial diversity of features, and can better identify different human postures and movements. The inverse reshaping function converts the secondary aggregation results into a traditional regional pattern suitable for subsequent processing, so that the frame-aggregated point cloud sequence of each frame of the point cloud sequence can be obtained more conveniently.

[0101] It should be noted that if Figure 6 As shown in FIG. 1 , the spatial aggregation process is divided into two modules, one is the spatial feature descriptor module SFD, and the other is the spatial feature aggregation module SFA. L The frame contains N points, and the size of the image generated by N points is The set of edges .gather Each of the N points in the local edge feature is obtained to obtain the local feature vector . Using the Multi-Local Max Pooling mechanism, the size is Calculate locally to get the pooled result corresponding to each center point And use the pooled results to get the spatial feature descriptor results The feature descriptors are divided into g groups, each group contains D / g feature descriptors.

[0102] In the spatial aggregation module, N points are selected using the spatial feature aggregation module As the center point, obtain N local regions. For each center point, determine its corresponding local region. Use the reshape function to divide the local region into The g groups of local regions after division correspond to the g groups of spatial feature descriptors. The reshaping function is In the set, the number of channels is c, there are g local region groups in total, and each group contains m descriptors. The regions in the set are shuffled and flattened as the input of the secondary aggregation.

[0103] In one embodiment, Figure 7 As shown, the primary aggregation operation is as follows:

[0104] ;

[0105] is the primary aggregation result of any frame point cloud sequence, The output dimension is ,in and denote the number of regions, neighbors and groups respectively, Indicates the number of channels per group; and / or,

[0106] Using the spatial feature aggregation module, a regional shuffling method is used based on the primary aggregation results to perform a secondary aggregation operation to enhance the spatial diversity of the human body, including: using the spatial feature aggregation module to aggregate the primary aggregation results Transpose , and flatten it to obtain a flattened result, and perform a secondary aggregation operation based on the flattened result.

[0107] Among them, the operation of primary aggregation adopts the following method: ,in is the primary aggregation result of any frame point cloud sequence, The output dimension is ,in and denote the number of regions, neighbors and groups respectively, Indicates the number of channels in each group. For the primary aggregation results of each frame, a regional shuffling method is performed to enhance the spatial diversity of the human body. Specifically, the primary aggregation results are aggregated using the spatial feature aggregation module. Transpose The primary aggregation result is transposed in rows and columns. Here, the number of channels m and the number of groups g are transposed. The transposed result is flattened to obtain a flattened result, which can be used as the input of the secondary aggregation operation.

[0108] In this embodiment, the output dimension of the primary aggregation result considers the number of regions, neighbors and groups, as well as the number of channels in each group, which helps to fuse multi-dimensional feature information and provide a more comprehensive point cloud sequence for the model. Through the aggregation operation, features from different regions and neighbors can be integrated to capture more comprehensive spatial information, which facilitates the understanding of complex human behavior. The regional shuffling method disrupts the original spatial arrangement, combines features from different regions with each other, and enhances the expression of features. Transposition and flattening operations can simplify the data results, making subsequent secondary aggregation operations more efficient. Regional shuffling can reduce the dependence of the point relationship guidance network model on a specific arrangement, thereby reducing the risk of overfitting.

[0109] In one embodiment, Figure 8 As shown, the inter-frame motion coding layer of the point relationship-guided network model is used to perform inter-frame fusion of multi-frame spatial features, including the following steps:

[0110] Step 801: using the inter-frame motion coding layer, pooling is performed on each frame spatial feature to obtain a feature vector corresponding to each frame spatial feature;

[0111] Specifically, for the spatial features of a multi-frame point cloud sequence, each frame has a corresponding spatial feature vector. The inter-frame motion coding layer is used to perform a pooling operation on these multi-frame spatial features, with the aim of extracting the most representative feature vector from each frame. The pooling operation can be maximum pooling, that is, the largest eigenvalue is selected from the feature vector of each frame as the representative. After the pooling operation, multiple feature vectors representing the entire point cloud frame are obtained, which capture the key spatial information in each frame. And the feature vector is sequential in the time dimension, so it can together constitute the features of a time series, which is used to capture the dynamic characteristics of the point cloud sequence over time. Before aggregating these features, the position code is injected into the feature vector of each frame. Specifically, the position code is generated using sine and cosine functions of different frequencies. For example , and .in represents the number of channels in the feature vector, Indicates the time position, Indicates the dimension position. Represents the feature vector after injection position encoding. Since the feature vector of each frame comes from different points, an MLP block is introduced to fuse information from different feature dimensions.

[0112] Step 802: using the inter-frame motion coding layer, dividing multiple feature vectors corresponding to the multi-frame spatial features into a preset number of time partitions, and performing a maximum pooling operation on the feature vectors in each time partition to obtain a preset number of spatiotemporal feature vectors;

[0113] The feature vector is divided into multiple time partitions, and the maximum pooling operation is performed in each partition to achieve cross-frame feature vector aggregation. A two-layer structure is used to divide the feature vector into three time partitions, and the maximum pooling operation is performed in each partition. The maximum pooling operation extracts the most representative feature value from the feature vector of each time partition to obtain three spatiotemporal feature vectors. , , , each vector represents the key spatiotemporal information in the corresponding time partition. For example, in the three time partitions, represents the entire time period T, Represents 0-2 / T, Represents 2 / T—T.

[0114] Specifically, the detailed process of determining the spatiotemporal feature vector is as follows: Fig. 9As shown, the time period is T, m represents the number of descriptors, N is the number of spatial coordinate points in each frame of the point cloud sequence, and there are t frames in total after maximum pooling, which are represented by 0 to t-1 in the figure. Fi represents the feature vector after the injection position encoding. Using the inter-frame motion coding layer, multiple feature vectors are divided into a preset number of time partitions, using a two-layer structure. The feature vector is divided into three time partitions, the first time partition contains the time period T, the second time partition contains half of the time period 0-2 / T, and the third time partition contains half of the time period 2 / T-T. And perform the maximum pooling operation in each time partition, the maximum pooling operation extracts the most representative feature value from the feature vector of each time partition, and obtains three spatiotemporal feature vectors. , , , each vector represents the key spatiotemporal information in the corresponding time partition. In the three time partitions, is the spatiotemporal feature vector obtained by maximum pooling over the entire time period T, is the spatiotemporal feature vector obtained by maximum pooling within the 0-2 / T period, It is the spatiotemporal feature vector obtained by maximum pooling within the 2 / T-T period.

[0115] Step 803: using the inter-frame motion coding layer, a preset number of spatiotemporal feature vectors are fused.

[0116] Specifically, according to the above steps, the three spatiotemporal feature vectors , , The fused spatiotemporal feature vector F is obtained. The fused spatiotemporal feature vector F is input into a fully connected layer, where the fully connected layer can learn the high-level representation of the spatiotemporal feature vector and map it to the category space. The output of the fully connected layer can be expressed as: O = fully connected layer (F), where O is the output of the fully connected layer. Softmax normalization is applied to the output O of the fully connected layer to convert the output into a probability distribution, where each element represents the probability of the corresponding category. The result of Softmax normalization is a c-dimensional vector, where each dimension represents the probability of an action category. The action category may be leg raising, hip raising, or squatting. The category with the highest probability is considered to be the action represented by the current point cloud sequence.

[0117] In this embodiment, through inter-frame fusion, the model can capture dynamic features that change over time in the point cloud sequence. The fused spatiotemporal feature vector can provide more comprehensive action information, which helps to improve the accuracy of action recognition. Pooling and partitioning operations can reduce the high-dimensional spatiotemporal data to a more easily processed dimension while retaining relationship information. Dividing the feature vector into time partitions and performing maximum pooling operations can provide the before-and-after relationship of actions within each time partition, which helps to identify actions with time dependence. For complex actions that contain multiple stages or multiple action units, fusing the spatiotemporal feature vector can provide action coherence information across time, which helps to analyze and identify complex actions.

[0118] In one embodiment, Fig.10 As shown, a multi-frame original point cloud sequence of the target object in a time period T is obtained, and a convolution layer of the point relationship-guided network model is used to perform a dynamic convolution operation on the multi-frame original point cloud sequence to obtain a multi-frame point cloud sequence. It should be noted that in this embodiment, two consecutive convolution layers are used to downsample and increase the dimension of the multi-frame original point cloud sequence of the target object in the time period T. The following steps are included:

[0119] Step 1001: obtaining a multi-frame original point cloud sequence of the target object in a time period T, and using the first convolutional layer of the point relationship-guided network model to downsample and up-scale the multi-frame original point cloud sequence to obtain a multi-frame secondary point cloud sequence;

[0120] Among them, for the multi-frame original sequence of the target object acquired at the beginning within the time period T, since it contains a large amount of repeated information, the first convolutional layer of the network model guided by the point relationship is used to downsample and increase the dimension of the multi-frame original point cloud sequence to obtain a multi-frame secondary point cloud sequence. The FPS farthest point distance sampling method can be used to downsample the number of point clouds in each frame to halve its number. Specifically, if the number of original point cloud sequences per frame is N0, then the number of target points is determined to be N, so that N=N0 / 2. Use the FPS algorithm to randomly select a starting point from each frame of the original point cloud. For each point, calculate its distance to all unselected points in the point cloud, select the point with the farthest distance, and add it to the point set. Repeat the above process until the point set contains N points. The final point set is the number of point clouds after halving, which contains the most representative N points in the original point cloud of each frame. The first convolution layer is designed to contain multiple convolution kernels, each of which has a certain number of weight parameters. By increasing the number of convolution kernels, the feature dimension of each point in the original point cloud sequence is increased from the original dimension to a higher dimension.

[0121] Step 1002: Use the second convolutional layer of the relation-guided network model to downsample and up-dimensionalize the multi-frame secondary point cloud sequence to obtain a multi-frame point cloud sequence.

[0122] Among them, the multi-frame secondary point cloud sequence generated by the first convolution layer is downsampled and dimensionally increased by the second convolution layer to obtain a secondary multi-frame point cloud sequence. Similarly, the FPS method is used to downsample the secondary point cloud sequence, and the number is halved; by increasing the number of convolution kernels, the dimension of the secondary point cloud sequence is increased. After the second convolution layer performs a dynamic convolution operation, a multi-frame point cloud sequence is obtained.

[0123] In this embodiment, two consecutive convolutional layers are used to downsample and increase the dimension of a multi-frame original point cloud sequence of the target object within a time period T. Through two downsampling operations, the number of points in the point cloud is significantly reduced, the density of the data is reduced, thereby reducing the computational burden of subsequent processing, while reducing a large amount of duplicate information and improving the accuracy of subsequent recognition. The two dimensionality-up operations enable the model to extract more abstract and higher-level feature representations from the original point cloud data, which is conducive to understanding complex human behaviors and action patterns. Despite the downsampling, the dynamic convolution operation can retain the key information and structural features in the point cloud, ensuring the integrity and usefulness of the features. Through the downsampling and dimensionality-up operations of the two convolutional layers, the model can more accurately identify and distinguish different actions, improving the effect of action recognition.

[0124] It should be noted that if Fig.11 The entire network framework structure shown in the figure, for the target object contains multiple frames in the time period T, the total number of points in the time period can be represented by N, the dimension of each frame is represented by C0, and each frame can be represented as t, t-1, t-2, etc. For the multi-frame original point cloud sequence (for each frame of the original point cloud sequence, each frame contains N0 points, and the dimension of the point is C0), select the tth L Frame as input, tth LThe frame is subjected to two dynamic convolution operations through the first and second convolution layers, respectively obtaining the first convolution results N1, C1 (N1 is the number of point clouds per frame, C1 is the dimension) and the second convolution results N2, C2 (N2 is the number of point clouds per frame, C2 is the dimension). The point relationship-guided network model is used to obtain the third downsampling and dimension-upgraded multi-frame point cloud sequence for the second convolution result, that is, the third convolution result N3, C3 (N3 is the number of point clouds per frame, C3 is the dimension). The third convolution result is subjected to a dynamic convolution operation on the newly generated multi-frame point cloud sequence through the fourth convolution layer, obtaining the fourth downsampling and dimension-upgraded multi-frame point cloud sequence, that is, the fourth convolution result N4, C4 (N4 is the number of point clouds per frame, C4 is the dimension). The feature interpolation connection technology is used for the multi-frame point cloud sequence generated by these four downsampling and dimension-upgrade, that is, the results obtained by these four convolutions, and the outputs of the convolution layers extracted at different stages are merged to aggregate a richer spatial display. Here, we use multi-layer feature aggregation of features extracted from different layers. Since the output size of each layer is different from the original input, feature interpolation is applied to restore it to the input space. Similar to PointNet++, we use the 3 nearest neighbors The inverse distance weighted average method is used to calculate the point Interpolation characteristics of as follows:

[0125] ;

[0126] in , Represents the inverse distance weight.

[0127] The interpolated features are added to the embedding of the original points to generate the final output at each level. After propagating the features of each stage, they are aggregated as the input to the inter-frame motion coding layer. The aggregated spatiotemporal feature vector is input into a classifier, and the output of the classifier is a c-dimensional vector, where each dimension represents the probability of an action category, which may be leg raising, hip raising, or squatting. The category with the highest probability is considered to be the action represented by the current point cloud sequence.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A three-dimensional human behavior recognition method based on point cloud data, characterized in that: Obtain a multi-frame original point cloud sequence of the target object in a time period T, and use the point relationship to guide the convolution layer of the network model to perform dynamic convolution operations on the multi-frame original point cloud sequence to obtain a multi-frame point cloud sequence; wherein each frame of the point cloud sequence includes N spatial coordinate points, and each spatial coordinate point has K nearest neighboring spatial coordinate points; For each frame of point cloud sequence, the spatial feature operation layer of the network model is guided by the point relationship to construct a local association for each frame of point cloud sequence to obtain a spatial feature descriptor , and use the point relationship to guide the spatial feature operation layer of the network model to convert the spatial feature descriptor Divide into groups, and each group determines the importance of the feature descriptors of the group through the activation function to guide the aggregation operation of spatial features; For each frame of point cloud sequence, the spatial feature operation layer of the network model is guided by the point relationship to decompose each frame of point cloud sequence into N local areas, and the N local areas are divided into Group to match Group feature descriptors; For each frame of point cloud sequence, the spatial feature operation layer of the network model is guided by the point relationship to determine the aggregation weight of each local area according to the feature descriptor corresponding to each local area, and the spatial feature aggregation operation is completed based on the aggregation weight of each local area to obtain a frame aggregation point cloud sequence of each frame of point cloud sequence; The multi-frame point cloud sequence and the multi-frame aggregated point cloud sequence are spliced ​​using the feature connection layer of the point relationship guidance network model to obtain multi-frame spatial features, and the inter-frame motion coding layer of the point relationship guidance network model is used to perform inter-frame fusion on the multi-frame spatial features to determine the behavior of the target object in the time period T.

2. The three-dimensional human behavior recognition method according to claim 1, characterized in that: The spatial feature operation layer includes a spatial feature descriptor module; For each frame of point cloud sequence, the spatial feature operation layer of the network model is guided by the point relationship to construct a local association for each frame of point cloud sequence to obtain a spatial feature descriptor ,include: For each frame of the point cloud sequence, the spatial feature descriptor module is used to construct a local association for each frame of the point cloud sequence based on a nearest neighbor algorithm to determine the local features of the human body; For each frame of point cloud sequence, the spatial feature descriptor module is used to Perform a maximum pooling operation on the local features of the human body within the set to obtain a pooling result, and determine a spatial feature descriptor based on the pooling result .

3. The three-dimensional human behavior recognition method according to claim 2, characterized in that: For each frame of the point cloud sequence, the spatial feature descriptor module is used to construct a local association for each frame of the point cloud sequence based on the nearest neighbor algorithm to determine the local features of the human body, including: For each frame of point cloud sequence, the spatial feature descriptor module is used to determine the center point feature Its nearest k neighbor point features to obtain edge features, and compare the edge features with the center point features Perform stitching to determine the local features of the human body ;in, represents edge features, represents the center point feature, , , For the Frame point cloud sequence, is any one of the N points in each frame of the point cloud sequence; and / or, For each frame of point cloud sequence, the spatial feature descriptor module is used to Perform a maximum pooling operation on the local features of the human body within the set to obtain a pooling result, and determine a spatial feature descriptor based on the pooling result ,include: For each frame of point cloud sequence, the spatial feature descriptor module is used in the The local features of the human body within the collection Perform the maximum pooling operation to obtain the pooling result , spatial feature descriptor , It refers to the updated feature vector of each channel in the t-th frame point cloud sequence. Indicates the length of the feature channel.

4. The three-dimensional human behavior recognition method according to claim 1, characterized in that: The spatial feature operation layer includes a spatial feature descriptor module; The spatial feature descriptor module is used to convert the spatial feature descriptor Divide into group, where the Importance between group feature descriptors: ； in , Represents the number of feature descriptors contained in each group, represents the sigmoid activation function, For the Important factors of group feature descriptors.

5. The three-dimensional human behavior recognition method according to claim 1, characterized in that: The spatial feature operation layer includes a spatial feature aggregation module; For each frame of point cloud sequence, the spatial feature operation layer of the network model is guided by the point relationship to decompose each frame of point cloud sequence into N local areas, and the N local areas are divided into Group to match Group feature descriptors, including: For each frame of point cloud sequence, the spatial feature aggregation module is used to select N points As the center point, N local regions are obtained, and the N local regions are divided into Group to match Group feature descriptors; The reshaping function is , , the local area is divided into Group .

6. The three-dimensional human behavior recognition method according to claim 1, characterized in that: The spatial feature operation layer includes a spatial feature aggregation module; For each frame of point cloud sequence, the spatial feature operation layer of the network model is guided by the point relationship to determine the aggregation weight of each local area according to the feature descriptor corresponding to each local area, including: For each frame of point cloud sequence, the spatial feature aggregation module is used to multiply each set of feature descriptors with its corresponding local area to determine the aggregation weight of each local area. ; in, , each descriptor Multiply it with its corresponding local area to obtain the aggregation weights of different local areas.

7. The three-dimensional human behavior recognition method according to claim 6, characterized in that: The spatial feature aggregation operation is completed based on the aggregation weight of each local area, and a frame aggregation point cloud sequence of each frame point cloud sequence is obtained, including: Using the spatial feature aggregation module to perform a primary aggregation operation based on the aggregation weight of each local area, to obtain a primary aggregation result corresponding to each frame of the point cloud sequence; The spatial feature aggregation module is used to perform a secondary aggregation operation based on the primary aggregation result using a regional shuffling method to enhance the spatial diversity of the human body, and the secondary aggregation operation result is transformed using an inverse reshaping function to obtain a frame-aggregated point cloud sequence for each frame of the point cloud sequence.

8. The three-dimensional human behavior recognition method according to claim 7, characterized in that: The primary polymerization operation is carried out in the following manner: is the primary aggregation result of any frame point cloud sequence, The output dimension is ,in and denote the number of regions, neighbors and groups respectively, Indicates the number of channels in each group; and / or, Using the spatial feature aggregation module, a secondary aggregation operation is performed based on the primary aggregation result using a regional shuffling method to enhance the spatial diversity of the human body, including: using the spatial feature aggregation module to aggregate the primary aggregation result Transpose , and flattening is performed to obtain a flattened result, and a secondary aggregation operation is performed based on the flattened result.

9. The three-dimensional human behavior recognition method according to any one of claims 1 to 8, characterized in that: Using the point relationship to guide the inter-frame motion coding layer of the network model to perform inter-frame fusion on the multi-frame spatial features, including: Using the inter-frame motion coding layer, pooling is performed on each frame spatial feature to obtain a feature vector corresponding to each frame spatial feature; Using the inter-frame motion coding layer, multiple feature vectors corresponding to the multi-frame spatial features are divided into a preset number of time partitions, and a maximum pooling operation is performed on the feature vectors in each time partition to obtain a preset number of spatiotemporal feature vectors; The inter-frame motion coding layer is used to fuse a preset number of spatiotemporal feature vectors.

10. The three-dimensional human behavior recognition method according to claim 1, characterized in that: Obtain a multi-frame original point cloud sequence of the target object in the time period T, use the point relationship to guide the convolution layer of the network model to perform dynamic convolution operations on the multi-frame original point cloud sequence, and obtain a multi-frame point cloud sequence, including: Acquire a multi-frame original point cloud sequence of the target object in a time period T, and use the point relationship to guide the first convolutional layer of the network model to downsample and increase the dimension of the multi-frame original point cloud sequence to obtain a multi-frame secondary point cloud sequence; The second convolutional layer of the relationship-guided network model is used to downsample and increase the dimension of the multi-frame secondary point cloud sequence to obtain a multi-frame point cloud sequence.

Images