A gait recognition method, system, device and medium based on point cloud

Abstract

The application discloses a gait recognition method, system, device and medium based on point cloud, which comprises the following steps: performing point cloud projection on gait point cloud data to obtain a depth image sequence; encoding the depth image sequence to obtain initial features, inputting the initial features into a large-scale spatial attention module and a small-scale spatial attention module to extract multi-scale enhanced features, and performing feature fusion on the initial features and the multi-scale enhanced features to obtain first depth features; inputting the first depth features into a multi-scale space-time convolution network module to perform space dimension and time dimension feature extraction and fusion through a pseudo 3D convolution chain of several levels to obtain second depth features, and performing aggregation on the second depth features to obtain target depth features; and inputting real-time acquired gait data into a preset gait recognition model to determine a recognition object. The application can improve the accuracy and efficiency of gait recognition.

Description

A gait recognition method, system, device, and medium based on point clouds Technical Field

[0001] This invention relates to the field of public safety, and in particular to a gait recognition method, system, device, and medium based on point clouds. Background Technology

[0002] In the field of public safety, gait recognition technology plays an irreplaceable role. This technology identifies individuals by analyzing their unique walking postures. Its non-contact, long-distance recognition and insensitivity to lighting conditions give it unique advantages in a variety of complex scenarios. For example, a target person may deliberately disguise their appearance to avoid facial recognition systems, but they cannot simultaneously change their unique walking posture. Therefore, gait recognition can accurately identify individuals from a distance without alerting them, providing crucial clues for quickly locating and tracking targets, thereby improving the efficiency of case investigation and enhancing public safety.

[0003] Currently, many gait recognition systems rely primarily on contour map-based methods. These methods utilize only limited shape information, resulting in relatively simple feature extraction and difficulty in fully leveraging rich 3D structural information. Furthermore, in real-world reconnaissance scenarios, changes in terrain and weather significantly impact pedestrian gait. For example, gait patterns differ considerably when walking on snow or muddy surfaces, potentially leading to recognition errors. Additionally, wearing heavy clothing or carrying items can interfere with gait feature extraction, reducing recognition accuracy. Existing systems often operate on a fixed scale and cannot dynamically adapt to these changing conditions, resulting in inaccurate gait feature extraction and limited recognition accuracy and efficiency, thus affecting their practical application in public security reconnaissance. Summary of the Invention

[0004] This invention provides a point cloud-based gait recognition method, system, device, and medium to improve the accuracy and efficiency of gait recognition.

[0005] One embodiment of the present invention provides a gait recognition method based on point clouds, comprising:

[0006] Gait point cloud data within a preset area is acquired, and point cloud projection is performed on the gait point cloud data to obtain a depth map sequence;

[0007] The depth map sequence is encoded to obtain initial features, and the initial features are sequentially input into a large-scale spatial attention module and a small-scale spatial attention module to extract multi-scale enhanced features. The initial features and the multi-scale enhanced features are then fused to obtain the first depth features.

[0008] The first depth feature is input into a multi-scale spatiotemporal convolutional network module, and features in spatial and temporal dimensions are extracted and fused through several levels of pseudo-3D convolutional chains to obtain the second depth feature. The second depth feature is then aggregated to obtain the target depth feature.

[0009] The real-time acquired gait data is input into a preset gait recognition model to determine the recognition object, wherein the gait recognition model is trained based on the target depth features.

[0010] This invention, through point cloud projection of the gait point cloud data, converts 3D point cloud data into a 2D depth map sequence, clearly demonstrating the depth changes of human gait at different time points, while reducing data dimensionality and facilitating subsequent feature extraction. Feature extraction via large-scale and small-scale spatial attention modules allows for the separate focus on features at different scales within the gait, improving feature discriminativeness. Furthermore, feature fusion integrates information from different scales, resulting in a first depth feature that better characterizes the gait, thereby improving the accuracy of gait recognition. The multi-scale spatiotemporal convolutional network module simultaneously considers the spatial characteristics of the gait. By combining gait and temporal features, a second deep feature is obtained that accurately describes the spatiotemporal characteristics of gait. By aggregating these second deep features, features at different levels and scales extracted by a multi-scale spatiotemporal convolutional network can be integrated. The resulting target deep feature can more comprehensively characterize gait, thereby more accurately identifying the gait characteristics of different individuals, improving the accuracy of gait recognition, and reducing the processing burden of subsequent gait recognition models, thus improving the efficiency of gait recognition. By inputting the real-time acquired gait data into a preset gait recognition model, the identity of the identified object can be accurately and quickly identified, improving the accuracy of gait recognition. Compared with existing technologies, this application can improve the accuracy and efficiency of gait recognition.

[0011] Further, the step of projecting the gait point cloud data to obtain a depth map sequence specifically involves:

[0012] The point cloud coordinates and angular resolution parameters of each point in the gait point cloud data in 3D space are obtained, wherein the angular resolution is determined according to the lidar transmitter that acquires the gait point cloud data;

[0013] The point cloud depth value corresponding to each point in the gait point cloud data is determined based on the point cloud coordinates, and the projection coordinates of each point in the depth map are determined based on the point cloud coordinates and the angle resolution parameter.

[0014] Project all points of the gait point cloud data onto the projection coordinates corresponding to the depth map, and assign the point cloud depth value corresponding to the projection coordinates to obtain a depth map sequence.

[0015] By projecting the gait point cloud data, the 3D point cloud data can be converted into a 2D depth map sequence, which can clearly show the depth changes of human gait at different time points, while reducing the data dimensionality and facilitating subsequent feature extraction.

[0016] Furthermore, the initial features are sequentially input into the large-scale spatial attention module and the small-scale spatial attention module to extract multi-scale enhanced features, specifically as follows:

[0017] The initial features are convolved by the first convolutional layer to obtain the first output feature. The first output feature is then input into the large-scale spatial attention module to perform multiple pooling operations through the first pooling layer to obtain the second output feature. The second output feature is then activated by the first activation function to obtain the large-scale spatial attention weight. The large-scale spatial attention weight and the first output feature are multiplied to obtain the third output feature.

[0018] The third output feature is convolved by the second convolutional layer to obtain the fourth output feature. The fourth output feature is then input into the small-scale spatial attention module, where it undergoes multiple pooling operations through the second pooling layer to obtain the fifth output feature. The fifth output feature is then activated by the second activation function to obtain the small-scale spatial attention weights. The small-scale spatial attention weights and the fourth output feature are then multiplied to obtain the multi-scale enhanced feature.

[0019] By extracting features using large-scale and small-scale spatial attention modules, we can focus on features at different scales in the gait, thereby improving the discriminative power of the features.

[0020] Furthermore, each of the pseudo-3D convolutional chains includes spatial convolutional layers and temporal convolutional layers. Specifically, the first depth feature is input into a multi-scale spatiotemporal convolutional network module, and features in the spatial and temporal dimensions are extracted and fused through several levels of pseudo-3D convolutional chains to obtain the second depth feature.

[0021] The first depth feature is input into a multi-scale spatiotemporal convolutional network module, and the first depth feature is subjected to initial spatial dimension processing through a first spatial convolutional layer to obtain a first feature map. The spatial convolutional layer includes a first spatial convolutional layer and a second spatial convolutional layer.

[0022] The first feature map is subjected to temporal feature extraction through the temporal convolutional layer to obtain the second feature map, and the second feature map is subjected to secondary spatial processing through the second spatial convolutional layer to obtain the third feature map.

[0023] The third feature map and the first depth feature are fused to obtain a fused feature map. This process continues until each of the pseudo-3D convolutional chains has finished processing the fused feature map to obtain a second depth feature. The kernel size of each pseudo-3D convolutional chain gradually decreases.

[0024] In this way, the multi-scale spatiotemporal convolutional network module can simultaneously consider the spatial and temporal features of gait, obtain the second deep feature that accurately describes the spatiotemporal characteristics of gait, and thus facilitate subsequent gait recognition.

[0025] Further, the step of performing feature aggregation on the second depth feature to obtain the target depth feature specifically involves:

[0026] The second deep feature is input into the third pooling layer to perform maximum pooling operation on the second deep feature along the time dimension to obtain the first aggregated feature;

[0027] The first aggregated feature is input into the fourth pooling layer to perform horizontal pyramid pooling on the first aggregated feature to obtain the target depth feature.

[0028] By aggregating the second depth features, features at different levels and scales extracted by the multi-scale spatiotemporal convolutional network can be integrated. The resulting target depth features can more comprehensively represent gait, thereby more accurately identifying the gait features of different individuals, improving the accuracy of gait recognition, and reducing the processing burden of subsequent gait recognition models, thus improving the efficiency of gait recognition.

[0029] Further, the step of performing horizontal pyramid pooling on the first aggregated feature to obtain the target depth feature specifically involves:

[0030] The first aggregated feature is input into the fourth pooling layer to horizontally segment the first aggregated feature in the width dimension, resulting in several blocks;

[0031] Average pooling and max pooling operations are performed on each block to obtain the second aggregated feature corresponding to each block. The second aggregated features of each block are then concatenated to obtain the target depth feature.

[0032] By aggregating the second depth features, features at different levels and scales extracted by the multi-scale spatiotemporal convolutional network can be integrated. The resulting target depth features can more comprehensively represent gait, thereby more accurately identifying the gait features of different individuals, improving the accuracy of gait recognition, and reducing the processing burden of subsequent gait recognition models, thus improving the efficiency of gait recognition.

[0033] Furthermore, the step of inputting the real-time acquired gait data into a preset gait recognition model to determine the recognition object specifically involves:

[0034] The real-time acquired gait data is input into a preset gait recognition model to determine the corresponding query feature vector through model parameters, wherein the model parameters are obtained by training the model using a triplet loss function and a cross-entropy loss function.

[0035] Calculate the cosine similarity between the query feature vector and all library feature vectors in the preset database, and take the object corresponding to the library feature vector with the highest cosine similarity as the identification object.

[0036] By inputting the real-time acquired gait data into a preset gait recognition model, the gait recognition model can use the learned gait feature patterns to identify features in the gait data and match the identified features with a known gait feature library, thereby accurately identifying the identity of the identified object and improving the accuracy of gait recognition.

[0037] Another embodiment of the present invention provides a gait recognition system based on point clouds, comprising: an acquisition module, a first extraction module, a second extraction module, and a recognition module;

[0038] The acquisition module is used to acquire gait point cloud data within a preset area and perform point cloud projection on the gait point cloud data to obtain a depth map sequence.

[0039] The first extraction module is used to encode the depth map sequence to obtain initial features, and input the initial features into the large-scale spatial attention module and the small-scale spatial attention module in sequence to extract multi-scale enhanced features, and fuse the initial features with the multi-scale enhanced features to obtain the first depth feature;

[0040] The second extraction module is used to input the first depth feature into a multi-scale spatiotemporal convolutional network module, and extract and fuse the spatial and temporal dimensions through several levels of pseudo-3D convolutional chains to obtain the second depth feature, and perform feature aggregation on the second depth feature to obtain the target depth feature;

[0041] The recognition module is used to input real-time acquired gait data into a preset gait recognition model to determine the recognition object, wherein the gait recognition model is trained based on the target depth features.

[0042] This invention, through point cloud projection of the gait point cloud data, converts 3D point cloud data into a 2D depth map sequence, clearly demonstrating the depth changes of human gait at different time points, while reducing data dimensionality and facilitating subsequent feature extraction. Feature extraction via large-scale and small-scale spatial attention modules allows for the separate focus on features at different scales within the gait, improving feature discriminativeness. Furthermore, feature fusion integrates information from different scales, resulting in a first depth feature that better characterizes the gait, thereby improving the accuracy of gait recognition. The multi-scale spatiotemporal convolutional network module simultaneously considers the spatial characteristics of the gait. By combining gait and temporal features, a second deep feature is obtained that accurately describes the spatiotemporal characteristics of gait. By aggregating these second deep features, features at different levels and scales extracted by a multi-scale spatiotemporal convolutional network can be integrated. The resulting target deep feature can more comprehensively characterize gait, thereby more accurately identifying the gait characteristics of different individuals, improving the accuracy of gait recognition, and reducing the processing burden of subsequent gait recognition models, thus improving the efficiency of gait recognition. By inputting the real-time acquired gait data into a preset gait recognition model, the identity of the identified object can be accurately and quickly identified, improving the accuracy of gait recognition. Compared with existing technologies, this application can improve the accuracy and efficiency of gait recognition.

[0043] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the steps of the point cloud-based gait recognition method of the present invention.

[0044] Another embodiment of the present invention provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of the point cloud-based gait recognition method of the present invention. Attached Figure Description

[0045] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 is a flowchart illustrating an embodiment of the point cloud-based gait recognition method provided in this application;

[0047] Figure 2 is a schematic diagram of the structure of the depth-sensing attention module provided in this application;

[0048] Figure 3 is a flowchart illustrating steps S301 to S302 provided in this application;

[0049] Figure 4 is a flowchart illustrating steps S401 to S403 provided in this application;

[0050] Figure 5 is a schematic diagram of the structure of the multi-scale spatiotemporal convolutional network module provided in this application;

[0051] Figure 6 is a flowchart illustrating steps S601 to S602 provided in this application;

[0052] Figure 7 is a schematic diagram of an embodiment of the point cloud-based gait recognition system provided in this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0055] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0056] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0057] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0058] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0059] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0060] In the field of public safety, gait recognition technology is extremely important in scenarios such as target tracking. This technology analyzes an individual's walking posture to identify them, and its non-contact, long-distance, and light-insensitive characteristics give it unique advantages in complex scenarios. For example, while a target can easily disguise their appearance to evade facial recognition, they are unlikely to change their walking posture. Gait recognition can accurately identify individuals, helping to locate and track targets and improve public safety. Currently, most gait recognition systems rely on contour map-based methods, utilizing only limited shape information. Feature extraction is simple, but it is difficult to fully utilize 3D structural information. In actual reconnaissance, changes in terrain and weather can alter gait patterns. For example, walking on snow or muddy surfaces differs significantly from normal conditions, easily leading to recognition errors. Heavy clothing and carried items also interfere with gait feature extraction, reducing accuracy.

[0061] Referring to Figure 1, in order to solve the above problems, an embodiment of the present invention provides a gait recognition method based on point cloud, including steps S101 to S104;

[0062] Step S101: Obtain gait point cloud data within a preset area, and perform point cloud projection on the gait point cloud data to obtain a depth map sequence;

[0063] In some embodiments, acquiring gait point cloud data within a preset area specifically involves installing high-precision lidar or multi-view depth cameras in the preset area to collect gait point cloud data of individuals related to the case without infringing on citizens' privacy.

[0064] In some embodiments, projecting the gait point cloud data to obtain a depth map sequence includes: acquiring the point cloud coordinates and angular resolution parameters of each point in the gait point cloud data in 3D space, wherein the angular resolution is determined based on the lidar transmitter that acquired the gait point cloud data; determining the point cloud depth value corresponding to each point in the gait point cloud data based on the point cloud coordinates, and determining the projection coordinates of each point in the depth map based on the point cloud coordinates and the angular resolution parameters; projecting all points of the gait point cloud data onto the projection coordinates corresponding to the depth map, and assigning the point cloud depth value corresponding to the projection coordinates to obtain a depth map sequence. Specifically, firstly, the 3D spatial coordinates, i.e., the point cloud coordinates, of each point are directly read from the gait point cloud data captured by devices such as lidar sensors. The angular resolution parameters are determined based on the specifications of the lidar transmitter used, including the horizontal angular resolution. and vertical angle resolution Then, the point cloud depth value for each point cloud coordinate is calculated. The relevant calculation formula is: And combined with horizontal angular resolution Calculate the column coordinates of the point in the depth map. And combined with vertical angle resolution Calculate the row coordinates of the point in the depth map. This allows us to determine the projected coordinates of each point in the depth map. Finally, based on the calculated projected coordinates... The point cloud depth value of each point Fill to the corresponding pixel position in the depth map The above steps are repeated for the point cloud data at each timestamp to generate a series of depth maps, forming a depth map sequence. , In the formula, Represents the number of channels (e.g.) =3), T is the number of frames, and H and W are the height and width of the depth map sequence, respectively.

[0065] In some embodiments, projected coordinates The relevant calculation formula is as follows:

[0066]

[0067] In the formula, These are the column and row coordinates of the point in the depth map; and These are the horizontal angular resolution and the vertical angular resolution, respectively. These represent the coordinates of the point in 3D space.

[0068] It should be noted that the horizontal angular resolution and vertical angle resolution The average angular rate and vertical angular resolution between continuous lidar emitters are typically provided by the lidar manufacturer and reflect the angular spacing between adjacent emitted rays in the horizontal and vertical directions.

[0069] It should be noted that during the projection process, if multiple points are projected to the same location, the smallest depth value (representing the nearest point) is retained or the average depth value of these points is calculated.

[0070] By projecting the gait point cloud data, the 3D point cloud data can be converted into a 2D depth map sequence, which can clearly show the depth changes of human gait at different time points, while reducing the data dimensionality and facilitating subsequent feature extraction.

[0071] Step S102: Encode the depth map sequence to obtain initial features, and input the initial features into the large-scale spatial attention module and the small-scale spatial attention module in sequence to extract multi-scale enhanced features, and fuse the initial features with the multi-scale enhanced features to obtain the first depth feature;

[0072] It should be noted that the depth encoder, large-scale spatial attention module, and small-scale spatial attention module can be collectively referred to as the depth perception attention module. The structural diagram of the depth perception attention module is shown in Figure 2, and the specific content is as follows.

[0073] In some embodiments, the depth map sequence is encoded to obtain initial features, specifically by: encoding the depth map sequence... The input is fed into a depth encoder for preliminary processing to obtain initial features. Its dimensions are .

[0074] Referring to Figure 3, in some embodiments, the initial features are sequentially input into a large-scale spatial attention module and a small-scale spatial attention module to extract multi-scale enhanced features, including steps S301 to S302.

[0075] Step S301: The initial features are convolved through the first convolutional layer to obtain the first output feature. The first output feature is then input into the large-scale spatial attention module to perform multiple pooling operations through the first pooling layer to obtain the second output feature. The second output feature is then activated through the first activation function to obtain the large-scale spatial attention weight. The large-scale spatial attention weight and the first output feature are then multiplied to obtain the third output feature.

[0076] In some embodiments, firstly, the initial features are processed through a first convolutional layer. Perform a convolution operation to obtain the first output feature. The relevant formula is: In the formula, The kernel size represents the convolution kernel size. The first output feature is then input into the large-scale spatial attention module, where it undergoes multiple pooling operations through the first pooling layer, including max pooling, standard deviation pooling, and average pooling, to obtain... , and and will , and The second output feature is obtained by concatenating the features along the channel dimension. Secondly, regarding Perform a convolution operation (with a kernel size of 5×5) and activate the system using the Sigmoid activation function (the first activation function) to generate large-scale spatial attention weights. The relevant formula is: , The kernel size represents the convolution kernel size. 2D convolutional layers; finally, large-scale spatial attention weights are applied. With the first output feature Multiplying these together yields the features processed by the large-scale spatial attention module, which is the third output feature. This can be understood as a global feature, where the relevant formula is: In the formula, This represents a large-scale spatial attention module.

[0077] Step S302: The third output feature is convolved by the second convolutional layer to obtain the fourth output feature. The fourth output feature is then input into the small-scale spatial attention module to perform multiple pooling operations through the second pooling layer to obtain the fifth output feature. The fifth output feature is then activated by the second activation function to obtain the small-scale spatial attention weights. The small-scale spatial attention weights and the fourth output feature are multiplied to obtain the multi-scale enhanced feature.

[0078] In some embodiments, multiple pooling operations are performed through a first pooling layer to obtain the second output feature. The relevant formula is:

[0079] ;

[0080] ;

[0081] ;

[0082] ;

[0083] In the formula, , and These are the pooling results obtained after max pooling, standard deviation pooling, and average pooling operations, respectively. This is the first output feature; This is the second output feature.

[0084] In some embodiments, firstly, the third output feature is... Perform a convolution operation (with a kernel size of 3×3) to obtain the fourth output feature. The relevant formula is and the fourth output feature The small-scale spatial attention module is input to perform multiple pooling operations through the second pooling layer, including max pooling, standard deviation pooling, and average pooling. The pooling results are then concatenated along the channel dimension (not shown in the figure) to obtain the fifth output feature. Subsequently, the fifth output feature was analyzed. Perform a convolution operation (with a 3×3 kernel) and activate the system using the Sigmoid activation function (also known as the second activation function) to generate small-scale spatial attention weights. Finally, the small-scale spatial attention weights are... With the fourth output feature Multiplying these together yields the features processed by the small-scale spatial attention module, i.e., the multi-scale enhanced features. This can be understood as a local feature, where the relevant formula is: In the formula, This represents a small-scale spatial attention module.

[0085] It should be noted that traditional contour-based gait recognition models primarily rely on the ResNet architecture for feature extraction because binary contours contain relatively sparse gait information. However, the introduction of depth maps is a significant advancement, providing richer spatial and geometric information. This enhanced data pattern necessitates a more complex backbone network architecture to effectively capture and process the inherently complex distance relationships and 3D features within depth information.

[0086] It should be noted that the spatial attention module captures various statistical properties of spatial features through combinations of different pooling operations (mean, maximization, and standard deviation). By combining convolutional kernels of different sizes with the sigmoid activation function, attention weights are generated for each spatial location within different receptive fields. This method can dynamically adjust the attention scores at different spatial locations, enhancing the model's ability to focus on salient regions.

[0087] By extracting features using large-scale and small-scale spatial attention modules, we can focus on features at different scales in the gait, thereby improving the discriminative power of the features.

[0088] In some embodiments, the initial features are fused with the multi-scale enhanced features to obtain a first deep feature, specifically by fusing the initial features with the multi-scale enhanced features. and the multi-scale enhancement features Feature fusion is performed to obtain the first deep feature. The relevant formula is: Its dimensions remain unchanged. .

[0089] Step S103: Input the first depth feature into the multi-scale spatiotemporal convolutional network module, and extract and fuse the spatial and temporal dimensions through several levels of pseudo 3D convolutional chains to obtain the second depth feature. Then, perform feature aggregation on the second depth feature to obtain the target depth feature.

[0090] Referring to Figure 4, in some embodiments, each of the pseudo-3D convolutional chains includes a spatial convolutional layer and a temporal convolutional layer. The first depth feature is input into a multi-scale spatiotemporal convolutional network module, and the spatial and temporal features are extracted and fused through several levels of pseudo-3D convolutional chains to obtain the second depth feature. The kernel size of each pseudo-3D convolutional chain gradually decreases, including steps S401 to S403.

[0091] Please refer to Figure 5, which is a schematic diagram of the structure of the multi-scale spatiotemporal convolutional network module provided in this application. The multi-scale spatiotemporal convolutional network module includes several levels of pseudo-3D convolutional chains. Each level of the pseudo-3D convolutional chain includes two spatial convolutional layers and one temporal convolutional layer. It should be noted that the pseudo-3D convolutional chain is a convolutional operation used in deep learning to process spatiotemporal data. It combines the advantages of 2D convolution and 1D convolution to effectively capture spatial and temporal features while maintaining computational efficiency.

[0092] Step S401: Input the first depth feature into the multi-scale spatiotemporal convolutional network module, and perform initial spatial dimension processing on the first depth feature through the first spatial convolutional layer to obtain the first feature map, wherein the spatial convolutional layer includes the first spatial convolutional layer and the second spatial convolutional layer.

[0093] In some embodiments, the first depth feature Input a multi-scale spatiotemporal convolutional network module, where the first spatial convolutional layer in the first-level pseudo-3D convolutional chain is used to process the first depth feature. Initial spatial processing is performed to obtain the first feature map. The first spatial convolutional layer can be a 3D convolutional layer with a kernel size of 1×5×5, and the relevant formula is as follows: In the formula, The kernel size represents the convolution kernel size. 3D convolutional layers.

[0094] Step S402: Extract the temporal dimension features from the first feature map through the temporal convolutional layer to obtain the second feature map, and perform secondary spatial processing on the second feature map through the second spatial convolutional layer to obtain the third feature map;

[0095] In some embodiments, the first feature map is processed by a 3D convolutional layer with a kernel size of 5×1×1, i.e., a temporal convolutional layer. Feature extraction is performed along the time dimension to obtain the second feature map. The relevant formula is: In the formula, The kernel size represents the convolution kernel size. The second feature map is then processed by a 3D convolutional layer with a kernel size of 1×5×5, also known as the second spatial convolutional layer. A second spatial processing is performed to obtain the third feature map. The relevant formula is: In the formula, The kernel size represents the convolution kernel size. 3D convolutional layers.

[0096] Step S403: The third feature map and the first depth feature are fused to obtain a fused feature map. This process continues until each of the pseudo-3D convolutional chains has finished processing the fused feature map to obtain the second depth feature. The kernel size of each pseudo-3D convolutional chain gradually decreases.

[0097] In some embodiments, the third feature map and the first depth feature The fusion is performed to obtain the fused feature map of this level of pseudo-3D convolutional chain. The relevant formula is: Then, the above process is repeated twice, using convolutional layers with kernel sizes of 1×3×3, 3×1×1, and 1×3×3, and a convolutional layer with a kernel size of 1×1×1, respectively, to progressively capture finer local structural information. The number of channels is doubled each time, while the feature map size is halved (the size remains unchanged after the final processing), ultimately yielding the second depth feature. During processing, the kernel size is successively reduced to 1×3×3, 3×1×1, and 1×1×1 to adapt to the feature map size and capture increasingly finer local structural information. Simultaneously, after each level (each stage) of processing, the number of channels is doubled, and the feature map size is halved (the size remains unchanged after the final processing), until after multiple (e.g., 3) similar processing iterations, the second depth feature is obtained. At this point, the size of the tensor is .

[0098] It should be noted that multi-scale spatiotemporal convolutional networks can obtain refined spatiotemporal fusion features by processing in both spatial and temporal dimensions using convolutional kernels of different sizes. Therefore, through the processing of the multi-scale spatiotemporal convolutional network module, multi-scale spatiotemporal information is extracted from the feature map enhanced by the depth perception attention module, ultimately resulting in a second deep feature that integrates global and local spatial information as well as short-term and long-term temporal dependencies. This provides a more comprehensive and accurate spatiotemporal feature representation for subsequent gait recognition tasks, which helps improve the accuracy of gait recognition.

[0099] It should be noted that the traditional ResNet architecture also has shortcomings in handling temporal information. To address these challenges, this invention proposes a multi-scale spatiotemporal convolutional network module. This network aims to learn and extract multi-scale spatiotemporal information from the features highlighted by the deep perception attention module. The multi-scale spatiotemporal convolutional network module uses convolutional kernels of different sizes in the temporal and spatial dimensions to capture local and global spatial information, short-term and long-term temporal dependencies, and cross-scale spatiotemporal feature relationships. Furthermore, to optimize computational efficiency while maintaining performance, a pseudo-3D convolutional network structure is used, which effectively reduces computational overhead while ensuring the extraction of spatiotemporal information.

[0100] In this way, the multi-scale spatiotemporal convolutional network module can simultaneously consider the spatial and temporal features of gait, obtain the second deep feature that accurately describes the spatiotemporal characteristics of gait, and thus facilitate subsequent gait recognition.

[0101] Please refer to Figure 6. In some embodiments, the step of performing feature aggregation on the second depth feature to obtain the target depth feature includes steps S601 to S602.

[0102] Step S601: Input the second deep feature into the third pooling layer to perform maximum pooling operation on the second deep feature along the time dimension to obtain the first aggregated feature;

[0103] In some embodiments, the second depth feature The input is the third pooling layer, also known as the temporal pooling layer, which performs maximum pooling on the feature map sequence along the time dimension, aggregates the feature map sequence, and outputs the globally understood features, which is the first aggregated feature.

[0104] Step S602: Input the first aggregated feature into the fourth pooling layer to perform horizontal pyramid pooling on the first aggregated feature to obtain the target depth feature.

[0105] In some embodiments, the method includes: inputting the first aggregated feature into a fourth pooling layer to horizontally segment the first aggregated feature in the width dimension to obtain several blocks; performing average pooling and max pooling operations on each block to obtain a second aggregated feature corresponding to each block; and concatenating the second aggregated features of each block to obtain the target depth feature. Specifically, firstly, the first aggregated feature is input into the fourth pooling layer to perform horizontal pyramid pooling to obtain multi-scale information of the input feature. At this time, the first aggregated feature needs to be segmented into several blocks in the width dimension W. For example, if the specified number of blocks is 16, then the width of each block is W / 16. Then, average pooling and max pooling operations are performed on each block to obtain the second aggregated feature corresponding to each block. Finally, the second aggregated features after average pooling and max pooling are concatenated along the channel dimension to form the final target depth feature.

[0106] By combining average pooling and max pooling, the details of local features are preserved (max pooling), while the statistical information of global features is obtained (average pooling). Finally, the results of the two pooling methods are concatenated along the last dimension to form the final output feature.

[0107] By aggregating the second depth features, features at different levels and scales extracted by the multi-scale spatiotemporal convolutional network can be integrated. The resulting target depth features can more comprehensively represent gait, thereby more accurately identifying the gait features of different individuals, improving the accuracy of gait recognition, and reducing the processing burden of subsequent gait recognition models, thus improving the efficiency of gait recognition.

[0108] By aggregating the second depth features, features at different levels and scales extracted by the multi-scale spatiotemporal convolutional network can be integrated to obtain compact, informative, and more comprehensive target depth features that can characterize gait. This allows for more accurate identification of gait features of different individuals, improving the accuracy of gait recognition. At the same time, it reduces the processing burden of subsequent gait recognition models, thereby improving the efficiency of gait recognition.

[0109] Step S104: Input the real-time acquired gait data into a preset gait recognition model to determine the recognition object, wherein the gait recognition model is trained based on the target depth features.

[0110] In some embodiments, the gait recognition model is trained based on the target depth features. Specifically, after obtaining the target depth features, the target depth features are input into the initial gait recognition model to map the features to a new feature space through multiple fully connected layers to obtain a target feature vector (for example, the target depth features are mapped to a new feature space through 16 independent fully connected layers, each fully connected layer mapping the input features to a 256-dimensional feature vector). Then, the target feature vector is batch normalized, and the normalized features are used to train the cross-entropy loss and cross-entropy loss. When the weighted sum of the cross-entropy loss and cross-entropy loss, i.e., the total loss function value, is minimized, the model parameters of the gait recognition model can be determined to directly identify the query feature vector in the gait data for inference.

[0111] It should be noted that the training process uses a weighted sum of triplet loss and cross-entropy loss as the loss function. The formula for the triplet loss function is as follows: In the formula, The value of the triplet loss function. Denotes the set of positive sample pairs. Represents the set of negative sample pairs. Indicates anchor point Distance between positive samples Indicates anchor point The distance between the negative sample pairs This indicates that the boundary value is set to 0.2; and the formula for the cross-entropy loss function is: In the formula, The cross-entropy loss function value is given, where C is the number of categories, i.e., the number of identity IDs in the dataset. It is the one-hot encoding of the actual category label. This represents the class probability predicted by the model; the total loss function is... ,in, and These are preset weighting parameters. During training, an optimizer and learning rate scheduling strategy are used to update the model parameters. The model is adjusted in each iteration during training to improve recognition accuracy on the dataset. The specific training process is not the focus of this application and will not be elaborated here.

[0112] It should be noted that the trained gait recognition model will be tested on the SUSTech1K dataset, and the evaluation criteria are Rank-1 and Rank-5 accuracy. These metrics are used to evaluate the model's recognition performance.

[0113] In some embodiments, inputting the real-time acquired gait data into a preset gait recognition model to determine the recognition object specifically involves: inputting the real-time acquired gait data into the preset gait recognition model to determine the corresponding query feature vector through model parameters, wherein the model parameters are obtained after training the model using a triplet loss function and a cross-entropy loss function; calculating the cosine similarity between the query feature vector and all library feature vectors in a preset database, and selecting the object corresponding to the library feature vector with the highest cosine similarity as the recognition object. Specifically, after training the gait recognition model, the real-time acquired gait data is input into the preset gait recognition model to determine the corresponding query feature vector through model parameters. Then, the similarity is measured by calculating the cosine similarity between the query feature vector and all library feature vectors in the preset database, and the object corresponding to the library feature vector with the highest cosine similarity is selected as the recognition object, i.e., the relevant target person is determined.

[0114] It's important to note that cosine similarity is a metric used to measure the similarity between two vectors in a direction. Values ​​typically range from -1 to 1, where 1 indicates perfect similarity, -1 indicates complete dissimilarity, and 0 indicates no similarity (orthogonality). Cosine similarity calculates the cosine of the angle between the two vectors, therefore it is unaffected by the magnitude of the vectors and primarily reflects their directional similarity.

[0115] By inputting the real-time acquired gait data into a preset gait recognition model, the gait recognition model can use the learned gait feature patterns to identify features in the gait data and match the identified features with a known gait feature library, thereby accurately identifying the identity of the identified object and improving the accuracy of gait recognition.

[0116] This invention, through point cloud projection of the gait point cloud data, converts 3D point cloud data into a 2D depth map sequence, clearly demonstrating the depth changes of human gait at different time points, while reducing data dimensionality and facilitating subsequent feature extraction. Feature extraction via large-scale and small-scale spatial attention modules allows for the separate focus on features at different scales within the gait, improving feature discriminativeness. Furthermore, feature fusion integrates information from different scales, resulting in a first depth feature that better characterizes the gait, thereby improving the accuracy of gait recognition. The multi-scale spatiotemporal convolutional network module simultaneously considers the spatial characteristics of the gait. By combining gait and temporal features, a second deep feature is obtained that accurately describes the spatiotemporal characteristics of gait. By aggregating these second deep features, features at different levels and scales extracted by a multi-scale spatiotemporal convolutional network can be integrated. The resulting target deep feature can more comprehensively characterize gait, thereby more accurately identifying the gait characteristics of different individuals, improving the accuracy of gait recognition, and reducing the processing burden of subsequent gait recognition models, thus improving the efficiency of gait recognition. By inputting the real-time acquired gait data into a preset gait recognition model, the identity of the identified object can be accurately and quickly identified, improving the accuracy of gait recognition. Compared with existing technologies, this application can improve the accuracy and efficiency of gait recognition.

[0117] As shown in Figure 7, based on the above method embodiments, corresponding device embodiments are provided;

[0118] An embodiment of the present invention provides a gait recognition system based on point clouds, comprising: an acquisition module 100, a first extraction module 200, a second extraction module 300, and a recognition module 400;

[0119] The acquisition module 100 is used to acquire gait point cloud data within a preset area and perform point cloud projection on the gait point cloud data to obtain a depth map sequence.

[0120] The first extraction module 200 is used to encode the depth map sequence to obtain initial features, and input the initial features into a large-scale spatial attention module and a small-scale spatial attention module in sequence to extract multi-scale enhanced features, and fuse the initial features with the multi-scale enhanced features to obtain the first depth features;

[0121] The second extraction module 300 is used to input the first depth feature into a multi-scale spatiotemporal convolutional network module, and extract and fuse the spatial and temporal dimensions through several levels of pseudo-3D convolutional chains to obtain the second depth feature, and perform feature aggregation on the second depth feature to obtain the target depth feature.

[0122] The recognition module 400 is used to input real-time acquired gait data into a preset gait recognition model to determine the recognition object, wherein the gait recognition model is trained based on the target depth features.

[0123] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the point cloud-based gait recognition method provided by any of the above-described method embodiments of the present invention.

[0124] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0125] Based on the above embodiments of the point cloud-based gait recognition method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the point cloud-based gait recognition method of any embodiment of the present invention.

[0126] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0127] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0128] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0129] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the point cloud-based gait recognition method described in any of the above-described method embodiments of the present invention.

[0130] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0131] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A gait recognition method based on point clouds, characterized in that, include: Gait point cloud data within a preset area is acquired, and point cloud projection is performed on the gait point cloud data to obtain a depth map sequence; The depth map sequence is encoded to obtain initial features, which are then sequentially input into a large-scale spatial attention module and a small-scale spatial attention module to extract multi-scale enhanced features. The initial features and the multi-scale enhanced features are then fused to obtain a first depth feature. Specifically, the process of sequentially inputting the initial features into the large-scale spatial attention module and the small-scale spatial attention module to extract multi-scale enhanced features involves: performing a convolution operation on the initial features through a first convolutional layer to obtain a first output feature; inputting the first output feature into the large-scale spatial attention module; performing multiple pooling operations, including max pooling, standard deviation pooling, and average pooling, through a first pooling layer to obtain a second output feature; activating the second output feature through a first activation function to obtain large-scale spatial attention weights; multiplying the large-scale spatial attention weights and the first output feature to obtain a third output feature; and performing a convolution operation on the third output feature through a second convolutional layer to obtain a fourth output feature. The system extracts features and inputs the fourth output feature into the small-scale spatial attention module. Multiple pooling operations, including max pooling, standard deviation pooling, and average pooling, are performed through the second pooling layer to obtain the fifth output feature. The fifth output feature is then activated using a second activation function to obtain small-scale spatial attention weights. These small-scale spatial attention weights are multiplied by the fourth output feature to obtain the multi-scale enhanced feature. The first depth feature is input into the multi-scale spatiotemporal convolutional network module. It undergoes feature extraction and fusion in both spatial and temporal dimensions through several levels of pseudo-3D convolutional chains to obtain the second depth feature. The second depth feature is then aggregated to obtain the target depth feature. Each level of the pseudo-3D convolutional chain includes a first spatial convolutional layer, a temporal convolutional layer, and a second spatial convolutional layer, with the kernel size of each level of the pseudo-3D convolutional chain gradually decreasing. Real-time acquired gait data is input into a preset gait recognition model to determine the recognition object. The gait recognition model is trained based on the target depth feature.

2. The gait recognition method based on point clouds according to claim 1, characterized in that, The step of projecting the gait point cloud data to obtain a depth map sequence specifically involves: acquiring the point cloud coordinates and angular resolution parameters of each point in the gait point cloud data in 3D space, wherein the angular resolution is determined based on the lidar transmitter that acquired the gait point cloud data; determining the point cloud depth value corresponding to each point in the gait point cloud data based on the point cloud coordinates, and determining the projection coordinates of each point in the depth map based on the point cloud coordinates and the angular resolution parameters; projecting all points of the gait point cloud data onto the projection coordinates corresponding to the depth map, and assigning the point cloud depth value corresponding to the projection coordinates to obtain the depth map sequence.

3. The gait recognition method based on point clouds according to claim 1, characterized in that, The process of inputting the first depth feature into a multi-scale spatiotemporal convolutional network module, and extracting and fusing spatial and temporal features through several levels of pseudo-3D convolutional chains to obtain the second depth feature, specifically involves: inputting the first depth feature into the multi-scale spatiotemporal convolutional network module; performing initial spatial processing on the first depth feature through a first spatial convolutional layer to obtain a first feature map; extracting temporal features from the first feature map through the temporal convolutional layer to obtain a second feature map; performing secondary spatial processing on the second feature map through the second spatial convolutional layer to obtain a third feature map; fusing the third feature map and the first depth feature to obtain a fused feature map; and repeating this process until each of the pseudo-3D convolutional chains has processed the fused feature map to obtain the second depth feature.

4. The gait recognition method based on point clouds according to claim 1, characterized in that, The step of aggregating the second depth features to obtain the target depth features specifically involves: inputting the second depth features into a third pooling layer to perform maximum pooling along the time dimension to obtain a first aggregated feature; and inputting the first aggregated feature into a fourth pooling layer to perform horizontal pyramid pooling to obtain the target depth features.

5. The gait recognition method based on point clouds according to claim 4, characterized in that, The step of performing horizontal pyramid pooling on the first aggregated feature to obtain the target depth feature specifically involves: inputting the first aggregated feature into a fourth pooling layer to horizontally segment the first aggregated feature in the width dimension to obtain several blocks; performing average pooling and max pooling operations on each block to obtain the second aggregated feature corresponding to each block; and concatenating the second aggregated features of each block to obtain the target depth feature.

6. The gait recognition method based on point clouds according to claim 1, characterized in that, The step of inputting the real-time acquired gait data into a preset gait recognition model to determine the recognition object specifically involves: inputting the real-time acquired gait data into the preset gait recognition model to determine the corresponding query feature vector through model parameters, wherein the model parameters are obtained after training the model using a triplet loss function and a cross-entropy loss function; calculating the cosine similarity between the query feature vector and all library feature vectors in the preset database, and taking the object corresponding to the library feature vector with the highest cosine similarity as the recognition object.

7. A gait recognition system based on point clouds, characterized in that, include: The module comprises an acquisition module, a first extraction module, a second extraction module, and a recognition module. The acquisition module is used to acquire gait point cloud data within a preset area and project the gait point cloud data to obtain a depth map sequence. The first extraction module is used to encode the depth map sequence to obtain initial features, and input the initial features sequentially into a large-scale spatial attention module and a small-scale spatial attention module to extract multi-scale enhanced features. The initial features and the multi-scale enhanced features are then fused to obtain a first depth feature. Specifically, inputting the initial features sequentially into the large-scale spatial attention module and the small-scale spatial attention module to extract multi-scale enhanced features involves: performing a convolution operation on the initial features through a first convolutional layer to obtain a first output feature; inputting the first output feature into the large-scale spatial attention module to perform multiple pooling operations, including max pooling, standard deviation pooling, and average pooling, through a first pooling layer to obtain a second output feature; activating the second output feature through a first activation function to obtain large-scale spatial attention weights; multiplying the large-scale spatial attention weights and the first output feature to obtain a third output feature; and then performing a second convolutional layer on the initial features to obtain a third output feature. The third output feature is convolved to obtain a fourth output feature, which is then input into the small-scale spatial attention module. This fourth output feature undergoes multiple pooling operations (max pooling, standard deviation pooling, and average pooling) through a second pooling layer to obtain a fifth output feature. A second activation function is then applied to the fifth output feature to obtain small-scale spatial attention weights. These weights are multiplied by the fourth output feature to obtain the multi-scale enhanced feature. The second extraction module inputs the first depth feature into a multi-scale spatiotemporal convolutional network module. This module extracts and fuses spatial and temporal features through several levels of pseudo-3D convolutional chains to obtain a second depth feature. The second depth feature is then aggregated to obtain the target depth feature. Each level of the pseudo-3D convolutional chain includes a first spatial convolutional layer, a temporal convolutional layer, and a second spatial convolutional layer, with the kernel size decreasing progressively at each level. The recognition module inputs real-time acquired gait data into a preset gait recognition model to determine the recognition object. This gait recognition model is trained based on the target depth feature.

8. A terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the point cloud-based gait recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the point cloud-based gait recognition method as described in any one of claims 1-6.

Images