A method and system for identifying abnormal gait of skeleton based on generalized convolutional neural network

Abstract

The present invention belongs to the field of gait analysis and provides a method and system for identifying abnormal gait in a skeleton based on a generalized convolutional neural network. The method comprises constructing a gait graph using lower limb joints as nodes and skeletons as edges; grouping adjacent nodes in the gait graph based on distance and spatial relationships, and normalizing the non-Euclidean gait graph into a Euclidean feature graph, wherein the positional relationships of the nodes encode the connectivity between the joints; using a generalized convolutional network to model the geometric features encoded in the normalized Euclidean feature graph, generalizing the convolution kernel's ability to extract texture representations from images to extract geometric features from the gait graph, and obtaining a gait feature graph that aggregates the spatiotemporal neighbor node features; and predicting abnormal gait classification results based on the gait feature graph.

Description

Technical Field

[0001] The present invention belongs to the field of gait analysis, and in particular relates to a method and system for identifying abnormal gait of skeleton based on generalized convolutional neural network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Various age-related degenerative diseases and neurological disorders, such as Parkinson's disease and Alzheimer's disease, can lead to abnormal gait. Traditional gait analysis methods rely on the subjective experience of medical experts or specialized measurement equipment, which is inefficient and expensive. With the development of sensor technology and artificial intelligence, automatic abnormal gait recognition is gaining increasing attention.

[0004] In the field of gait recognition, RGB video can be used to achieve gait identity recognition. However, due to changes in clothing and scenes, abnormal gait patterns based on RGB video are difficult to be effectively identified. Optical motion capture systems (MOCAP) can capture accurate gait data, but they are inefficient and expensive. Inexpensive wearable inertial measurement devices are widely used for abnormal gait recognition, but wearable devices restrict the patient's activities. Since low-cost depth sensors can track the position of 3D human skeleton joints in real time without interfering with the patient's activities, abnormal gait recognition technology based on skeleton data has made great progress.

[0005] Some studies have used artificial gait features extracted based on clinical prior knowledge for abnormal gait recognition. However, due to the omission of important features, artificial gait features lack representativeness and generalizability. Recurrent neural networks (RNNs) and temporal convolutional networks (TCNs) excel in automatically extracting data features and have been widely used in abnormal gait recognition, significantly improving its performance. However, because gait skeleton data is inherently a series of non-Euclidean graphs, these methods cannot effectively learn the underlying spatial dependencies among skeleton joints.

[0006] Graph convolutional networks (GCNs) extend convolution from images to graph-structured data, achieving excellent performance in graph-structured feature modeling in many fields. In skeleton-based action recognition, spatiotemporal graph convolution first uses graph convolution to extract the spatiotemporal features of skeleton data. Building on this, some studies have proposed adaptive graph convolution methods to construct non-local spatial features. Others have proposed three-dimensional spatiotemporal skeleton graphs to promote the flow of spatiotemporal information. However, because the collection of patient gait data requires many formal licenses and special laboratory settings, few benchmark datasets are open in the field of abnormal gait recognition. Although these graph convolution-based methods can achieve good results in action feature modeling, it is difficult to achieve better results on limited abnormal gait datasets. Summary of the Invention

[0007] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a method and system for identifying abnormal gait in skeletons based on a generalized convolutional neural network. The method can design a gait graph and a corresponding neighbor node grouping strategy based on the human body structure, which can effectively reflect the geometric relationship between joints. The non-Euclidean gait graph is normalized into a Euclidean feature graph, in which the positional relationship of the nodes encodes the connectivity between the joints. Then, a generalized convolutional network is proposed to model the geometric features encoded in the normalized gait feature graph, generalizing the ability of traditional convolution kernels to extract texture representations from images to extract geometric features of skeleton graphs.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A first aspect of the present invention provides a method for recognizing abnormal gait of skeletons based on a generalized convolutional neural network.

[0010] A skeleton abnormal gait recognition method based on a generalized convolutional neural network, comprising:

[0011] The gait graph is constructed with lower limb joints as nodes and bones as edges;

[0012] Adjacent nodes in the gait graph are grouped according to distance and spatial relationship, and the non-Euclidean structure of the gait graph is normalized into a Euclidean feature graph, in which the positional relationship of the nodes encodes the connectivity between joints.

[0013] A generalized convolutional network is used to model the geometric features encoded in the normalized Euclidean feature map. The ability of the convolution kernel to extract texture representation from the image is generalized to extract the geometric features of the gait graph, resulting in a gait feature map that aggregates the features of spatiotemporal neighbor nodes.

[0014] According to the gait feature map, the abnormal gait classification results are predicted.

[0015] A second aspect of the present invention provides a skeleton abnormal gait recognition system based on a generalized convolutional neural network.

[0016] A skeleton abnormal gait recognition system based on generalized convolutional neural network, comprising:

[0017] A graph construction module is configured to: construct a gait graph using lower limb joints as nodes and bones as edges;

[0018] a normalization module configured to group adjacent nodes in the gait graph according to distance and spatial relationship, and normalize the non-Euclidean structure of the gait graph into a Euclidean feature graph; wherein the positional relationship of the nodes encodes the connectivity between the joints;

[0019] A modeling module is configured to: use a generalized convolutional network to model the geometric features encoded in the normalized Euclidean feature map, generalize the ability of the convolution kernel to extract texture representation from the image to extract gait graph geometric features, and obtain a gait feature map that aggregates spatiotemporal neighbor node features;

[0020] The prediction module is configured to predict the abnormal gait classification result based on the gait feature map.

[0021] A third aspect of the present invention provides a computer-readable storage medium.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for identifying abnormal gait of a skeleton based on a generalized convolutional neural network as described in the first aspect above.

[0023] A fourth aspect of the present invention provides a computer device.

[0024] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for identifying abnormal gait of a skeleton based on a generalized convolutional neural network as described in the first aspect above are implemented.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention designs a gait graph and a corresponding neighbor node grouping strategy according to the human body structure, which can effectively reflect the geometric relationship between joints.

[0027] This paper normalizes the non-Euclidean gait graph into a Euclidean feature graph, and then proposes a generalized convolutional network to model the geometric features encoded in the normalized gait feature graph. It generalizes the ability of traditional convolution kernels to extract texture representations from images to extract skeleton graph geometric features, effectively improving the classification accuracy of abnormal gait recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0029] Figure 1 1 is a flow chart of a method for identifying abnormal gait based on a skeleton based on a generalized convolutional neural network.

[0030] FIG2( a ) is a gait diagram shown in the present invention;

[0031] FIG2( b ) shows a gait graph of the present invention being decomposed into five subgraphs according to the grouping of neighboring nodes;

[0032] FIG3( a ) is a schematic diagram of a normalized gait graph according to the present invention;

[0033] FIG3( b ) is a spatiotemporal step-wise convolution diagram shown in the present invention;

[0034] FIG3( c ) is a synchronous spatiotemporal convolution diagram shown in the present invention;

[0035] FIG4( a ) is an algorithm structure diagram of the NG-GCNs model shown in the present invention;

[0036] Figure 4(b) is a block flow chart of the NG-CNN based on the spatiotemporal decomposition convolutional network shown in the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0039] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0040] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to the various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code can include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of the boxes in the flowchart and / or block diagram, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0041] Example 1

[0042] like Figure 1 As shown, this embodiment provides a method for identifying abnormal gait of skeleton based on generalized convolutional neural network. This embodiment takes the application of this method to a server as an example. It can be understood that this method can also be applied to a terminal, and can also be applied to a system including a terminal, a server, and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0043] The gait graph is constructed with lower limb joints as nodes and bones as edges;

[0044] Adjacent nodes in the gait graph are grouped according to distance and spatial relationship, and the non-Euclidean structure of the gait graph is normalized into a Euclidean feature graph, in which the positional relationship of the nodes encodes the connectivity between joints.

[0045] A generalized convolutional network is used to model the geometric features encoded in the normalized Euclidean feature map. The ability of the convolution kernel to extract texture representation from the image is generalized to extract the geometric features of the gait graph, resulting in a gait feature map that aggregates the features of spatiotemporal neighbor nodes.

[0046] According to the gait feature map, the abnormal gait classification results are predicted.

[0047] In image processing, texture representation is usually achieved through the inherent properties of convolution kernels. This method generalizes the characteristics of convolutional neural networks to abnormal gait recognition based on skeleton data to learn the spatiotemporal geometric features of gait.

[0048] This embodiment proposes a gait skeleton graph and neighbor node grouping strategy based on the geometric relationship of human joints. The gait graph is constructed with the lower limb joints as vertices and the bones as edges, as shown in Figure 2(a). Grouping adjacent nodes according to distance and spatial relationship can reflect the spatial posture relationship characteristics of the joint points. Specifically, if the spatial perception domain is 2-hop, the neighbor nodes are divided into 5 subsets, and the gait graph is divided into 5 subgraphs as shown in Figure 2(b). Correspondingly, the adjacency matrix A of the gait graph is also divided into 5 parts {A0, A1, A2, A3, A4,}.

[0049] In order to aggregate the features of neighbor nodes, the adjacency matrix is ​​used in the graph convolution to perform weighted averaging of neighbor features. Since the neighbor nodes are grouped in the skeleton graph, the step-by-step aggregation rules of spatial graph convolution and temporal convolution are as follows:

[0050]

[0051] in, is a learnable parameter, is the feature map of layer l, Is the normalized adjacency matrix, σ(·) is the activation function, and TCN is the time convolution. If a symmetric normalized adjacency matrix is ​​used Or normalized adjacency matrix in graph convolution (GCNs) Where I is the identity matrix, D is the metric matrix of A, yes The metric matrix of will add self-loops of the root node in each subgraph. As the number of neighbor subgraphs increases, the number of self-loops will also increase accordingly, which is obviously not conducive to modeling the dependency relationship between nodes. If it is a random walk matrix As a normalized adjacency matrix, redundant root node self-loops are no longer generated, and the node feature update strategy can be converted to:

[0052]

[0053] According to formula (2), we rearrange the positions of the joints in the gait feature graph to achieve normalization of the gait graph, as shown in Figure 3(a). The positional relationship of the joints on the gait feature graph encodes the connectivity between nodes. By converting the non-Euclidean structure gait graph into a Euclidean structure gait feature graph, the traditional convolution kernel can aggregate the neighbor node features on the normalized gait feature graph, as shown in Figure 3(b). In addition, the traditional convolution kernel can achieve synchronous aggregation of spatiotemporal features without constructing a cross-spatiotemporal neighbor graph and its corresponding adjacency matrix, as shown in Figure 3(c). The spatiotemporal step-by-step aggregation of neighbor features can be expressed as formula (3), which is equivalent to the graph convolution with random walk normalized adjacency matrix shown in formula (2).

[0054]

[0055] in, is a normalized gait feature map, and Spatial-CNN is a spatial convolutional network. As shown in Figures 3(b) and 3(c), the spatiotemporal dependencies of skeleton nodes can be modeled by sliding the convolution kernel along the spatiotemporal dimension on the normalized gait feature map, thereby generalizing the convolution kernel's ability to extract image texture features to the gait skeleton map. The generalized convolutional neural network is applicable to neighborhood graphs of arbitrary size. The spatial size k of the convolution kernel means that the spatial receptive domain of the convolution is a (k-1) / 2-hop neighborhood graph.

[0056] As shown in Figure 3(b), in spatiotemporal step-by-step aggregation, nodes can only be connected to the same node in consecutive frames. The decomposed spatial convolution kernel and temporal convolution kernel slide sequentially on the normalized gait feature map to model the spatiotemporal correlation of joints. As shown in Figure 3(c), in spatiotemporal synchronous aggregation, all nodes in the spatiotemporal perception domain are connected to the root node, and the spatiotemporal synchronous convolution kernel slides on the gait feature map to model the spatiotemporal relationship of nodes.

[0057] Network structure and training

[0058] The model of this embodiment consists of three NG-CNN modules, as shown in Figure 4(a), where the three modules have 64, 128, and 256 output channels respectively; each NG-CNN module aggregates the spatiotemporal neighbor features to generate a new gait feature map, and the three blocks are updated layer by layer to obtain a deep gait feature map. The time step of all modules is set to 2. Finally, global average pooling, fully connected layers, and softmax are used to infer the abnormal pattern of a given gait. In addition, we use spatiotemporal step convolution ( Figure 3b ) to construct the NG-CNN module, as shown in Figure 4(b). First, the spatial neighbor node information is aggregated through 1D spatial convolution. After data normalization and activation function ReLU, 1D temporal convolution is used to aggregate the temporal neighbor node features. Finally, the updated gait feature map is obtained through data normalization and activation function.

[0059] In order to verify the accuracy of this implementation, the following experiments were conducted:

[0060] We trained the proposed model end-to-end using the Adam optimizer, with an input data size of 32 per iteration and an initial learning rate of 0.1, decaying by a factor of 0.1 every 400 epochs. Training ended after 1200 epochs. The weight decay value was set to 0.0005. Each input data consisted of 56 consecutive frames of skeleton data. All experiments were conducted using the PyTorch deep learning framework and an RTX 2080 graphics card.

[0061] 1 Dataset

[0062] MMGS: Khokhlova et al. used 7-centimeter shoe soles in their data collection. They induced abnormal gait patterns by placing special insoles in the shoes ("liners") or by asking participants to not bend their right knee ("kneeling") while walking. Twenty-seven subjects participated in the study, walking five to seven times with each gait type. The MMGS database contains 489 videos, with eight subjects serving as test data and 19 subjects serving as training data.

[0063] 2 Ablation Experiment

[0064] Because we randomly sample 56 frames from each video, the input for each training run is actually different. We train each model 30 times and take the average of the 30 test results as the final result for each trained model. The average "avg" and standard deviation "stdev" of the 30 training results are shown in Tables 1, 2, and 3.

[0065] 2.1 Normalized Matrix

[0066] Table 1 compares three commonly used normalized adjacency matrices: a symmetric normalized adjacency matrix, the normalized adjacency matrix used in graph convolutional networks (GCNs), and a random walk normalized adjacency matrix. The receptive field of the spatial convolution is set to k = 3 (1-hop), and the receptive field of the temporal convolution is set to τ = 9. The random walk normalized matrix adopted in the proposed NG-CNN achieves the best performance. NG-CNN achieves a 3.63% improvement in average accuracy over models using the GCN normalized adjacency matrix and a 6.52% improvement over models using the symmetric normalized adjacency matrix. This demonstrates that the random walk normalized matrix is ​​more suitable for modeling the spatiotemporal geometric features of gait graphs.

[0067] Table 1 Model accuracy (%) with various normalized adjacency matrices

[0068]

[0069] 2.2 Spatiotemporal Stepped Convolutional Network

[0070] The NG-CNN spatiotemporal strided convolutional network module (Figure 4(b)) proposed in this example is verified with different spatial perception domains k and temporal perception domains τ as shown in Table 2. The model with a spatial perception domain k = 3 and a temporal perception domain τ = 7 achieves the best performance. Among all temporal receptive field models, the accuracy of the spatial perception domain k = 3 significantly outperforms the other models. For different spatial perception domains, a temporal perception domain τ = 7 achieves even better performance. These results confirm that smaller spatial and temporal perception domains can achieve better performance.

[0071] Table 2 Accuracy (%) of spatiotemporal strided convolutional NG-GCN blocks with different spatial perception domains (k) and temporal perception domains (τ)

[0072]

[0073]

[0074] 3 Comparative experiments

[0075] We compared our model with state-of-the-art abnormal gait recognition methods based on skeleton data on the MMGS dataset. As shown in Table 3, our method achieved an average precision of 92.92% on the MMGS dataset, an average precision improvement of 0.66% compared to AGS-GCN. It is worth noting that AGS-GCN constructs gait-link based on prior knowledge and uses an attention module to improve fine-grained gait feature modeling. However, the NG-CNN proposed in this example only constructs a gait graph and a neighbor node grouping strategy based on the connectivity of human joints, fully demonstrating the effectiveness of the generalized convolutional network proposed in this example in extracting spatiotemporal geometric features from gait graphs.

[0076] Table 3 Comparison with the most advanced MMGS methods

[0077]

[0078] Example 2

[0079] This embodiment provides a skeleton abnormal gait recognition system based on a generalized convolutional neural network.

[0080] A skeleton abnormal gait recognition system based on generalized convolutional neural network, comprising:

[0081] A graph construction module is configured to: construct a gait graph using lower limb joints as nodes and bones as edges;

[0082] a normalization module configured to group adjacent nodes in the gait graph according to distance and spatial relationship, and normalize the non-Euclidean structure of the gait graph into a Euclidean feature graph; wherein the positional relationship of the nodes encodes the connectivity between the joints;

[0083] A modeling module is configured to: use a generalized convolutional network to model the geometric features encoded in the normalized Euclidean feature map, generalize the ability of the convolution kernel to extract texture representation from the image to extract gait graph geometric features, and obtain a gait feature map that aggregates spatiotemporal neighbor node features;

[0084] The prediction module is configured to predict the abnormal gait classification result based on the gait feature map.

[0085] It should be noted that the examples and application scenarios implemented by the graph construction module, normalization module, modeling module, and prediction module described above are the same as those in the steps of Example 1, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0086] Example 3

[0087] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the method for identifying abnormal gait of a skeleton based on a generalized convolutional neural network as described in the first embodiment above are implemented.

[0088] Example 4

[0089] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for identifying abnormal gait based on a generalized convolutional neural network as described in the first embodiment are implemented.

[0090] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0091] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0094] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0095] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A skeleton abnormal gait recognition method based on generalized convolutional neural network, characterized in that: include: The gait graph is constructed with lower limb joints as nodes and bones as edges; Adjacent nodes in the gait graph are grouped according to distance and spatial relationship, and the non-Euclidean structure of the gait graph is normalized into a Euclidean feature graph, in which the positional relationship of the nodes encodes the connectivity between joints. The grouping of adjacent nodes in the gait graph according to the distance and spatial relationship specifically includes: dividing the neighboring nodes in the gait graph into a certain number of subsets according to the set spatial perception domain, so that the gait graph is divided into a certain number of subgraphs; The adjacency matrix of the gait graph is also divided into a certain number of sub-matrices; In graph convolution, the adjacency matrix is ​​used to perform weighted averaging of neighbor features. Neighbor nodes are grouped in the skeleton graph. The step-by-step aggregation rules of spatial graph convolution and temporal convolution are as follows: in, is a learnable parameter, is the feature map of layer l, is the normalized adjacency matrix, σ(g) is the activation function, and TCN is the temporal convolution; If it is a random walk matrix As the normalized adjacency matrix, the update strategy of node features is converted to: Rearrange the positions of the joints in the gait feature graph according to the formula, where D is the metric matrix of A; A generalized convolutional network is used to model the geometric features encoded in the normalized Euclidean feature map. The ability of the convolution kernel to extract texture representation from the image is generalized to extract the geometric features of the gait graph, resulting in a gait feature map that aggregates the features of spatiotemporal neighbor nodes. The generalization of the convolution kernel's ability to extract texture representations from images to extracting geometric features of gait graphs specifically includes: modeling the spatiotemporal dependencies of skeleton nodes by sliding the convolution kernel along the spatiotemporal synchronization dimension on the normalized gait graph, thereby generalizing the convolution kernel's ability to extract image texture features to the gait graph; According to the gait feature map, the abnormal gait classification results are predicted.

2. The method for identifying abnormal gait based on skeleton based on generalized convolutional neural network according to claim 1, characterized in that: The method of modeling the spatiotemporal dependencies of skeleton nodes by sliding the convolution kernel along the spatiotemporal synchronization dimension on the normalized gait graph specifically includes: constructing the spatial dependencies of skeleton nodes by sliding the spatial synchronization convolution kernel on the normalized gait graph, and constructing the temporal dependencies of skeleton nodes by sliding the temporal synchronization convolution kernel on the normalized gait graph.

3. The method for identifying abnormal gait based on skeleton based on generalized convolutional neural network according to claim 1, characterized in that: The neighbor node feature aggregation is equivalent to the graph convolution with random walk normalized adjacency matrix: Among them, TCN is time convolution, is a normalized l-layer gait feature map, and Spatial-CNN is a spatial convolutional network.

4. The method for identifying abnormal gait based on skeleton based on generalized convolutional neural network according to claim 1, characterized in that: The predicting of the abnormal gait classification result based on the gait feature map specifically includes: inputting the gait feature map into the global average pooling layer, the fully connected layer and the soft max function in sequence to obtain the abnormal gait classification result.

5. A skeleton abnormal gait recognition system based on generalized convolutional neural network, characterized in that: include: A graph construction module is configured to: construct a gait graph using lower limb joints as nodes and bones as edges; a normalization module configured to group adjacent nodes in the gait graph according to distance and spatial relationship, and normalize the non-Euclidean structure of the gait graph into a Euclidean feature graph; wherein the positional relationship of the nodes encodes the connectivity between the joints; The grouping of adjacent nodes in the gait graph according to the distance and spatial relationship specifically includes: dividing the neighboring nodes in the gait graph into a certain number of subsets according to the set spatial perception domain, so that the gait graph is divided into a certain number of subgraphs; The adjacency matrix of the gait graph is also divided into a certain number of sub-matrices; In graph convolution, the adjacency matrix is ​​used to perform weighted averaging of neighbor features. Neighbor nodes are grouped in the skeleton graph. The step-by-step aggregation rules of spatial graph convolution and temporal convolution are as follows: in, is a learnable parameter, is the feature map of layer l, is the normalized adjacency matrix, σ(g) is the activation function, and TCN is the temporal convolution; If it is a random walk matrix As the normalized adjacency matrix, the update strategy of node features is converted to: Rearrange the positions of the joints in the gait feature graph according to the formula, where D is the metric matrix of A; A modeling module is configured to: use a generalized convolutional network to model the geometric features encoded in the normalized Euclidean feature map, generalize the ability of the convolution kernel to extract texture representation from the image to extract gait graph geometric features, and obtain a gait feature map that aggregates spatiotemporal neighbor node features; The generalization of the convolution kernel's ability to extract texture representations from images to extracting geometric features of gait graphs specifically includes: modeling the spatiotemporal dependencies of skeleton nodes by sliding the convolution kernel along the spatiotemporal synchronization dimension on the normalized gait graph, thereby generalizing the convolution kernel's ability to extract image texture features to the gait graph; The prediction module is configured to predict the abnormal gait classification result based on the gait feature map.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for identifying abnormal gait of a skeleton based on a generalized convolutional neural network are implemented as described in any one of claims 1 to 4.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the method for identifying abnormal gait of skeleton based on a generalized convolutional neural network are implemented as described in any one of claims 1 to 4.

Images