Resource flexibility skeleton action classification method based on adaptive structure optimization

Abstract

The present invention provides a resource-flexible skeleton motion classification method based on adaptive structural optimization, comprising the following steps: collecting skeleton video samples to be trained and preprocessing them to construct a qualified dataset; using the qualified dataset as the basic input of an IC-GC spatial domain model to obtain the unique features of each joint and the interaction information between different joints; using the output of the IC-GC spatial domain model as the input of an IC-TC temporal domain model to obtain common features and salient features; introducing a Skip-Trans jump link mechanism to achieve information transfer between different levels of the model and retain key information in deep iteration processes; and designing two model variants, IC-GCN-Plus and IC-GCN-Light, to implement skeleton motion classification based on different hardware environments and efficiency requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of human motion recognition and classification, and in particular to a resource flexibility skeleton motion classification method based on adaptive structure optimization. Background Art

[0002] Skeleton-based behavior recognition technology is primarily used in intelligent surveillance systems, human-computer interaction, virtual reality, and robotics. This technology identifies and understands human behavior by analyzing the motion trajectory and posture of the human skeleton. It offers advantages such as low data volume, high semantics, and immunity to background interference. Furthermore, compared to other modalities such as RGB or depth image sequences, skeleton-based behavior recognition demonstrates greater robustness against complex backgrounds, changes in body scale, perspective, and speed. With the advancement of deep learning technology, skeleton-based behavior recognition has become a research hotspot due to its efficient feature extraction and generalization capabilities.

[0003] Skeleton data consists of a time series of joint positions, depicting human motion and structure. In skeletal action recognition, the human skeleton is viewed as a graph, with joints as nodes and their relative positional relationships defining edges. Graph Convolutional Networks (GCNs) are well-suited to this type of graph data, directly learning the connection patterns between joints to extract identifying features. The advantage of GCNs lies in their ability to perform convolution on graph data, effectively processing irregular skeleton data and capturing the spatial relationships and temporal dynamics of joints, providing features for action recognition.

[0004] The invention with publication number CN116343338A proposes an optimized skeleton action recognition technology that focuses on the comprehensive extraction of spatiotemporal features. This method realizes the hierarchical extraction of spatiotemporal features by analyzing body parts and time series. However, this method may be limited by sensor accuracy and posture prediction algorithms when processing skeletal data, thereby introducing noise and recognition difficulties. The patent with publication number CN114170683A demonstrates an action recognition technology based on a multimodal attention fusion network, which extracts RGB and skeletal features through ResNeXt101 and Shift-GCN networks and fuses them. Although this method combines RGB and skeleton data to improve recognition results, it relies on a large amount of labeled data, has large RGB data storage requirements, and is sensitive to environmental changes.

[0005] Therefore, although GCN-based research has made significant progress in skeleton spatiotemporal modeling, there are still some challenges and limitations in terms of the static nature of the adjacency matrix, the balance between universality and specificity of feature extraction, computational complexity and real-time performance, and the ability to process diverse datasets. Summary of the Invention

[0006] In response to the technical issues raised above, a resource-flexible skeleton action classification method based on adaptive structural optimization is provided. The present invention introduces the IC-GC spatial domain model and the IC-TC temporal domain model to respectively extract key features and generate robust topological matrices, capturing the key dynamic changes of actions in the temporal dimension; introduces the Skip-Trans jump link mechanism to enhance the retention of key information in deep iterations; and designs two model variants, IC-GCN-Plus and IC-GCN-Light, for different hardware environments to achieve skeleton-based action classification.

[0007] The technical means adopted in the present invention are as follows:

[0008] A resource flexibility skeleton action classification method based on adaptive structural optimization, comprising:

[0009] Collect skeleton video samples to be trained, perform preprocessing, and construct a qualified dataset;

[0010] The qualified dataset is used as the basic input to the IC-GC spatial domain model to obtain the unique features of each joint and the interaction information between different joints;

[0011] The output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model to obtain common features and salient features;

[0012] Introducing the Skip-Trans jump link mechanism to achieve information transfer between different layers of the model and retain key information in the deep iteration process;

[0013] In response to different hardware environments and efficiency requirements, two model variants, IC-GCN-Plus and IC-GCN-Light, are designed to achieve skeleton action classification.

[0014] Furthermore, constructing a qualified data set includes:

[0015] Collect video samples containing human skeleton movements and preprocess the video data. The preprocessing includes: obtaining skeleton data, eliminating noise and normalizing the viewing angle, and storing the preprocessed video data in a qualified data set.

[0016] Furthermore, the qualified dataset is used as the basic input to the IC-GC spatial domain model to obtain the unique features of each joint and the common features between different joints, specifically including:

[0017] The IC-GC spatial domain model includes a spatial individuality stream S-IS and a spatial commonality stream S-CS. The spatial individuality stream S-IS is used to extract the unique features of a specific joint, and the spatial commonality stream S-CS is used to capture the interaction information between different joints.

[0018] The IC-GC spatial domain model is expressed as:

[0019] X out =CFC(U(S c (A 0,1 ,X in ),A p ),E(X in ))

[0020] Among them, X in represents the input features, X out represents the output feature, A 0,1 Represents the original one-hot adjacency matrix, A p represents the fully parameterized adjacency matrix, S c represents the spatial individuality stream S-IS and spatial commonality stream S-CS combination module, U represents the self-attention adjacency matrix module, E represents the feature conversion module, and CFC represents the channel feature fusion module;

[0021] The feature extraction process of spatial individuality flow and spatial commonality flow is as follows:

[0022] V s =S c (A 0,1 ,X in )=[I s (X in )||C s (A 0,1 ,X in )] T W

[0023]

[0024] in, represents the space prototype vector, represents the original one-hot adjacency matrix, || represents the connection operation, P avg represents adaptive average pooling, θ(·) and δ(·) represent activation functions, I s (·) represents the input X in Transpose and perform dimensionality reduction, C s (·) represents the combination of the reduced dimension features and A 0,1 Multiply to complete the first feature fusion, connect the corresponding channels, and then use 1×1 convolution to fuse the channels, and finally get the prototype vector of the next part;

[0025] Generate N×N self-attention adjacency matrix A k :

[0026]

[0027] in, represents matrix multiplication, R e represents vector replication, represents the fully parameterized adjacency matrix;

[0028] The process of feature learning and updating is expressed as:

[0029] X m =E(X in )=X in W

[0030] Where W represents a size of C in ×C out The real matrix, C in and C out Respectively represent the number of input and output channels;

[0031] Combined with A k and X m To obtain the final output of channel feature fusion, it is expressed as:

[0032]

[0033] in, Represents the output of the spatial domain model, A k Represents the self-attention adjacency matrix, CFC(A k ,X m ) represents the adjacency matrix A k and feature X m ; each and Corresponding to X m and A k The ith channel of is combined by matrix multiplication, and all channels are combined using the connection operation to obtain the output X out .

[0034] Furthermore, the output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model to obtain common features and salient features, specifically including:

[0035] The IC-TC time domain model includes a time individuality stream T-IS and a time commonality stream T-CS. The time individuality stream T-IS is used to maintain intra-frame information, and the time commonality stream T-CS is used for inter-frame interaction and processing.

[0036] The output X of the IC-GC spatial domain model out As the input Y of IC-TC time domain model in , then the output Y of the IC-TC time domain model is out for:

[0037]

[0038] Among them, C t (·) represents the time commonality flow, I t (·) indicates the flow of time personality;

[0039] In the temporal personality stream T-IS, the input features are adaptively averaged pooled P avg To extract common features, adaptive maximum pooling P max To extract salient features:

[0040]

[0041] in, θ(·),δ(·) represent activation functions;

[0042] A multi-scale temporal convolution strategy is adopted in the temporal commonality stream T-CS to capture features at different time scales and complete complex time series analysis tasks.

[0043] Furthermore, the Skip-Trans jump link mechanism introduces skip residual connections and adopts an adaptive pyramid structure, which enables the network to transmit and fuse features at different levels, establishes a direct information transmission path between the shallow and deep layers of the network, and enables the shallow feature map to directly act on the deep layer, realizing seamless transmission of shallow features to the deep network.

[0044] Furthermore, the IC-GCN-Plus model is used to improve recognition performance for large-scale and complex datasets, addressing the problem of numerous action categories and diverse action variations. The IC-GCN-Plus model enhances recognition performance by increasing the depth, width, and connection diversity of the network and combining it with graph convolution operations.

[0045] The IC-GCN-Plus model is an extension of the graph convolutional network. In terms of optimizing the processing performance of large-scale datasets, it covers complex adjacency matrix construction, multi-scale feature fusion methods, and more in-depth network architecture design.

[0046] Furthermore, the IC-GCN-Light model is used to implement skeleton behavior recognition on devices with limited computing resources while maintaining high recognition accuracy. The IC-GCN-Light model achieves model lightweighting by simplifying the model structure and reducing the number of parameters while retaining key functions.

[0047] The lightweight design of the IC-GCN-Light model includes: adopting a shallower network architecture, reducing the number of channels, and implementing efficient graph convolution operations and feature fusion strategies.

[0048] The present invention also includes a storage medium, which includes a stored program, wherein when the program is run, the resource flexibility skeleton action classification method based on adaptive structure optimization is executed.

[0049] The present invention also includes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the resource flexibility skeleton action classification method based on adaptive structural optimization through the computer program.

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] The resource-flexible skeleton action classification method based on adaptive structural optimization provided by the present invention realizes efficient feature updating by constructing adaptive individual and common structures. The individual and common graph convolution modules respectively extract key features and generate robust topological matrices, and the individual and common time convolution modules capture the key dynamic changes of actions in the time dimension. The jump connection module enhances the retention of key information in deep iterations, while the Plus and Light variants adapt to different hardware environments and are suitable for large-scale data sets and resource-constrained environments, respectively. This adaptive structural optimization design reduces the computational complexity and improves the efficiency of the model while ensuring the accuracy of the model, making it more suitable for practical application scenarios.

[0052] The resource-flexible skeleton action classification method based on adaptive structural optimization, proposed in this paper, introduces a temporal convolution module combining individual and common features to capture key dynamic changes in actions in the temporal dimension and integrate them with spatial features. This overcomes the shortcomings of existing technologies that focus solely on static features or process only time series, achieving comprehensive extraction of action spatiotemporal features and improving the ability to capture dynamic changes.

[0053] The resource-flexible skeleton action classification method based on adaptive structural optimization provided by the present invention designs a skip connection module. By introducing skip residual connections, it realizes the seamless transmission of shallow features to deep networks, ensuring the retention of key information. This design not only helps to maintain the original information of the features, but also effectively alleviates the gradient vanishing phenomenon and improves the training efficiency of the network. Compared with traditional residual connections, Skip-Trans adopts an adaptive pyramid structure, which is more suitable for skeleton behavior recognition tasks, reduces the loss of information during network propagation, and enhances the expressiveness of the network. Thus, the skip connection module effectively improves the classification performance of the model.

[0054] The resource-flexible skeleton action classification method based on adaptive structural optimization provided by the present invention designs two variants, IC-GCN-Plus and IC-GCN-Light, which are suitable for large-scale data sets and resource-constrained environments, respectively. IC-GCN-Plus improves the complexity and recognition ability of the model by increasing the number of channels and the number of adjacency matrices, and is suitable for large-scale and complex data sets; while IC-GCN-Light reduces the computational complexity of the model by reducing the number of channels and the number of adjacency matrices, and is suitable for resource-constrained environments and real-time application scenarios. This diversified model variant design enables the present invention to select the appropriate model according to different application requirements, thereby better meeting the needs of actual application scenarios.

[0055] In summary, the present invention not only achieves significant advantages in high-precision motion classification performance, but also demonstrates outstanding potential in terms of flexibility and robustness in practical applications. This gives the invention a significant competitive advantage in various fields requiring efficient motion classification, such as human-computer interaction, motion analysis, intelligent monitoring, virtual reality, and robotics. The design of the present invention can effectively address the needs of different data sets and tasks, while taking into account computational efficiency and resource consumption, providing a flexible solution for various application scenarios.

[0056] Based on the above reasons, the present invention can be widely promoted in the technical field of human action recognition and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0058] Figure 1 This is a flow chart of the resource flexibility skeleton action classification method based on adaptive structure optimization of the present invention.

[0059] Figure 2 This is the framework diagram of the IC-GC spatial domain model of the present invention.

[0060] Figure 3 This is the framework diagram of the IC-TC time domain model of the present invention.

[0061] Figure 4 This is the overall structure diagram of Skip-Trans and the network in the present invention.

[0062] Figure 5 Schematic diagram of parameter design in an embodiment of the present invention.

[0063] Figure 6 This is the visualization and skeleton heat map in the embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0065] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0066] like Figure 1 As shown, the present invention provides a resource flexibility skeleton action classification method based on adaptive structure optimization, comprising:

[0067] Collect skeleton video samples to be trained, perform preprocessing, and construct a qualified dataset;

[0068] In specific implementation, as a preferred embodiment of the present invention, the construction of a qualified data set includes:

[0069] Collect video samples containing human skeleton movements and preprocess the video data. The preprocessing includes: obtaining skeleton data, eliminating noise and normalizing the viewing angle, and storing the preprocessed video data in a qualified data set.

[0070] The qualified dataset is used as the basis for inputting the IC-GC spatial domain model, such as Figure 2 As shown, the unique features of each joint and the interaction information between different joints are obtained;

[0071] In specific implementation, as a preferred embodiment of the present invention, the qualified data set is used as the basic input IC-GC spatial domain model to obtain the unique features of each joint and the common features between different joints, specifically including:

[0072] The IC-GC spatial domain model includes a spatial individuality stream S-IS and a spatial commonality stream S-CS. The spatial individuality stream S-IS is used to extract the unique features of specific joints, and the spatial commonality stream S-CS is used to capture the interaction information between different joints. During implementation, the motivation is to capture the specificity and co-occurrence information between different joints and frames in skeleton data in real time, thereby improving the model's ability to recognize action details and adapt to dynamic changes.

[0073] The IC-GC spatial domain model is expressed as:

[0074] X out =CFC(U(S c (A 0,1 ,X in ),A p ),E(X in ))

[0075] Among them, X in represents the input features, X out represents the output feature, A 0,1 Represents the original one-hot adjacency matrix, A p represents the fully parameterized adjacency matrix, S c represents the spatial individuality stream S-IS and spatial commonality stream S-CS combination module, U represents the self-attention adjacency matrix module, E represents the feature conversion module, and CFC represents the channel feature fusion module;

[0076] The feature extraction process of spatial individuality flow and spatial commonality flow is as follows:

[0077] V s =S c (A 0,1 ,X in )=[I s (X in )||C s (A 0,1 ,X in )] T W

[0078]

[0079] in, represents the space prototype vector, represents the original one-hot adjacency matrix, || represents the connection operation, P avg represents adaptive average pooling, θ(·) and δ(·) represent activation functions, I s (·) represents the input X in Transpose and perform dimensionality reduction, C s(·) represents the combination of the reduced dimension features and A 0,1 Multiply to complete the first feature fusion, connect the corresponding channels, and then use 1×1 convolution to fuse the channels, and finally get the prototype vector of the next part;

[0080] Generate N×N self-attention adjacency matrix A k :

[0081]

[0082] in, represents matrix multiplication, R e represents vector replication, represents the fully parameterized adjacency matrix;

[0083] The process of feature learning and updating is expressed as:

[0084] X m =E(X in )=X in W

[0085] Where W represents a size of C in ×C out The real matrix, C in and C out Respectively represent the number of input and output channels;

[0086] Combined with A k and X m To obtain the final output of channel feature fusion, it is expressed as:

[0087]

[0088] in, Represents the final output of the spatial domain model, A k Represents the self-attention adjacency matrix, CFC(A k ,X m ) represents the adjacency matrix A k and feature X m ; each and Corresponding to X m and A k The ith channel of is combined by matrix multiplication, and all channels are combined using the connection operation to obtain the output X out .

[0089] The output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model, as follows: Figure 3 As shown, common features and prominent features are obtained;

[0090] In specific implementation, as a preferred embodiment of the present invention, the output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model to obtain common features and prominent features, which specifically include:

[0091] The IC-TC temporal domain model includes a temporal individual stream T-IS and a temporal common stream T-CS. The temporal individual stream T-IS is used to maintain intra-frame information, and the temporal common stream T-CS is used for inter-frame interaction and processing. During implementation, the design motivation is to effectively extract the temporal attributes of the skeleton sequence through temporal convolution processing and maintain high sensitivity to key frames to achieve accurate recognition of action details.

[0092] The output X of the IC-GC spatial domain model out As the input Y of IC-TC time domain model in , then the output Y of the IC-TC time domain model is out for:

[0093]

[0094] Among them, C t (·) represents the time commonality flow, I t (·) indicates the flow of time personality;

[0095] In the temporal personality stream T-IS, the input features are adaptively averaged pooled P avg To extract common features, adaptive maximum pooling P max To extract salient features:

[0096]

[0097] in, θ(·),δ(·) represent activation functions;

[0098] A multi-scale temporal convolution strategy is adopted in the temporal commonality stream T-CS to effectively capture features at different time scales, enhance the model's ability to understand and represent time series data, and thus achieve better performance in complex time series analysis tasks.

[0099] In implementation, individual and common convolution modes based on unified channel separation are achieved through IC-TC and IC-GC.

[0100] The output of IC-TC and IC-GC is introduced into the Skip-Trans jump link mechanism, such as Figure 4 As shown in Figure 3, information transfer between different layers of the model is realized, and key information in the deep iteration process is retained.

[0101] During implementation, the Skip-Trans jump link mechanism realizes the effective fusion of information between different layers of the network. By establishing a direct channel between the shallow and deep layers of the network, the shallow feature map can directly affect the deep layer, thereby improving the network's perception sensitivity to the input data.

[0102] In specific implementation, as a preferred embodiment of the present invention, the Skip-Trans jump link mechanism, by introducing a skip residual connection and adopting an adaptive pyramid structure, enables the network to transmit and fuse features at different levels, establishes a direct information transmission path between the shallow and deep layers of the network, enables the shallow feature map to directly act on the deep layer, and realizes the seamless transmission of shallow features to the deep network.

[0103] During implementation, the Skip-Trans mechanism promotes the effective integration of information between various layers of the network, which not only helps to maintain the original information of the features, but also effectively alleviates the gradient vanishing phenomenon and improves the training efficiency of the network. Compared with traditional residual connections, Skip-Trans adopts an adaptive pyramid structure, which enables the network to transmit and fuse features at different levels. At the same time, considering the characteristics of skeleton data, it is more suitable for skeleton behavior recognition tasks, reduces the loss of information during network propagation, and enhances the expressiveness of the network. In addition, the design of Skip-Trans is simple and efficient, and does not significantly increase the computational cost, which is particularly critical for real-time applications and devices with limited computing resources.

[0104] In response to different hardware environments and efficiency requirements, two model variants, IC-GCN-Plus and IC-GCN-Light, are designed to achieve skeleton action classification.

[0105] In specific implementation, as a preferred embodiment of the present invention, the IC-GCN-Plus model is used to improve recognition performance for large-scale and complex data sets, solving the problem of numerous action categories and diverse action variations. The IC-GCN-Plus model enhances recognition performance by increasing the depth, width, and connection diversity of the network and combining graph convolution operations.

[0106] The IC-GCN-Plus model is an extension of the graph convolutional network. In terms of optimizing the processing performance of large-scale datasets, it covers complex adjacency matrix construction, multi-scale feature fusion methods, and more in-depth network architecture design.

[0107] When implemented, the significant advantages of the IC-GCN-Plus model are reflected in its excellent feature extraction capabilities and high adaptability, which can accurately recognize complex motion patterns and meet the needs of different data sets and tasks.

[0108] In specific implementation, as a preferred embodiment of the present invention, the IC-GCN-Light model is used to implement skeleton behavior recognition on devices with limited computing resources while maintaining high recognition accuracy. The IC-GCN-Light model achieves lightweight modeling by simplifying the model structure and reducing the number of parameters while retaining key functions.

[0109] The lightweight design of the IC-GCN-Light model includes: adopting a shallower network architecture, reducing the number of channels, and implementing efficient graph convolution operations and feature fusion strategies.

[0110] During implementation, the core advantages of the IC-GCN-Light model are reflected in its efficient computing performance and low resource allocation requirements, which can significantly reduce the computational and storage burden of the model while maintaining reasonable accuracy.

[0111] The present invention also includes a storage medium, which includes a stored program, wherein when the program is run, the resource flexibility skeleton action classification method based on adaptive structure optimization is executed.

[0112] The present invention also includes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the resource flexibility skeleton action classification method based on adaptive structural optimization through the computer program.

[0113] Example

[0114] like Figure 1 As shown, the present invention provides a resource flexibility skeleton action classification method based on adaptive structure optimization. The operating system used in the experiment is Ubuntu 22.04.1, and the deep learning framework adopted is PyTorch. The specific configurations involved in the experiment are shown in Table 1.

[0115] Table 1 Experimental configuration table

[0116] project Configuration Processor (CPU) 12th Gen Intel(R)Core(TM)i9-12900KF cpu@3.20GHz Graphics card (GPU) GeForce RTX 3090×2 operating system Ubuntu 22.04.1 frame PyTorch

[0117] Table 2 demonstrates the effectiveness of the innovative submodules proposed in this paper through extensive ablation experiments. Tables 3, 4, and 5 demonstrate the performance advantages of this paper on four different datasets through comparative experiments with existing GCN-based and Transformer-based methods. Figure 5 The parameter design is shown. Figure 6 Visualization and skeleton heatmaps are shown, which means that the IC-GCN model achieves better results in action classification.

[0118] Table 2 Ablation experiment table of individual flow, common flow, and jump module on IC-GCN-Basic model

[0119] method Accuracy (%) Param. FLOPs Baseline(Spatial) 93.95 3.45M <![CDATA[ ～ 3.98G]]> Baseline+Skip-Block 94.58↑0.63 3.49M <![CDATA[ ～ 4.02G]]> Baseline+S-IS 94.58↑0.63 3.12M <![CDATA[ ～ 3.57G]]> Baseline+S-CS 94.62↑0.67 3.12M <![CDATA[ ～ 3.57G]]> Baseline+S-ICS 94.78↑0.83 3.12M <![CDATA[ ～ 3.57G]]> Baseline+S-ICS+Skip-Block 95.38↑1.43 3.17M <![CDATA[ ～ 3.61G]]> Baseline+T-CS 94.76↑0.81 1.52M <![CDATA[ ～ 1.92G]]> Baseline+T-ICS 94.98↑1.03 1.52M <![CDATA[ ～ 1.92G]]> Baseline+T-ICS+Skip-Block 95.00↑1.05 1.57M <![CDATA[ ～ 1.96G]]> Baseline+S-ICS+T-ICS 95.07↑1.12 1.21M <![CDATA[ ～ 1.51G]]> Baseline+S-ICS+T-ICS+Skip-Block 95.40↑1.45 1.25M <![CDATA[ ～ 1.55G]]>

[0120] Table 3 Comparative experiments of IC-GCN on NTU dataset

[0121]

[0122]

[0123] Table 4 Comparative experiments of IC-GCN on the NW-UCLA dataset

[0124]

[0125] Table 5 Comparative experiments of IC-GCN on the Kinetics-400 dataset

[0126]

[0127]

[0128] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0129] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0131] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0132] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A resource flexibility skeleton action classification method based on adaptive structural optimization, characterized by: include: Collect skeleton video samples to be trained, perform preprocessing, and construct a qualified dataset; The qualified dataset is used as the basic input to the IC-GC spatial domain model to obtain the unique features of each joint and the interaction information between different joints; The qualified dataset is used as the basic input to the IC-GC spatial domain model to obtain the unique features of each joint and the common features between different joints, specifically including: The IC-GC spatial domain model includes a spatial individuality stream S-IS and a spatial commonality stream S-CS. The spatial individuality stream S-IS is used to extract the unique features of a specific joint, and the spatial commonality stream S-CS is used to capture the interaction information between different joints. The IC-GC spatial domain model is expressed as: X out =CFC(U(S c (A 0,1 ,X in ),A p ),E(X in )) Among them, X in represents the input features, X out represents the output feature, A 0,1 Represents the original one-hot adjacency matrix, A p represents the fully parameterized adjacency matrix, S c represents the spatial individuality stream S-IS and spatial commonality stream S-CS combination module, U represents the self-attention adjacency matrix module, E represents the feature conversion module, and CFC represents the channel feature fusion module; The feature extraction process of spatial individuality flow and spatial commonality flow is as follows: V s =S c (A 0,1 ,X in )=[I s (X in )||C s (A 0,1 ,X in )] T W in, represents the space prototype vector, represents the original one-hot adjacency matrix, || represents the connection operation, P avg represents adaptive average pooling, θ(·) and δ(·) represent activation functions, I s (·) represents the input X in Transpose and perform dimensionality reduction, C s (·) represents the combination of the reduced dimension features and A 0,1 Multiply to complete the first feature fusion, connect the corresponding channels, and then use 1×1 convolution to fuse the channels, and finally get the prototype vector of the next part; Generate N×N self-attention adjacency matrix A k : in, represents matrix multiplication, R e represents vector replication, represents the fully parameterized adjacency matrix; The process of feature learning and updating is expressed as: X m =E(X in )=X in W Where W represents a size of C in ×C out The real matrix, C in and C out Respectively represent the number of input and output channels; Combined with A k and X m To obtain the final output of channel feature fusion, it is expressed as: in, Represents the output of the spatial domain model, A k Represents the self-attention adjacency matrix, CFC(A k ,X m ) represents the adjacency matrix A k and feature X m ; each and Corresponding to X m and A k The ith channel of is combined by matrix multiplication, and all channels are combined using the connection operation to obtain the output X out ; The output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model to obtain common features and salient features; Introducing the Skip-Trans jump link mechanism to achieve information transfer between different layers of the model and retain key information in the deep iteration process; In response to different hardware environments and efficiency requirements, two model variants, IC-GCN-Plus and IC-GCN-Light, are designed to achieve skeleton action classification.

2. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The constructing of a qualified data set includes: Collect video samples containing human skeleton movements and preprocess the video data. The preprocessing includes: obtaining skeleton data, eliminating noise and normalizing the viewing angle, and storing the preprocessed video data in a qualified data set.

3. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model to obtain common features and salient features, specifically including: The IC-TC time domain model includes a time individuality stream T-IS and a time commonality stream T-CS. The time individuality stream T-IS is used to maintain intra-frame information, and the time commonality stream T-CS is used for inter-frame interaction and processing. The output X of the IC-GC spatial domain model out As the input Y of IC-TC time domain model in , then the output Y of the IC-TC time domain model is out for: Among them, C t (·) represents the time commonality flow, I t (·) indicates the flow of time personality; In the temporal personality stream T-IS, the input features are adaptively averaged pooled P avg To extract common features, adaptive maximum pooling P max To extract salient features: in, θ(·),δ(·) represent activation functions; A multi-scale temporal convolution strategy is adopted in the temporal commonality stream T-CS to capture features at different time scales and complete complex time series analysis tasks.

4. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The Skip-Trans jump link mechanism introduces skip residual connections and adopts an adaptive pyramid structure, which enables the network to transmit and fuse features at different levels, establishes a direct information transmission path between the shallow and deep layers of the network, and enables the shallow feature map to directly act on the deep layer, realizing seamless transmission of shallow features to the deep network.

5. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The IC-GCN-Plus model is designed to improve recognition performance for large-scale and complex datasets, addressing the challenges of numerous action categories and diverse action variations. The IC-GCN-Plus model enhances recognition performance by increasing the depth, width, and connectivity of the network, combined with graph convolution operations. The IC-GCN-Plus model is an extension of the graph convolutional network. In terms of optimizing the processing performance of large-scale datasets, it covers complex adjacency matrix construction, multi-scale feature fusion methods, and more in-depth network architecture design.

6. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The IC-GCN-Light model is used to implement skeleton behavior recognition on devices with limited computing resources while maintaining high recognition accuracy. The IC-GCN-Light model achieves model lightweighting by simplifying the model structure and reducing the number of parameters while retaining key functions. The lightweight design of the IC-GCN-Light model includes: adopting a shallower network architecture, reducing the number of channels, and implementing efficient graph convolution operations and feature fusion strategies.

7. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the method according to any one of claims 1 to 6 is executed.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the method according to any one of claims 1 to 6 by running the computer program.

Images