Resource flexibility skeleton action classification method based on adaptive structure optimization

Abstract

The invention provides a resource flexibility skeleton action classification method based on adaptive structure optimization, and the method comprises the steps: collecting a to-be-trained skeleton video sample, carrying out the preprocessing, and constructing a qualified data set; inputting the qualified data set as a basis into the IC-GC spatial domain model to obtain unique features of each joint and interaction information among different joints; taking the output of the IC-GC spatial domain model as the input of an IC-TC time domain model to obtain universal features and prominent features; a Skip-Trans jump link mechanism is introduced, so that information transmission among different levels of the model is realized, and key information in a deep iteration process is reserved; according to different hardware environments and efficiency requirements, two model variants of IC-GCN-Plus and IC-GCN-Light are designed, and skeleton action classification is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of human motion recognition and classification, and in particular to a resource flexibility skeleton motion classification method based on adaptive structure optimization. Background Art

[0002] Skeleton behavior recognition technology is mainly used in intelligent monitoring systems, human-computer interaction, virtual reality, and robotics. This technology recognizes and understands human behavior by analyzing the motion trajectory and posture of the human skeleton. It has the advantages of small data volume, high semantics, and no interference from background information. In addition, compared with other modalities such as RGB image sequences or depth image sequences, skeleton behavior recognition technology shows stronger robustness when facing complex backgrounds, changes in human scale, changes in perspective, and changes in movement speed. With the development of deep learning technology, skeleton-based behavior recognition technology has gradually become a research hotspot due to its efficient feature extraction and generalization capabilities.

[0003] Skeleton data consists of a time series of joint positions, depicting human motion and structure. In skeleton behavior recognition, the human skeleton is viewed as a graph, with joints as nodes and relative position relationships defining edges. Graph convolutional networks (GCNs) are suitable for this type of graph data, and can directly learn the connection patterns between joints to extract recognition features. The advantage of GCNs is that they can convolve graph data, effectively process irregular skeleton data, capture the spatial relationships and temporal dynamics of joints, and provide features for behavior recognition.

[0004] The invention with publication number CN116343338A proposes an optimized skeleton action recognition technology that focuses on the comprehensive extraction of spatiotemporal features. This method realizes the hierarchical extraction of spatiotemporal features by analyzing body parts and time series. However, this method may be limited by sensor accuracy and posture prediction algorithms when processing skeleton data, thereby introducing noise and recognition difficulties. The patent with publication number CN114170683A demonstrates behavior recognition technology based on a multimodal attention fusion network, which extracts RGB and skeleton features through ResNeXt101 and Shift-GCN networks and fuses them. Although this method combines RGB and skeleton data to improve recognition, it relies on a large amount of labeled data, has large storage requirements for RGB data, and is sensitive to environmental changes.

[0005] Therefore, although GCN-based research has made significant progress in skeleton spatiotemporal modeling, there are still some challenges and limitations in terms of the static nature of the adjacency matrix, the balance between the generality and specificity of feature extraction, computational complexity and real-time performance, and the ability to handle differential data sets. Summary of the invention

[0006] According to the technical problems raised above, a resource flexibility skeleton action classification method based on adaptive structural optimization is provided. The present invention introduces the IC-GC spatial domain model and the IC-TC temporal domain model to respectively extract key features and generate robust topological matrices to capture key dynamic changes of actions in the time dimension; introduces the Skip-Trans jump link mechanism to enhance the retention of key information in deep iterations; designs two model variants, IC-GCN-Plus and IC-GCN-Light, for different hardware environments to achieve skeleton-based action classification.

[0007] The technical means adopted by the present invention are as follows:

[0008] A resource flexibility skeleton action classification method based on adaptive structural optimization, comprising:

[0009] Collect skeleton video samples to be trained, perform preprocessing, and construct a qualified data set;

[0010] The qualified dataset is used as the basic input to the IC-GC spatial domain model to obtain the unique features of each joint and the interaction information between different joints;

[0011] The output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model to obtain common features and salient features;

[0012] Introducing the Skip-Trans jump link mechanism to achieve information transfer between different levels of the model and retain key information in the deep iteration process;

[0013] Aiming at different hardware environments and efficiency requirements, two model variants, IC-GCN-Plus and IC-GCN-Light, are designed to realize skeleton action classification.

[0014] Furthermore, constructing a qualified data set includes:

[0015] Collect video samples containing human skeleton movements and preprocess the video data, wherein the preprocessing includes: obtaining skeleton data, eliminating noise, and normalizing the viewing angle, and storing the preprocessed video data in a qualified data set.

[0016] Furthermore, the qualified data set is used as the basic input into the IC-GC spatial domain model to obtain the unique features of each joint and the common features between different joints, specifically including:

[0017] The IC-GC spatial domain model includes a spatial individual stream S-IS and a spatial common stream S-CS, wherein the spatial individual stream S-IS is used to extract unique features of a specific joint, and the spatial common stream S-CS is used to capture interactive information between different joints;

[0018] The IC-GC spatial domain model is expressed as:

[0019] X out =CFC(U(S c (A 0,1 ,X in ),A p ),E(X in ))

[0020] Among them, X in represents the input features, X out represents the output feature, A 0,1 represents the original one-hot adjacency matrix, A p represents the fully parameterized adjacency matrix, S c represents the combined module of spatial individual stream S-IS and spatial common stream S-CS, U represents the self-attention adjacency matrix module, E represents the feature conversion module, and CFC represents the channel feature fusion module;

[0021] The feature extraction process of spatial individuality flow and spatial commonality flow is as follows:

[0022] V s =S c (A 0,1 ,X in )=[I s (X in )||C s (A 0,1 ,X in )] T W

[0023]

[0024] in, represents the space prototype vector, represents the original one-hot adjacency matrix, || represents the connection operation, P avg represents adaptive average pooling, θ(·) and δ(·) represent activation functions, and I s (·) indicates that the input X in Transpose and perform dimensionality reduction, C s (·) represents the feature after dimension reduction and A 0,1 Multiply to complete the first feature fusion, connect the corresponding channels, and then use 1×1 convolution to fuse the channels, and finally get the prototype vector of the next part;

[0025] Generate an N×N self-attention adjacency matrix A k :

[0026]

[0027] in, represents matrix multiplication, R e represents vector replication, represents a fully parameterized adjacency matrix;

[0028] The process of feature learning and updating is expressed as:

[0029] X m =E(X in )=X in W

[0030] Where W represents a size of C in ×C out The real matrix, C in and C out Respectively represent the number of input and output channels;

[0031] Combined with A k and X m To obtain the final output of channel feature fusion, it is expressed as:

[0032]

[0033] in, represents the output of the spatial domain model, A k represents the self-attention adjacency matrix, CFC(A k ,X m ) represents the combination of adjacency matrix A k and feature X m Each and Corresponding to X m and A k The ith channel of is combined by matrix multiplication, and all channels are combined by concatenation to obtain the output X out .

[0034] Furthermore, the output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model to obtain common features and salient features, which specifically include:

[0035] The IC-TC time domain model includes a time individuality stream T-IS and a time commonality stream T-CS, wherein the time individuality stream T-IS is used to maintain intra-frame information, and the time commonality stream T-CS is used to perform inter-frame interaction and processing;

[0036] The output X of the IC-GC spatial domain model out As the input Y of IC-TC time domain model in , then the output Y of the IC-TC time domain model is out for:

[0037]

[0038] Among them, C t (·) represents the time commonality flow, I t (·) indicates the flow of time personality;

[0039] In the temporal personality stream T-IS, the input features are adaptively averaged pooled P avg To extract common features, adaptive maximum pooling P is used max To extract salient features:

[0040]

[0041] in, θ(·),δ(·) represent activation functions;

[0042] In the temporal commonality stream T-CS, a multi-scale temporal convolution strategy is adopted to capture features at different time scales and complete complex time series analysis tasks.

[0043] Furthermore, the Skip-Trans jump link mechanism, by introducing skip residual connections and adopting an adaptive pyramid structure, enables the network to transmit and fuse features at different levels, establishes a direct information transmission path between the shallow and deep layers of the network, enables the shallow feature map to directly act on the deep layer, and realizes seamless transmission of shallow features to the deep network.

[0044] Furthermore, the IC-GCN-Plus model is used to improve the recognition performance for large-scale and complex data sets, and solve the problem of numerous action categories and varied actions; the IC-GCN-Plus model enhances the recognition performance by enhancing the depth, width and diversity of connections of the network, combined with graph convolution operations;

[0045] The IC-GCN-Plus model is an extension of the graph convolutional network. In terms of optimizing the processing performance of large-scale data sets, it covers complex adjacency matrix construction, multi-scale feature fusion methods, and deeper network architecture design.

[0046] Furthermore, the IC-GCN-Light model is used to implement skeleton behavior recognition on devices with limited computing resources, maintaining high recognition accuracy; the IC-GCN-Light model achieves lightweight model by simplifying the model structure, reducing the number of parameters, and retaining key functions;

[0047] The lightweight design of the IC-GCN-Light model includes: adopting a shallower network architecture, reducing the number of channels, and implementing efficient graph convolution operations and feature fusion strategies.

[0048] The present invention also includes a storage medium, which includes a stored program, wherein when the program is run, the resource flexibility skeleton action classification method based on adaptive structure optimization is executed.

[0049] The present invention also includes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the resource flexibility skeleton action classification method based on adaptive structural optimization through the computer program.

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] The resource flexibility skeleton action classification method based on adaptive structural optimization provided by the present invention realizes efficient feature updating by constructing adaptive individual and common structures. The individual and common graph convolution modules respectively refine key features and generate robust topological matrices, and the individual and common time convolution modules capture the key dynamic changes of actions in the time dimension. The jump connection module enhances the retention of key information in deep iterations, while the Plus and Light variants adapt to different hardware environments, suitable for large-scale data sets and resource-constrained environments, respectively. This adaptive structural optimization design reduces the computational complexity and improves the model efficiency while ensuring the accuracy of the model, making it more suitable for practical application scenarios.

[0052] The resource flexibility skeleton action classification method based on adaptive structural optimization provided by the present invention introduces individual and common temporal convolution modules to capture the key dynamic changes of actions in the time dimension and integrate them with spatial features. This overcomes the shortcomings of the prior art that only focuses on static features or only processes time series, realizes the comprehensive extraction of action spatiotemporal features, and improves the ability to capture dynamic changes.

[0053] The resource-flexible skeleton action classification method based on adaptive structural optimization provided by the present invention designs a skip connection module, and realizes the seamless transmission of shallow features to deep networks by introducing skip residual connections, thereby ensuring the retention of key information. This design not only helps to maintain the original information of the features, but also effectively alleviates the gradient vanishing phenomenon and improves the training efficiency of the network. Compared with traditional residual connections, Skip-Trans adopts an adaptive pyramid structure, which is more suitable for skeleton behavior recognition tasks, reduces the loss of information during network propagation, and enhances the expressiveness of the network. Therefore, the skip connection module effectively improves the classification performance of the model.

[0054] The resource-flexible skeleton action classification method based on adaptive structural optimization provided by the present invention designs two variants, IC-GCN-Plus and IC-GCN-Light, which are respectively suitable for large-scale data sets and resource-constrained environments. IC-GCN-Plus improves the complexity and recognition ability of the model by increasing the number of channels and the number of adjacency matrices, and is suitable for large-scale and complex data sets; while IC-GCN-Light reduces the computational complexity of the model by reducing the number of channels and the number of adjacency matrices, and is suitable for resource-constrained environments and real-time application scenarios. This diversified model variant design enables the present invention to select a suitable model according to different application requirements, thereby better meeting the needs of actual application scenarios.

[0055] In summary, the present invention not only achieves significant advantages in high-precision motion classification performance, but also shows excellent potential in flexibility and robustness in practical applications. This makes the invention have significant competitive advantages in various fields that require efficient motion classification, such as human-computer interaction, motion analysis, intelligent monitoring, virtual reality and robotics. The design of the present invention can effectively respond to the needs of different data sets and tasks, while taking into account computing efficiency and resource consumption, and provides flexible solutions for various application scenarios.

[0056] Based on the above reasons, the present invention can be widely promoted in the technical field of human motion recognition and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 This is a flow chart of the resource flexibility skeleton action classification method based on adaptive structure optimization of the present invention.

[0059] Figure 2 This is the framework diagram of the IC-GC spatial domain model of the present invention.

[0060] Figure 3 This is a framework diagram of the IC-TC time domain model of the present invention.

[0061] Figure 4 This is the overall structure diagram of Skip-Trans and the network in the present invention.

[0062] Figure 5 It is a schematic diagram of parameter design in an embodiment of the present invention.

[0063] Figure 6 Schematic diagram of the visualization and skeleton heat map in the embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0066] like Figure 1 As shown, the present invention provides a resource flexibility skeleton action classification method based on adaptive structure optimization, comprising:

[0067] Collect skeleton video samples to be trained, perform preprocessing, and construct a qualified data set;

[0068] In specific implementation, as a preferred embodiment of the present invention, the construction of a qualified data set includes:

[0069] Collect video samples containing human skeleton movements and preprocess the video data, wherein the preprocessing includes: obtaining skeleton data, eliminating noise, and normalizing the viewing angle, and storing the preprocessed video data in a qualified data set.

[0070] The qualified dataset is used as the basis for inputting the IC-GC spatial domain model, such as Figure 2 As shown, the unique features of each joint and the interaction information between different joints are obtained;

[0071] In the specific implementation, as a preferred embodiment of the present invention, the qualified data set is used as the basic input IC-GC spatial domain model to obtain the unique features of each joint and the common features between different joints, specifically including:

[0072] The IC-GC spatial domain model includes a spatial individuality stream S-IS and a spatial commonality stream S-CS. The spatial individuality stream S-IS is used to extract the unique features of specific joints, and the spatial commonality stream S-CS is used to capture the interaction information between different joints. During implementation, the motivation is to capture the specificity and co-occurrence information between different joints and frames in skeleton data in real time, thereby improving the model's ability to recognize action details and adapt to dynamic changes.

[0073] The IC-GC spatial domain model is expressed as:

[0074] X out =CFC(U(S c (A 0,1 ,X in ),A p ),E(X in ))

[0075] Among them, X in represents the input features, X out represents the output feature, A 0,1 represents the original one-hot adjacency matrix, A p represents the fully parameterized adjacency matrix, S c represents the combined module of spatial individual stream S-IS and spatial common stream S-CS, U represents the self-attention adjacency matrix module, E represents the feature conversion module, and CFC represents the channel feature fusion module;

[0076] The feature extraction process of spatial individuality flow and spatial commonality flow is as follows:

[0077] V s =S c (A 0,1 ,X in )=[I s (X in )||C s (A 0,1 ,X in )] T W

[0078]

[0079] in, represents the space prototype vector, represents the original one-hot adjacency matrix, || represents the connection operation, P avg represents adaptive average pooling, θ(·) and δ(·) represent activation functions, and I s (·) indicates that the input X in Transpose and perform dimensionality reduction, C s(·) represents the feature after dimension reduction and A 0,1 Multiply to complete the first feature fusion, connect the corresponding channels, and then use 1×1 convolution to fuse the channels, and finally get the prototype vector of the next part;

[0080] Generate an N×N self-attention adjacency matrix A k :

[0081]

[0082] in, represents matrix multiplication, R e represents vector replication, represents a fully parameterized adjacency matrix;

[0083] The process of feature learning and updating is expressed as:

[0084] X m =E(X in )=X in W

[0085] Where W represents a size of C in ×C out The real matrix, C in and C out Respectively represent the number of input and output channels;

[0086] Combined with A k and X m To obtain the final output of channel feature fusion, it is expressed as:

[0087]

[0088] in, represents the final output of the spatial domain model, A k represents the self-attention adjacency matrix, CFC(A k ,X m ) represents the combination of adjacency matrix A k and feature X m Each and Corresponding to X m and A k The ith channel of is combined by matrix multiplication, and all channels are combined by concatenation to obtain the output X out .

[0089] The output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model, such as Figure 3 As shown, common features and prominent features are obtained;

[0090] In specific implementation, as a preferred embodiment of the present invention, the output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model to obtain common features and prominent features, which specifically include:

[0091] The IC-TC time domain model includes a temporal individual stream T-IS and a temporal common stream T-CS, wherein the temporal individual stream T-IS is used to maintain intra-frame information, and the temporal common stream T-CS is used for inter-frame interaction and processing; in implementation, the design motivation is to effectively extract the temporal attributes of the skeleton sequence through temporal convolution processing, and maintain high sensitivity to key frames to achieve accurate recognition of action details.

[0092] The output X of the IC-GC spatial domain model out As the input Y of IC-TC time domain model in , then the output Y of the IC-TC time domain model is out for:

[0093]

[0094] Among them, C t (·) represents the time commonality flow, I t (·) indicates the flow of time personality;

[0095] In the temporal personality stream T-IS, the input features are adaptively averaged pooled P avg To extract common features, adaptive maximum pooling P is used max To extract salient features:

[0096]

[0097] in, θ(·),δ(·) represent activation functions;

[0098] A multi-scale temporal convolution strategy is adopted in the temporal commonality stream T-CS to effectively capture features at different time scales and enhance the model's ability to understand and represent time series data, thereby achieving better performance in complex time series analysis tasks.

[0099] In implementation, individual and common convolution modes based on unified channel separation are realized through IC-TC and IC-GC.

[0100] The outputs of IC-TC and IC-GC are introduced into the Skip-Trans jump link mechanism, such as Figure 4 As shown in the figure, information transfer between different levels of the model is realized, and key information in the deep iteration process is retained.

[0101] During implementation, the Skip-Trans jump link mechanism realizes the effective fusion of information between different levels of the network. By establishing a direct channel between the shallow and deep layers of the network, the shallow feature map can directly affect the deep layer, thereby improving the network's perception sensitivity to the input data.

[0102] In specific implementation, as a preferred embodiment of the present invention, the Skip-Trans jump link mechanism, by introducing a skip residual connection and adopting an adaptive pyramid structure, enables the network to transmit and fuse features at different levels, establishes a direct information transmission path between the shallow and deep layers of the network, enables the shallow feature map to directly act on the deep layer, and realizes seamless transmission of shallow features to the deep network.

[0103] During implementation, the Skip-Trans mechanism promotes the effective integration of information between various levels of the network, which not only helps to maintain the original information of the features, but also effectively alleviates the gradient vanishing phenomenon and improves the training efficiency of the network. Compared with the traditional residual connection, Skip-Trans adopts an adaptive pyramid structure, which enables the network to transmit and fuse features at different levels. At the same time, considering the characteristics of skeleton data, it is more suitable for skeleton behavior recognition tasks, reduces the loss of information during network propagation, and enhances the expressiveness of the network. In addition, the design of Skip-Trans is simple and efficient, and will not significantly increase the computing cost, which is particularly critical for real-time applications and devices with limited computing resources.

[0104] Aiming at different hardware environments and efficiency requirements, two model variants, IC-GCN-Plus and IC-GCN-Light, are designed to realize skeleton action classification.

[0105] In specific implementation, as a preferred embodiment of the present invention, the IC-GCN-Plus model is used to improve the recognition performance of large-scale and complex data sets, and solve the problems of numerous action categories and varied actions; the IC-GCN-Plus model enhances the recognition performance by enhancing the depth, width and diversity of connections of the network, combined with graph convolution operations;

[0106] The IC-GCN-Plus model is an extension of the graph convolutional network. In terms of optimizing the processing performance of large-scale data sets, it covers complex adjacency matrix construction, multi-scale feature fusion methods, and deeper network architecture design.

[0107] When implemented, the significant advantages of the IC-GCN-Plus model are reflected in its excellent feature extraction capabilities and high adaptability, which can accurately recognize complex action patterns and meet the needs of different data sets and tasks.

[0108] In specific implementation, as a preferred embodiment of the present invention, the IC-GCN-Light model is used to realize skeleton behavior recognition on devices with limited computing resources, maintaining high recognition accuracy; the IC-GCN-Light model realizes the lightweight of the model by simplifying the model structure, reducing the number of parameters, and retaining key functions;

[0109] The lightweight design of the IC-GCN-Light model includes: adopting a shallower network architecture, reducing the number of channels, and implementing efficient graph convolution operations and feature fusion strategies.

[0110] During implementation, the core advantages of the IC-GCN-Light model are reflected in its efficient computing performance and low resource allocation requirements, which can significantly reduce the computational and storage burden of the model while maintaining a reasonable accuracy rate.

[0111] The present invention also includes a storage medium, which includes a stored program, wherein when the program is run, the resource flexibility skeleton action classification method based on adaptive structure optimization is executed.

[0112] The present invention also includes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the resource flexibility skeleton action classification method based on adaptive structural optimization through the computer program.

[0113] Example

[0114] like Figure 1 As shown, the present invention provides a resource flexibility skeleton action classification method based on adaptive structure optimization. The operating system used in the experiment is Ubuntu 22.04.1, the deep learning framework adopted is PyTorch, and the specific configuration involved in the experiment is shown in Table 1.

[0115] Table 1 Experimental configuration table

[0116] project Configuration Processor (CPU) 12th Gen Intel(R)Core(TM)i9-12900KF cpu@3.20GHz Graphics card (GPU) GeForce RTX 3090×2 operating system Ubuntu 22.04.1 frame PyTorch

[0117] Table 2 proves the effectiveness of the innovative submodules proposed in the present invention through sufficient ablation experiments. Tables 3, 4, and 5 demonstrate the performance advantages of the present invention on four different datasets through comparative experiments with existing GCN-based methods and Transformer-based methods. Figure 5 The parameter design is shown. Figure 6 Visualization and skeleton heatmaps are shown, which means that the IC-GCN model achieves better results in action classification.

[0118] Table 2 Ablation experiment table of individual flow, common flow and jump module on IC-GCN-Basic model

[0119] method Accuracy (%) Param. FLOPs Baseline(Spatial) 93.95 3.45M ～3.98G Baseline+Skip-Block 94.58↑0.63 3.49M ～4.02G Baseline+S-IS 94.58↑0.63 3.12M ～3.57G Baseline+S-CS 94.62↑0.67 3.12M ～3.57G Baseline+S-ICS 94.78↑0.83 3.12M ～3.57G Baseline+S-ICS+Skip-Block 95.38↑1.43 3.17M ～3.61G Baseline+T-CS 94.76↑0.81 1.52M ～1.92G Baseline+T-ICS 94.98↑1.03 1.52M ～1.92G Baseline+T-ICS+Skip-Block 95.00↑1.05 1.57M ～1.96G Baseline+S-ICS+T-ICS 95.07↑1.12 1.21M ～1.51G Baseline+S-ICS+T-ICS+Skip-Block 95.40↑1.45 1.25M ～1.55G

[0120] Table 3 Comparative experiments of IC-GCN on NTU dataset

[0121]

[0122]

[0123] Table 4 Comparative experiments of IC-GCN on NW-UCLA dataset

[0124]

[0125] Table 5 Comparative experiments of IC-GCN on Kinetics-400 dataset

[0126]

[0127]

[0128] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0129] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0132] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A resource flexibility skeleton action classification method based on adaptive structural optimization, characterized in that: include: Collect skeleton video samples to be trained, perform preprocessing, and construct a qualified data set; The qualified dataset is used as the basic input to the IC-GC spatial domain model to obtain the unique features of each joint and the interaction information between different joints; The output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model to obtain common features and salient features; Introducing the Skip-Trans jump link mechanism to achieve information transfer between different levels of the model and retain key information in the deep iteration process; Aiming at different hardware environments and efficiency requirements, two model variants, IC-GCN-Plus and IC-GCN-Light, are designed to realize skeleton action classification.

2. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The constructing of a qualified data set comprises: Collect video samples containing human skeleton movements and preprocess the video data, wherein the preprocessing includes: obtaining skeleton data, eliminating noise, and normalizing the viewing angle, and storing the preprocessed video data in a qualified data set.

3. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The qualified data set is used as the basic input IC-GC spatial domain model to obtain the unique features of each joint and the common features between different joints, specifically including: The IC-GC spatial domain model includes a spatial individual stream S-IS and a spatial common stream S-CS, wherein the spatial individual stream S-IS is used to extract unique features of a specific joint, and the spatial common stream S-CS is used to capture interactive information between different joints; The IC-GC spatial domain model is expressed as: X out =CFC(U(S c (A 0,1 ,X in ),A p ),E(X in )) Among them, X in represents the input features, X out represents the output feature, A 0,1 represents the original one-hot adjacency matrix, A p represents the fully parameterized adjacency matrix, S c represents the combined module of spatial individual stream S-IS and spatial common stream S-CS, U represents the self-attention adjacency matrix module, E represents the feature conversion module, and CFC represents the channel feature fusion module; The feature extraction process of spatial individuality flow and spatial commonality flow is as follows: V s =S c (A 0,1 ,X in )=[I s (X in )||C s (A 0,1 ,X in )] T W in, represents the space prototype vector, represents the original one-hot adjacency matrix, || represents the connection operation, P avg represents adaptive average pooling, θ(·) and δ(·) represent activation functions, and I s (·) indicates that the input X in Transpose and perform dimensionality reduction, C s (·) represents the feature after dimension reduction and A 0,1 Multiply to complete the first feature fusion, connect the corresponding channels, and then use 1×1 convolution to fuse the channels, and finally get the prototype vector of the next part; Generate an N×N self-attention adjacency matrix A k : in, represents matrix multiplication, R e represents vector replication, represents a fully parameterized adjacency matrix; The process of feature learning and updating is expressed as: X m =E(X in )=X in W Where W represents a size of C in ×C out The real matrix, C in and C out Respectively represent the number of input and output channels; Combined with A k and X m To obtain the final output of channel feature fusion, it is expressed as: in, represents the output of the spatial domain model, A k represents the self-attention adjacency matrix, CFC(A k ,X m ) represents the combination of adjacency matrix A k and feature X m Each and Corresponding to X m and A k The ith channel of is combined by matrix multiplication, and all channels are combined by concatenation to obtain the output X out .

4. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The output of the IC-GC spatial domain model is used as the input of the IC-TC temporal domain model to obtain common features and prominent features, which specifically include: The IC-TC time domain model includes a time individuality stream T-IS and a time commonality stream T-CS, wherein the time individuality stream T-IS is used to maintain intra-frame information, and the time commonality stream T-CS is used to perform inter-frame interaction and processing; The output X of the IC-GC spatial domain model out As the input Y of IC-TC time domain model in , then the output Y of the IC-TC time domain model is out for: Among them, C t (·) represents the time commonality flow, I t (·) indicates the flow of time personality; In the temporal personality stream T-IS, the input features are adaptively averaged pooled P avg To extract common features, adaptive maximum pooling P is used max To extract salient features: in, θ(·),δ(·) represent activation functions; In the temporal commonality stream T-CS, a multi-scale temporal convolution strategy is adopted to capture features at different time scales and complete complex time series analysis tasks.

5. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The Skip-Trans jump link mechanism introduces skip residual connections and adopts an adaptive pyramid structure, so that the network can transmit and fuse features at different levels, establish a direct information transmission path between the shallow and deep layers of the network, and enable the shallow feature map to directly act on the deep layer, thereby realizing seamless transmission of shallow features to the deep network.

6. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The IC-GCN-Plus model is used to improve the recognition performance of large-scale and complex data sets and solve the problem of numerous action categories and varied actions. The IC-GCN-Plus model enhances the recognition performance by enhancing the depth, width and diversity of connections of the network and combining graph convolution operations. The IC-GCN-Plus model is an extension of the graph convolutional network. In terms of optimizing the processing performance of large-scale data sets, it covers complex adjacency matrix construction, multi-scale feature fusion methods, and deeper network architecture design.

7. The resource flexibility skeleton action classification method based on adaptive structural optimization according to claim 1 is characterized in that: The IC-GCN-Light model is used to implement skeleton behavior recognition on devices with limited computing resources and maintain high recognition accuracy. The IC-GCN-Light model achieves lightweight model by simplifying the model structure and reducing the number of parameters while retaining key functions. The lightweight design of the IC-GCN-Light model includes: adopting a shallower network architecture, reducing the number of channels, and implementing efficient graph convolution operations and feature fusion strategies.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the method according to any one of claims 1 to 7 is executed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the method according to any one of claims 1 to 7 by running the computer program.

Images