Skeleton behavior recognition model training method and system, computer equipment and storage medium

Abstract

The invention provides a skeleton behavior recognition model training method and system, computer equipment and a storage medium, and the method comprises the steps: carrying out the data enhancement of skeleton data, and obtaining a skeleton sequence; extracting a time initial feature and a space initial feature of the skeleton sequence; performing feature extraction on the time initial feature and the space initial feature to obtain a final time feature and a final space feature; constructing a loss function of a skeleton behavior recognition model according to the final time features and the spatial features; and performing iterative training on the skeleton behavior recognition model by using the loss function, and when the loss function value is converged to a preset range, taking the corresponding skeleton behavior recognition model as a final skeleton behavior recognition model. According to the method, through the skeleton data enhancement strategy and initial feature extraction, multi-granularity spatial-temporal features are generated, and a multi-granularity and cross-level comparison loss mechanism is combined, so that the recognition capability of the model on complex behaviors is improved, efficient learning can be realized without a large amount of annotated data, and relatively high robustness is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of behavior recognition technology, and in particular relates to a skeleton behavior recognition model training method, system, computer equipment and storage medium. Background Art

[0002] Skeleton behavior recognition is a technology that uses the spatiotemporal dynamic information of key joints of the human body to identify and classify human behaviors. Skeleton data is usually generated by depth cameras (such as Kinect), motion capture systems (such as OptiTrack), or sensors (such as IMU sensors), representing the skeleton structure of the human body in three-dimensional space. Compared with traditional RGB video data, the temporal information of these joints has the advantages of being insensitive to lighting changes, low computational complexity, and small data volume. It is particularly suitable for resource-constrained scenarios such as intelligent monitoring, human posture analysis, and human-computer interaction systems.

[0003] However, existing skeleton behavior recognition technologies face several bottlenecks in practice, especially in terms of data annotation costs and data scarcity. Constructing large-scale annotated skeleton datasets usually requires the participation of professional annotators, and the annotation process is cumbersome and time-consuming, making it difficult to obtain high-quality large-scale annotated data. At the same time, due to the diversity of data acquisition equipment and differences in acquisition environments, the amount of unlabeled skeleton data often far exceeds the amount of labeled data. Therefore, how to effectively utilize a large amount of unlabeled skeleton data has become the focus of current research, which provides a background for the proposal of self-supervised learning methods.

[0004] Self-supervised learning enables the model to extract key features of skeleton data without labels by designing pre-training tasks, learn the intrinsic patterns of actions from massive unlabeled data, and thus improve the generalization ability of downstream tasks. Currently, the main self-supervised skeleton action recognition methods can be divided into two categories: generation-based methods and contrastive learning-based methods.

[0005] The generation-based skeleton behavior recognition method has the following problems: During the training process, the model is easily affected by the quality of the generated samples. If there is a significant gap between the generated samples and the real data, it may cause the model to learn the wrong pattern. In addition, the training process of such methods is relatively complex and requires finding an appropriate balance between the generation quality and the reconstruction quality, which increases the difficulty of implementation and may affect the stability and reliability of the model.

[0006] Skeleton action recognition methods based on contrastive learning have the advantage of improving the model's ability to distinguish in feature space, thereby improving the accuracy of action recognition to a certain extent. Such methods usually encode instance-level features of each action (i.e., feature representations of action samples in the entire sequence) and compare these features to identify the differences between different actions. However, despite their excellent performance in some cases, existing methods often fail to fully capture the complex spatiotemporal dynamic relationships of actions. In addition, a single contrast method limits the model's ability to explore action patterns from multiple perspectives, which may lead to insufficient performance when dealing with complex actions. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a skeleton behavior recognition model training method, system, computer equipment and storage medium.

[0008] In a first aspect, the present invention provides a skeleton behavior recognition model training method, comprising: Perform data enhancement on the skeleton data to obtain a skeleton sequence; Extracting temporal initial features and spatial initial features of the skeleton sequence; Extracting the initial time features and the initial space features to obtain the final time features and space features; Construct the loss function of the skeleton behavior recognition model based on the final temporal and spatial features; The skeleton behavior recognition model is iteratively trained using the loss function. When the loss function value converges to a preset range, the corresponding skeleton behavior recognition model is used as the final skeleton behavior recognition model.

[0009] Optionally, performing data enhancement on the skeleton data to obtain a skeleton sequence includes: Construct skeleton sequence x z The expression is: ; in, represents the skeleton data enhanced by cropping under probability p1; represents the skeleton data enhanced by rotation under probability p2; represents the skeleton data enhanced by flipping with probability p3; represents the skeleton data enhanced by temporal masking under probability p4; represents the skeleton data enhanced by spatial masking under probability p5; x∈R T×V×C , R is a real number, T is the number of time frames, V is the number of skeleton joints, and C is the dimension of each joint; the skeleton data after spatial masking is , ;m spatialrepresents a mask vector of length P, m spatial ∈{0,1} P ; The skeleton data after using the time mask is , ;m temporal represents a mask vector of length F, m temporal ∈{0,1} F .

[0010] Optionally, extracting the temporal initial features and the spatial initial features of the skeleton sequence includes: Use graph convolutional networks to extract primary features x of skeleton sequences i : x i =GCN(x z ,A); Among them, x z is the skeleton sequence; A is the adjacency matrix in graph convolution; GCN(·) is the graph convolution operation; , R is a real number, T is the number of time frames, V is the number of joints in the skeleton, C i is the channel dimension of the primary features of the skeleton sequence after being processed by the graph convolutional network; The primary feature x of the skeleton sequence i Perform a deformation operation to merge the channel dimension with the time dimension and the space dimension respectively to obtain the first time initial feature and the second space initial features ;in, , ; The first temporal initial feature and the second spatial initial feature are embedded respectively to obtain the final temporal initial feature and spatial initial feature: ; ; in, is the final temporal initial feature of the skeleton sequence; is the final spatial initial feature of the skeleton sequence; W1 is the first weight matrix; σ(·) represents the ReLU function; LN(·) represents the layer normalization operation; W t is the second weight matrix; b t is the bias corresponding to the initial feature at the first time; b1 is the bias output after adjusting the initial feature at the first time; W2 is the third weight matrix; W s is the fourth weight matrix; b s is the bias corresponding to the initial feature of the first space; b2 is the bias output after adjusting the initial feature of the first space.

[0011] Optionally, the extracting the temporal initial features and the spatial initial features to obtain the final temporal features and the spatial features includes: The spatial multi-granularity generation module is used to generate features of three different spatial granularities; the features of the three different spatial granularities are , and ; Will , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the spatial domain level features y s ;in, ; concat(·) represents the concatenation operation; f(·) represents the feature output by the Transformer network; The time multi-granularity division module is used to generate three features of different time granularities by using the time downsampling module multiple times; the features of the three different time granularities are , and ; Will , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the time domain level features y t ;in, .

[0012] Optionally, the loss function of constructing the skeleton behavior recognition model according to the final temporal features and spatial features includes: Connect the domain-level features of the same domain to obtain the instance-level features y i ; The multi-granularity contrast loss L is calculated according to the following formula mg : ; Where G is the total number of time granularity or space granularity; g = 1, 2, …, G; F (·) is the similarity calculation function; Indicates the spatial granularity corresponding to g=1 in encoder q; ; projector sq (·) represents the output of the spatial mapper in encoder q, which consists of two fully connected layers and an activation function; Represents the final spatial initial features of the skeleton sequence in encoder q; represents the g-th spatial granularity in encoder k; ; projector sk (·) represents the output of the spatial mapper in encoder k; is the g-th spatial granularity feature in encoder k; M sg is a set of spatial multi-granularity negative samples; m1 is M sg Negative samples in ; Indicates the time granularity corresponding to g=1 in encoder q; ; projector tq (·) represents the output of the temporal mapper in encoder q, which consists of two fully connected layers and an activation function; represents the final temporal initial features of the skeleton sequence in encoder q; represents the g-th time granularity in encoder k; ; projector tk (·) represents the output of the time mapper in encoder k; is the g-th time granularity feature in encoder k; M tg is a set of time-multi-granularity negative samples; m2 is M tg Negative samples in ; The cross-level contrast loss L is calculated according to the following formula cross : ; Among them, q s is the final spatial feature in encoder q; q s =projector sq (y sq );y sq is the spatial domain-level feature in encoder q; k i is the final instance-level feature in encoder k; k i =projector ik (y ik ); projector ik (·) represents the output of the instance-level mapper in encoder k; y ik M is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder k; i is the negative sample set of instance-level features; m3 is M i Negative samples in q t is the final temporal feature in encoder q; q t =projector tq (y tq );y tq is the time domain feature in encoder q; i is the final instance-level feature in encoder q; q i =projectoriq (y iq ); projector iq (·) represents the output of the instance-level mapper in encoder q; y iq k is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder q; s is the final spatial feature in encoder k; k s =projector sk (y sk );y sk is the spatial domain-level feature in encoder k; t is the final temporal feature in encoder k; k t =projector tk (y tk );y tk is the temporal domain-level feature in encoder k; M t is the negative sample set of the final time feature; m5 is M t Negative samples in ; The sum of multi-granularity contrast loss and cross-level contrast loss is taken as the loss of the skeleton action recognition model, and the loss function of the skeleton action recognition model is obtained.

[0013] In a second aspect, the present invention provides a skeleton behavior recognition model training system, comprising: A data enhancement module is used to enhance the skeleton data and obtain a skeleton sequence; The first feature extraction module is used to extract the temporal initial features and the spatial initial features of the skeleton sequence; The second feature extraction module is used to extract the time initial feature and the space initial feature to obtain the final time feature and space feature; A loss function construction module is used to construct a loss function of the skeleton behavior recognition model based on the final temporal features and spatial features; The model training module is used to iteratively train the skeleton behavior recognition model using the loss function. When the loss function value converges to a preset range, the corresponding skeleton behavior recognition model is used as the final skeleton behavior recognition model.

[0014] Optionally, the data enhancement module includes: Building blocks used to construct the skeleton sequence x z The expression is: ; in, represents the skeleton data enhanced by cropping under probability p1; represents the skeleton data enhanced by rotation under probability p2; represents the skeleton data enhanced by flipping with probability p3; represents the skeleton data enhanced by temporal masking under probability p4; represents the skeleton data enhanced by spatial masking under probability p5; x∈R T×V×C , R is a real number, T is the number of time frames, V is the number of skeleton joints, and C is the dimension of each joint; the skeleton data after spatial masking is , ;m spatial represents a mask vector of length P, m spatial ∈{0,1} P ; The skeleton data after using the time mask is , ;m temporal represents a mask vector of length F, m temporal ∈{0,1} F .

[0015] Optionally, the first feature extraction module includes: Feature extraction unit, used to extract primary features x of the skeleton sequence using a graph convolutional network i : x i =GCN(x z ,A); Among them, x z is the skeleton sequence; A is the adjacency matrix in graph convolution; GCN(·) is the graph convolution operation; , R is a real number, T is the number of time frames, V is the number of joints in the skeleton, C i is the channel dimension of the primary features of the skeleton sequence after being processed by the graph convolutional network; Feature deformation operation unit, used to transform the primary features x of the skeleton sequence i Perform a deformation operation to merge the channel dimension with the time dimension and the space dimension respectively to obtain the first time initial feature and the second space initial features ;in, , ; The feature embedding operation unit is used to perform embedding operations on the first time initial feature and the second space initial feature respectively to obtain the final time initial feature and space initial feature: ; ; in, is the final temporal initial feature of the skeleton sequence; is the final spatial initial feature of the skeleton sequence; W1 is the first weight matrix; σ(·) represents the ReLU function; LN(·) represents the layer normalization operation; W t is the second weight matrix; b t is the bias corresponding to the initial feature at the first time; b1 is the bias output after adjusting the initial feature at the first time; W2 is the third weight matrix; W s is the fourth weight matrix; b s is the bias corresponding to the initial feature of the first space; b2 is the bias output after adjusting the initial feature of the first space.

[0016] Optionally, the second feature extraction module includes: The first feature generation unit is used to generate three features of different spatial granularities using a spatial multi-granularity generation module; wherein the features of the three different spatial granularities are respectively , and ; The second feature generating unit is used to , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the spatial domain level features y s ;in, ; concat(·) represents the concatenation operation; f(·) represents the feature output by the Transformer network; The third feature generation unit is used to generate three features of different time granularities by using the time multi-granularity division module and repeatedly using the time downsampling module; wherein the features of the three different time granularities are respectively , and ; The fourth feature generating unit is used to , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the time domain level features y t ;in, .

[0017] Optionally, the loss function building module includes: The fifth feature generation unit is used to connect the domain-level features of the same domain to obtain the instance-level feature y i ; The first calculation unit is used to calculate the multi-granularity contrast loss L according to the following formula mg : ; Where G is the total number of time granularity or space granularity; g = 1, 2, …, G; F (·) is the similarity calculation function; Indicates the spatial granularity corresponding to g=1 in encoder q; ; projector sq (·) represents the output of the spatial mapper in encoder q, which consists of two fully connected layers and an activation function; Represents the final spatial initial features of the skeleton sequence in encoder q; represents the g-th spatial granularity in encoder k; ; projector sk (·) represents the output of the spatial mapper in encoder k; is the g-th spatial granularity feature in encoder k; M sg is a set of spatial multi-granularity negative samples; m1 is M sg Negative samples in ; Indicates the time granularity corresponding to g=1 in encoder q; ; projector tq (·) represents the output of the temporal mapper in encoder q, which consists of two fully connected layers and an activation function; represents the final temporal initial features of the skeleton sequence in encoder q; represents the g-th time granularity in encoder k; ; projector tk (·) represents the output of the time mapper in encoder k; is the g-th time granularity feature in encoder k; M tg is a set of time-multi-granularity negative samples; m2 is M tg Negative samples in .

[0018] The second calculation unit is used to calculate the cross-level contrast loss L according to the following formula cross : ; Among them, q s is the final spatial feature in encoder q; q s =projector sq (y sq );y sq is the spatial domain-level feature in encoder q; k i is the final instance-level feature in encoder k; k i =projector ik (y ik ); projector ik(·) represents the output of the instance-level mapper in encoder k; y ik M is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder k; i is the negative sample set of instance-level features; m3 is M i Negative samples in q t is the final temporal feature in encoder q; q t =projector tq (y tq );y tq is the time domain feature in encoder q; i is the final instance-level feature in encoder q; q i =projector iq (y iq ); projector iq (·) represents the output of the instance-level mapper in encoder q; y iq k is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder q; s is the final spatial feature in encoder k; k s =projector sk (y sk );y sk is the spatial domain-level feature in encoder k; t is the final temporal feature in encoder k; k t =projector tk (y tk );y tk is the temporal domain-level feature in encoder k; M t is the negative sample set of the final time feature; m5 is M t Negative samples in ; The loss determination unit is used to take the sum of the multi-granularity contrast loss and the cross-level contrast loss as the loss of the skeleton behavior recognition model to obtain the loss function of the skeleton behavior recognition model.

[0019] In a fourth aspect, the present invention provides a computer device comprising a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the steps of the skeleton behavior recognition model training method described in the first aspect are implemented.

[0020] In a fifth aspect, the present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the skeleton behavior recognition model training method described in the first aspect are implemented.

[0021] The present invention provides a skeleton behavior recognition model training method, system, computer equipment and storage medium, wherein the method includes data enhancement of skeleton data to obtain a skeleton sequence; extracting the time initial features and space initial features of the skeleton sequence; extracting features of the time initial features and space initial features to obtain final time features and space features; constructing a loss function of the skeleton behavior recognition model according to the final time features and space features; iteratively training the skeleton behavior recognition model using the loss function, and when the loss function value converges to a preset range, the corresponding skeleton behavior recognition model is used as the final skeleton behavior recognition model. The present invention takes into account the differences in spatial and temporal characteristics, and the spatial and temporal characteristics of different skeleton action samples, and uses the temporal features and spatial features of skeleton data to perform feature extraction and modeling, which can effectively construct the spatial dependency relationship between skeleton joints, thereby identifying different types of actions. Through the skeleton data enhancement strategy and initial feature extraction, multi-granularity spatiotemporal features are effectively generated, and the recognition ability of the skeleton behavior recognition model for complex behaviors is improved by combining multi-granularity and cross-level contrast loss mechanisms, and efficient learning can be achieved without a large amount of labeled data, with strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of a flow chart of a skeleton behavior recognition model training method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of an encoder q in contrastive learning provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a skeleton behavior recognition model training system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Example 1.

[0026] like Figure 1As shown, this embodiment provides a skeleton behavior recognition model training method, including: step 101, performing data enhancement on skeleton data to obtain a skeleton sequence.

[0027] In this step, data enhancement is performed in a contrastive learning framework. In order to improve data diversity and reduce information leakage, this embodiment proposes a spatiotemporal masking strategy. Spatial masking is based on the natural structure of the human skeleton, divides the skeleton into multiple parts (such as limbs and torso), and randomly selects parts of the skeleton for masking to enhance data diversity in the spatial dimension. Temporal masking divides the time series into multiple groups, and randomly selects a part of the groups for masking to reduce redundant information in the time dimension and enhance the encoder's ability to represent spatiotemporal features (temporal features and spatial features).

[0028] Exemplarily, construct the backbone sequence x z expression.

[0029] .

[0030] in, represents the skeleton data enhanced by cropping under probability p1; represents the skeleton data enhanced by rotation under probability p2; represents the skeleton data enhanced by flipping with probability p3; represents the skeleton data enhanced by temporal masking under probability p4; represents the skeleton data enhanced by spatial masking under probability p5; x∈R T×V×C , R is a real number, T is the number of time frames, V is the number of skeleton joints, and C is the dimension of each joint; the skeleton data after spatial masking is , ;m spatial represents a mask vector of length P, m spatial ∈{0,1} P ; The skeleton data after using the time mask is , ; ⊙ represents the dot multiplication operation of the matrix; m temporal represents a mask vector of length F, m temporal ∈{0,1} F In the spatial mask, 1 indicates that the part is retained, and 0 indicates that the part is masked; in the temporal mask, 1 indicates that the part is retained, and 0 indicates that the part is masked. The obtained skeleton sequence is input into the encoder q and the encoder k respectively, and each encoder performs the operation of steps 102-103.

[0031] Step 102: extracting the temporal initial features and spatial initial features of the skeleton sequence.

[0032] In this step, the graph convolutional network is used to perform preliminary feature extraction on the input skeleton sequence to generate the initial spatial feature representation. These spatial features are then passed to the spatial and temporal branches respectively to form a spatiotemporal feature extraction path, laying the foundation for subsequent multi-granular feature extraction.

[0033] Exemplarily, a graph convolutional network is used to extract the primary features x of the skeleton sequence in encoder q or encoder k. i .

[0034] x i =GCN(x z ,A).

[0035] Among them, x z is the skeleton sequence; A is the adjacency matrix in graph convolution; GCN(·) is the graph convolution operation; , R is a real number, T is the number of time frames, V is the number of joints in the skeleton, C i is the channel dimension of the primary features of the skeleton sequence after being processed by the graph convolutional network.

[0036] The primary feature x of the skeleton sequence i Perform a deformation operation to merge the channel dimension with the time dimension and the space dimension respectively to obtain the first time initial feature and the second space initial features ;in, , .

[0037] The first temporal initial feature and the second spatial initial feature are respectively embedded to obtain the final temporal initial feature and the final spatial initial feature.

[0038] .

[0039] .

[0040] in, is the final temporal initial feature of the skeleton sequence; is the final spatial initial feature of the skeleton sequence; W1 is the first weight matrix, that is, the weight matrix used in the encoder to further process the temporal features to obtain features; σ(·) represents the ReLU function; LN(·) represents the layer normalization operation; W t is the second weight matrix, i.e., the weight matrix used in the encoder to project the temporal features into the new feature space; b t is the bias corresponding to the initial feature at the first time; b1 is the bias output after adjusting the initial feature at the first time; W2 is the third weight matrix, that is, the weight matrix used in the encoder to further process the spatial features to obtain features; W sis the fourth weight matrix, i.e., the weight moment used in the encoder to project spatial features into the new feature space; b s is the bias corresponding to the initial feature of the first space; b2 is the bias output after adjusting the initial feature of the first space.

[0041] Step 103, extracting the temporal initial features and the spatial initial features to obtain final temporal features and spatial features.

[0042] In this step, if Figure 2 As shown in the figure, a multi-granularity feature extraction module is applied in the spatiotemporal branch. In the spatial branch, the spatial multi-granularity partitioning module is used to divide the skeleton joints into multiple sub-regions, and the key spatial features are extracted through maximum pooling; in the temporal branch, the temporal multi-granularity partitioning module and the temporal downsampling module (TDM) are used to aggregate the features of adjacent frames to capture the temporal dynamic characteristics of the action. Combining these modules with the Transformer encoder, the model generates a multi-granularity spatiotemporal representation, which significantly improves the feature extraction effect.

[0043] Features of four granularities are generated through a multi-granularity generation module and feature extraction is performed using the Transformer network.

[0044] The spatial multi-granularity generation module is used to generate features of three different spatial granularities; the features of the three different spatial granularities are , and .

[0045] Will , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the spatial domain level features y s ;in, ; concat(·) represents the concatenation operation; f(·) represents the features output by the Transformer network.

[0046] The time multi-granularity division module is used to generate three features of different time granularities by using the time downsampling module multiple times; the features of the three different time granularities are , and .

[0047] Will , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the time domain level features y t ;in, .

[0048] In the spatial branch, the skeleton joints are divided into multiple sub-regions through the spatial multi-granularity partitioning module, and the key spatial features are extracted using the maximum pooling operation. The spatial multi-granularity features in the encoder q are represented as , , and ; The spatial multi-granularity features in encoder k are expressed as , , and .

[0049] Spatial domain-level features in encoder q ; Spatial domain-level features in encoder k ; Time domain level features in encoder q ; Temporal domain-level features in encoder k .

[0050] In the time branch, the time multi-granularity partitioning module generates features with different time granularities by using the time downsampling module multiple times. The specific formula of the time downsampling module is as follows.

[0051] .

[0052] .

[0053] .

[0054] Among them, x salient Represents a time-significant sequence feature; MaxPool1d(·) represents the maximum pooling operation in the T dimension; represents the initial skeleton feature at time; x differ represents the time difference series feature; Abs(·) represents the absolute value operation; Represents the time series features composed of odd frames; Represents the time series features composed of even frames; It represents the temporal features of different granularities generated after being processed by the temporal downsampling module; FC(·) represents the fully connected layer; after the temporal multi-granularity partitioning module, features of different granularities are obtained.

[0055] Step 104, constructing a loss function of the skeleton behavior recognition model according to the final temporal features and spatial features.

[0056] This embodiment designs a multi-granularity and cross-level contrast loss mechanism. Through multi-granularity contrast loss, samples with a granularity of 1 are selected as anchor points and compared with samples with other granularities to enhance the model's ability to distinguish semantic similarities at different granularities. Through cross-level contrast loss, the fused spatial and temporal multi-granularity representations are used for instance-level comparison, further optimizing model performance and improving the ability to capture complex spatiotemporal features.

[0057] Since features of different granularities are derived from the same skeleton sample, they are similar in high-level semantics, so this embodiment uses the same queue to store them. In addition, the final expression with a granularity of 1 is regarded as an anchor sample, thereby forming a one-to-many multi-granularity loss.

[0058] Exemplarily, this step includes concatenating domain-level features of the same domain to obtain instance-level features y i .

[0059] The multi-granularity contrast loss L is calculated according to the following formula mg .

[0060] .

[0061] Wherein, G is the total number of time granularity or space granularity; g=1,2,…,G; in this embodiment, G is 4; F(·) is the similarity calculation function; Indicates the spatial granularity corresponding to g=1 in encoder q; ; projector sq (·) represents the output of the spatial mapper in encoder q, which consists of two fully connected layers and an activation function; Represents the final spatial initial features of the skeleton sequence in encoder q; represents the g-th spatial granularity in encoder k; ; projector sk (·) represents the output of the spatial mapper in encoder k; is the g-th spatial granularity feature in encoder k; M sg is a set of spatial multi-granularity negative samples; m1 is M sg Negative samples in ; Indicates the time granularity corresponding to g=1 in encoder q; ; projector tq (·) represents the output of the temporal mapper in encoder q, which consists of two fully connected layers and an activation function; represents the final temporal initial features of the skeleton sequence in encoder q; represents the g-th time granularity in encoder k; ; projector tk(·) represents the output of the time mapper in encoder k; is the g-th time granularity feature in encoder k; M tg is a set of time-multi-granularity negative samples; m2 is M tg Negative samples in .

[0062] The cross-level contrast loss L is calculated according to the following formula cross .

[0063] .

[0064] Among them, q s is the final spatial feature in encoder q; q s =projector sq (y sq );y sq is the spatial domain-level feature in encoder q; k i is the final instance-level feature in encoder k; k i =projector ik (y ik ); projector ik (·) represents the output of the instance-level mapper in encoder k; y ik M is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder k; i is the negative sample set of instance-level features; m3 is M i Negative samples in q t is the final temporal feature in encoder q; q t =projector tq (y tq );y tq is the time domain feature in encoder q; i is the final instance-level feature in encoder q; q i =projector iq (y iq ); projector iq (·) represents the output of the instance-level mapper in encoder q; y iq k is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder q; s is the final spatial feature in encoder k; k s =projector sk (y sk );y sk is the spatial domain-level feature in encoder k; t is the final temporal feature in encoder k; k t =projector tk (ytk );y tk is the temporal domain-level feature in encoder k; M t is the negative sample set of the final time feature; m5 is M t Negative samples in .

[0065] The sum of multi-granularity contrast loss and cross-level contrast loss is taken as the loss of the skeleton action recognition model, and the loss function of the skeleton action recognition model is obtained.

[0066] Step 105, iteratively training the skeleton behavior recognition model using the loss function, when the loss function value converges to a preset range, the corresponding skeleton behavior recognition model is used as the final skeleton behavior recognition model.

[0067] The skeleton behavior recognition model is trained using the sum of multi-granularity contrast loss and cross-level contrast loss. When the sum of the multi-granularity contrast loss and the cross-level contrast loss is within a preset loss value range, the corresponding skeleton behavior recognition model is used as the final skeleton behavior recognition model.

[0068] In summary, the skeleton behavior recognition model training method provided in this embodiment is based on spatial and temporal characteristics, and uses a spatial multi-granularity partitioning module to divide the skeleton joints into multiple sub-regions, which reflect the physical structure of the human body. Each sub-region uses a graph maximum pooling method to highlight and retain important features. On the other hand, the temporal multi-granularity partitioning module is composed of multiple temporal downsampling modules, which help to aggregate the features of adjacent frames and enhance motion-related information. Through the multi-granularity partitioning module and the subsequent Transformer network encoder, a spatiotemporal multi-granularity representation containing rich semantic information can be obtained. Then, by fusing the spatial and temporal multi-granularity representations respectively, the representations of the spatial domain and the temporal domain are obtained. All multi-granularity representations are finally combined into instance-level representations. Finally, for semantically similar representations at different granularities, this embodiment uses samples with a granularity of 1 as anchor samples and compares them with other samples to form multi-granularity losses; for domain-level and instance-level representations, this embodiment constructs cross-level comparison pairs to form cross-level losses. The skeleton behavior recognition model can improve its ability to recognize complex behaviors. It can achieve efficient learning without a large amount of labeled data and has strong robustness, making it suitable for scenarios with limited resources or scarce data.

[0069] Example 2.

[0070] Based on the same inventive concept as Example 1, this example provides a skeleton behavior recognition model training system. Since the principle of solving the problem by this system is similar to the aforementioned skeleton behavior recognition model training method, the implementation of this system can refer to the implementation of the skeleton behavior recognition model training method.

[0071] like Figure 3As shown, the skeleton behavior recognition model training system includes: a data enhancement module 10, which is used to perform data enhancement on the skeleton data to obtain a skeleton sequence.

[0072] The first feature extraction module 20 is used to extract the temporal initial features and the spatial initial features of the skeleton sequence.

[0073] The second feature extraction module 30 is used to extract the temporal initial features and the spatial initial features to obtain the final temporal features and spatial features.

[0074] The loss function construction module 40 is used to construct the loss function of the skeleton behavior recognition model according to the final temporal features and spatial features.

[0075] The model training module 50 is used to iteratively train the skeleton behavior recognition model using a loss function. When the loss function value converges to a preset range, the corresponding skeleton behavior recognition model is used as the final skeleton behavior recognition model.

[0076] Exemplarily, the data enhancement module includes: Building blocks used to construct the skeleton sequence x z The expression is: .

[0077] in, represents the skeleton data enhanced by cropping under probability p1; represents the skeleton data enhanced by rotation under probability p2; represents the skeleton data enhanced by flipping with probability p3; represents the skeleton data enhanced by temporal masking under probability p4; represents the skeleton data enhanced by spatial masking under probability p5; x∈R T×V×C , R is a real number, T is the number of time frames, V is the number of skeleton joints, and C is the dimension of each joint; the skeleton data after spatial masking is , ;m spatial represents a mask vector of length P, m spatial ∈{0,1} P ; The skeleton data after using the time mask is , ;m temporal represents a mask vector of length F, m temporal ∈{0,1} F .

[0078] Exemplarily, the first feature extraction module includes: Feature extraction unit, used to extract primary features x of the skeleton sequence using a graph convolutional network i : x i =GCN(x z ,A).

[0079] Among them, x z is the skeleton sequence; A is the adjacency matrix in graph convolution; GCN(·) is the graph convolution operation; , R is a real number, T is the number of time frames, V is the number of joints in the skeleton, C i is the channel dimension of the primary features of the skeleton sequence after being processed by the graph convolutional network.

[0080] Feature deformation operation unit, used to transform the primary features x of the skeleton sequence i Perform a deformation operation to merge the channel dimension with the time dimension and the space dimension respectively to obtain the first time initial feature and the second space initial features ;in, , .

[0081] The feature embedding operation unit is used to perform embedding operations on the first time initial feature and the second space initial feature respectively to obtain the final time initial feature and space initial feature: ; .

[0082] in, is the final temporal initial feature of the skeleton sequence; is the final spatial initial feature of the skeleton sequence; W1 is the first weight matrix; σ(·) represents the ReLU function; LN(·) represents the layer normalization operation; W t is the second weight matrix; b t is the bias corresponding to the initial feature at the first time; b1 is the bias output after adjusting the initial feature at the first time; W2 is the third weight matrix; W s is the fourth weight matrix; b s is the bias corresponding to the initial feature of the first space; b2 is the bias output after adjusting the initial feature of the first space.

[0083] Exemplarily, the second feature extraction module includes: The first feature generation unit is used to generate three features of different spatial granularities using the spatial multi-granularity generation module; wherein the features of the three different spatial granularities are respectively , and .

[0084] The second feature generating unit is used to , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the spatial domain level features y s ;in, ; concat(·) represents the concatenation operation; f(·) represents the features output by the Transformer network.

[0085] The third feature generation unit is used to generate three features of different time granularities by using the time multi-granularity division module and repeatedly using the time downsampling module; wherein the features of the three different time granularities are respectively , and .

[0086] The fourth feature generating unit is used to , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the time domain level features y t ;in, .

[0087] Exemplarily, the loss function building module includes: The fifth feature generation unit is used to connect the domain-level features of the same domain to obtain the instance-level feature y i .

[0088] The first calculation unit is used to calculate the multi-granularity contrast loss L according to the following formula mg : .

[0089] Where G is the total number of time granularity or space granularity; g = 1, 2, …, G; F (·) is the similarity calculation function; Indicates the spatial granularity corresponding to g=1 in encoder q; ; projector sq (·) represents the output of the spatial mapper in encoder q, which consists of two fully connected layers and an activation function; Represents the final spatial initial features of the skeleton sequence in encoder q; represents the g-th spatial granularity in encoder k; ; projector sk (·) represents the output of the spatial mapper in encoder k; is the g-th spatial granularity feature in encoder k; Msg is a set of spatial multi-granularity negative samples; m1 is M sg Negative samples in ; Indicates the time granularity corresponding to g=1 in encoder q; ; projector tq (·) represents the output of the temporal mapper in encoder q, which consists of two fully connected layers and an activation function; represents the final temporal initial features of the skeleton sequence in encoder q; represents the g-th time granularity in encoder k; ; projector tk (·) represents the output of the time mapper in encoder k; is the g-th time granularity feature in encoder k; M tg is a set of time-multi-granularity negative samples; m2 is M tg Negative samples in .

[0090] The second calculation unit is used to calculate the cross-level contrast loss L according to the following formula cross : .

[0091] Among them, q s is the final spatial feature in encoder q; q s =projector sq (y sq );y sq is the spatial domain-level feature in encoder q; k i is the final instance-level feature in encoder k; k i =projector ik (y ik ); projector ik (·) represents the output of the instance-level mapper in encoder k; y ik M is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder k; i is the negative sample set of instance-level features; m3 is M i Negative samples in q t is the final temporal feature in encoder q; q t =projector tq (y tq );y tq is the time domain feature in encoder q; i is the final instance-level feature in encoder q; q i =projector iq (y iq ); projectoriq (·) represents the output of the instance-level mapper in encoder q; y iq k is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder q; s is the final spatial feature in encoder k; k s =projector sk (y sk );y sk is the spatial domain-level feature in encoder k; t is the final temporal feature in encoder k; k t =projector tk (y tk );y tk is the temporal domain-level feature in encoder k; M t is the negative sample set of the final time feature; m5 is M t Negative samples in .

[0092] The loss determination unit is used to take the sum of the multi-granularity contrast loss and the cross-level contrast loss as the loss of the skeleton behavior recognition model to obtain the loss function of the skeleton behavior recognition model.

[0093] For more specific working processes of the above modules, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0094] Example 3.

[0095] This embodiment provides a skeleton behavior recognition method, including: Get the skeleton data.

[0096] The skeleton data is input into a skeleton behavior recognition model to recognize skeleton behavior; wherein the skeleton behavior recognition model is trained according to the skeleton behavior recognition model training method described in Example 1.

[0097] Example 4.

[0098] This embodiment provides a computer device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the steps of the skeleton behavior recognition model training method described in Example 1 are implemented.

[0099] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0100] Example 5.

[0101] This embodiment provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the skeleton behavior recognition model training method described in Example 1 are implemented.

[0102] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0103] Example 6.

[0104] This embodiment provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the steps of the skeleton behavior recognition model training method described in Example 1 are implemented.

[0105] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0106] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the systems, devices, storage media, and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0107] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0108] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0109] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0110] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0111] The present invention has been described in detail above in conjunction with specific implementations and exemplary examples, but these descriptions cannot be understood as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, a variety of equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its implementation methods, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A skeleton behavior recognition model training method, characterized in that: include: Perform data enhancement on the skeleton data to obtain a skeleton sequence; Extracting temporal initial features and spatial initial features of the skeleton sequence; Extracting the initial time features and the initial space features to obtain the final time features and space features; Construct the loss function of the skeleton behavior recognition model based on the final temporal and spatial features; The skeleton behavior recognition model is iteratively trained using the loss function. When the loss function value converges to a preset range, the corresponding skeleton behavior recognition model is used as the final skeleton behavior recognition model. The step of extracting the temporal initial features and the spatial initial features of the skeleton sequence includes: Use graph convolutional networks to extract primary features x of skeleton sequences i : x i =GCN(x z ,A); Among them, x z is the skeleton sequence; A is the adjacency matrix in graph convolution; GCN(·) is the graph convolution operation; , R is a real number, T is the number of time frames, V is the number of joints in the skeleton, C i is the channel dimension of the primary features of the skeleton sequence after being processed by the graph convolutional network; The primary feature x of the skeleton sequence i Perform a deformation operation to merge the channel dimension with the time dimension and the space dimension respectively to obtain the first time initial feature and the second space initial features ;in, , ; The first temporal initial features and the second spatial initial features are embedded respectively to obtain the final temporal initial features and spatial initial features: ； ； in, is the final temporal initial feature of the skeleton sequence; is the final spatial initial feature of the skeleton sequence; W1 is the first weight matrix; σ(·) represents the ReLU function; LN(·) represents the layer normalization operation; W t is the second weight matrix; b t is the bias corresponding to the initial feature at the first time; b1 is the bias output after adjusting the initial feature at the first time; W2 is the third weight matrix; W s is the fourth weight matrix; b s is the bias corresponding to the initial feature of the first space; b2 is the bias output after adjusting the initial feature of the first space; The feature extraction of the initial time feature and the initial space feature to obtain the final time feature and space feature includes: The spatial multi-granularity generation module is used to generate features of three different spatial granularities; the features of the three different spatial granularities are , and ; Will , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the spatial domain level features y s ;in, ; concat(·) represents the concatenation operation; f(·) represents the feature output by the Transformer network; The time multi-granularity division module is used to generate three features of different time granularities by using the time downsampling module multiple times; the features of the three different time granularities are , and ; Will , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the time domain level features y t ;in, ; The loss function of constructing the skeleton behavior recognition model according to the final temporal features and spatial features includes: Connect the domain-level features of the same domain to obtain the instance-level features y i ; The multi-granularity contrast loss L is calculated according to the following formula mg : ； Where G is the total number of time granularity or space granularity; g = 1, 2, …, G; F (·) is the similarity calculation function; Indicates the spatial granularity corresponding to g=1 in encoder q; ; projector sq (·) represents the output of the spatial mapper in encoder q, which consists of two fully connected layers and an activation function; Represents the final spatial initial features of the skeleton sequence in encoder q; represents the g-th spatial granularity in encoder k; ;projector sk (·) represents the output of the spatial mapper in encoder k; is the g-th spatial granularity feature in encoder k; M sg is a set of spatial multi-granularity negative samples; m1 is M sg Negative samples in ; Indicates the time granularity corresponding to g=1 in encoder q; ;projector tq (·) represents the output of the temporal mapper in encoder q, which consists of two fully connected layers and an activation function; represents the final temporal initial features of the skeleton sequence in encoder q; represents the g-th time granularity in encoder k; ; projector tk (·) represents the output of the time mapper in encoder k; is the g-th time granularity feature in encoder k; M tg is a set of time-multi-granularity negative samples; m2 is M tg Negative samples in ; The cross-level contrast loss L is calculated according to the following formula cross : ； Among them, q s is the final spatial feature in encoder q; q s =projector sq (y sq );y sq is the spatial domain-level feature in encoder q; k i is the final instance-level feature in encoder k; k i =projector ik (y ik ); projector ik (·) represents the output of the instance-level mapper in encoder k; y ik M is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder k; i is the negative sample set of instance-level features; m3 is M i Negative samples in q t is the final temporal feature in encoder q; q t =projector tq (y tq );y tq is the time domain feature in encoder q; i is the final instance-level feature in encoder q; q i =projector iq (y iq ); projector iq (·) represents the output of the instance-level mapper in encoder q; y iq k is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder q; s is the final spatial feature in encoder k; k s =projector sk (y sk );y sk is the spatial domain-level feature in encoder k; t is the final temporal feature in encoder k; k t =projector tk (y tk );y tk is the temporal domain-level feature in encoder k; M t is the negative sample set of the final time feature; m5 is M t Negative samples in ; The sum of multi-granularity contrast loss and cross-level contrast loss is taken as the loss of the skeleton action recognition model, and the loss function of the skeleton action recognition model is obtained.

2. The skeleton behavior recognition model training method according to claim 1, characterized in that: The skeleton data is enhanced to obtain a skeleton sequence, including: Construct skeleton sequence x z The expression is: ； in, represents the skeleton data enhanced by cropping under probability p1; represents the skeleton data enhanced by rotation under probability p2; represents the skeleton data enhanced by flipping with probability p3; represents the skeleton data enhanced by temporal masking under probability p4; represents the skeleton data enhanced by spatial masking under probability p5; x∈R T×V×C , R is a real number, T is the number of time frames, V is the number of skeleton joints, and C is the dimension of each joint; the skeleton data after spatial masking is , ;m spatial represents a mask vector of length P, m spatial ∈{0,1} P ; The skeleton data after using the time mask is , ;m temporal represents a mask vector of length F, m temporal ∈{0,1} F .

3. A skeleton behavior recognition model training system, characterized in that: include: A data enhancement module is used to enhance the skeleton data and obtain a skeleton sequence; The first feature extraction module is used to extract the temporal initial features and the spatial initial features of the skeleton sequence; The second feature extraction module is used to extract the time initial feature and the space initial feature to obtain the final time feature and space feature; A loss function construction module is used to construct a loss function of the skeleton behavior recognition model based on the final temporal features and spatial features; A model training module is used to iteratively train the skeleton behavior recognition model using a loss function. When the loss function value converges to a preset range, the corresponding skeleton behavior recognition model is used as the final skeleton behavior recognition model. Wherein, the first feature extraction module includes: Feature extraction unit, used to extract primary features x of the skeleton sequence using a graph convolutional network i : x i =GCN(x z ,A); Among them, x z is the skeleton sequence; A is the adjacency matrix in graph convolution; GCN(·) is the graph convolution operation; , R is a real number, T is the number of time frames, V is the number of joints in the skeleton, C i is the channel dimension of the primary features of the skeleton sequence after being processed by the graph convolutional network; Feature deformation operation unit, used to transform the primary features x of the skeleton sequence i Perform a deformation operation to merge the channel dimension with the time dimension and the space dimension respectively to obtain the first time initial feature and the second space initial features ;in, , ; The feature embedding operation unit is used to perform embedding operations on the first time initial feature and the second space initial feature respectively to obtain the final time initial feature and space initial feature: ； ； in, is the final temporal initial feature of the skeleton sequence; is the final spatial initial feature of the skeleton sequence; W1 is the first weight matrix; σ(·) represents the ReLU function; LN(·) represents the layer normalization operation; W t is the second weight matrix; b t is the bias corresponding to the initial feature at the first time; b1 is the bias output after adjusting the initial feature at the first time; W2 is the third weight matrix; W s is the fourth weight matrix; b s is the bias corresponding to the initial feature of the first space; b2 is the bias output after adjusting the initial feature of the first space; The second feature extraction module comprises: The first feature generation unit is used to generate three features of different spatial granularities using a spatial multi-granularity generation module; wherein the features of the three different spatial granularities are respectively , and ; The second feature generating unit is used to , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the spatial domain level features y s ;in, ; concat(·) represents the concatenation operation; f(·) represents the feature output by the Transformer network; The third feature generation unit is used to generate three features of different time granularities by using the time multi-granularity division module and repeatedly using the time downsampling module; wherein the features of the three different time granularities are respectively , and ; The fourth feature generating unit is used to , , and After being input into the Transformer network respectively, the output features are concatenated to obtain the time domain level features y t ;in, ; The loss function building module includes: The fifth feature generation unit is used to connect the domain-level features of the same domain to obtain the instance-level feature y i ; The first calculation unit is used to calculate the multi-granularity contrast loss L according to the following formula mg : ； Where G is the total number of time granularity or space granularity; g = 1, 2, …, G; F (·) is the similarity calculation function; Indicates the spatial granularity corresponding to g=1 in encoder q; ; projector sq (·) represents the output of the spatial mapper in encoder q, which consists of two fully connected layers and an activation function; Represents the final spatial initial features of the skeleton sequence in encoder q; represents the g-th spatial granularity in encoder k; ; projector sk (·) represents the output of the spatial mapper in encoder k; is the g-th spatial granularity feature in encoder k; M sg is a set of spatial multi-granularity negative samples; m1 is M sg Negative samples in ; Indicates the time granularity corresponding to g=1 in encoder q; ; projector tq (·) represents the output of the temporal mapper in encoder q, which consists of two fully connected layers and an activation function; represents the final temporal initial features of the skeleton sequence in encoder q; represents the g-th time granularity in encoder k; ; projector tk (·) represents the output of the time mapper in encoder k; is the g-th time granularity feature in encoder k; M tg is a set of time-multi-granularity negative samples; m2 is M tg Negative samples in ; The second calculation unit is used to calculate the cross-level contrast loss L according to the following formula cross : ； Among them, q s is the final spatial feature in encoder q; q s =projector sq (y sq );y sq is the spatial domain-level feature in encoder q; k i is the final instance-level feature in encoder k; k i =projector ik (y ik ); projector ik (·) represents the output of the instance-level mapper in encoder k; y ik M is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder k; i is the negative sample set of instance-level features; m3 is M i Negative samples in q t is the final temporal feature in encoder q; q t =projector tq (y tq );y tq is the time domain feature in encoder q; i is the final instance-level feature in encoder q; q i =projector iq (y iq ); projector iq (·) represents the output of the instance-level mapper in encoder q; y iq k is the instance-level feature obtained by connecting the domain-level features of the same domain in encoder q; s is the final spatial feature in encoder k; k s =projector sk (y sk );y sk is the spatial domain-level feature in encoder k; t is the final temporal feature in encoder k; k t =projector tk (y tk );y tk is the temporal domain-level feature in encoder k; M t is the negative sample set of the final time feature; m5 is M t Negative samples in ; The loss determination unit is used to take the sum of the multi-granularity contrast loss and the cross-level contrast loss as the loss of the skeleton behavior recognition model to obtain the loss function of the skeleton behavior recognition model.

4. The skeleton behavior recognition model training system according to claim 3, characterized in that: The data enhancement module comprises: Building blocks used to construct the skeleton sequence x z The expression is: ； in, represents the skeleton data enhanced by cropping under probability p1; represents the skeleton data enhanced by rotation under probability p2; represents the skeleton data enhanced by flipping with probability p3; represents the skeleton data enhanced by temporal masking under probability p4; represents the skeleton data enhanced by spatial masking under probability p5; x∈R T×V×C , R is a real number, T is the number of time frames, V is the number of skeleton joints, and C is the dimension of each joint; the skeleton data after spatial masking is , ;m spatial represents a mask vector of length P, m spatial ∈{0,1} P ; The skeleton data after using the time mask is , ;m temporal represents a mask vector of length F, m temporal ∈{0,1} F .

5. A computer device, characterized in that: It comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the steps of the skeleton behavior recognition model training method described in any one of claims 1-2 are implemented.

6. A computer-readable storage medium, characterized in that: Used to store computer programs; when the computer programs are executed by the processor, the steps of the skeleton behavior recognition model training method described in any one of claims 1-2 are implemented.

Images