Fine-grained sports detection method and device based on deep skeleton feature learning

Abstract

The present invention discloses a fine-grained sports detection method and device based on deep skeleton feature learning, the method comprising: reading a skeleton graph, extracting deep posture features using a graph convolutional network; inputting the deep posture features into a time boundary classifier to predict the boundary probability and action probability of each timestamp, and determining the possible start and end positions according to the boundary probability and the action probability, and generating a candidate nomination set; for the candidate nomination set, re-evaluating the possibility of each candidate nomination by extracting the time relationship of the boundary position, and obtaining a prediction confidence value; removing redundant candidate nominations by non-maximum suppression, and obtaining a final nomination set; inputting the final nomination set into a nomination classification module to estimate the sub-action category of each nomination. The present invention captures the dynamic changes of human body movements, extracts important posture change patterns, effectively characterizes the categories and boundaries of sports movements, and solves the problem of detecting fine-grained sports movements.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a fine-grained sports detection method and device based on deep skeleton feature learning. Background Art

[0002] The goal of the temporal action detection task is to extract the start time, end time and corresponding action category of the action clip in the untrimmed video. This task has long attracted much attention and is regarded as one of the important challenges in the field of computer vision. In recent years, with the rapid development of deep learning technology, researchers have continuously proposed new methods and techniques to solve this problem. The application of deep learning technology has brought new breakthroughs and progress to the task of temporal action detection, providing powerful tools and methods to solve this difficult problem. These methods can be roughly divided into two categories, namely staged and joint methods. Among the staged methods, the temporal action nomination generation task has become one of the most concerned video action understanding tasks. With the in-depth exploration of this field, fine-grained temporal action detection has gradually emerged and become a new research hotspot. The development of this field has brought more possibilities and new research directions for temporal action detection, providing a broad space for future research and technological breakthroughs.

[0003] Compared with traditional temporal action detection, fine-grained action detection requires more detailed and accurate action representation to achieve more accurate positioning of the start and end timestamps of sub-actions. In the study of fine-grained temporal action detection, researchers face multiple complex challenges, including background interference, distinction of highly similar actions, and changes in action duration. These challenges make the study of fine-grained temporal action detection more complicated, requiring higher-level methods and techniques to deal with these problems. To overcome these challenges, recent studies have explored various innovative methods. Among them, researchers proposed an innovative locally consistent deformable convolution method that effectively captures motion information in videos by exploiting the changes in receptive fields and imposing local coherence constraints. The model avoids the traditional approach of using independent spatial and temporal streams by jointly learning spatiotemporal features. Unlike learning the temporal component in pixel space (such as optical flow), this method learns from the feature space, making the generated features more flexible and can be used in combination with other long-term modeling networks, such as ST-CNN, GC-DCNN, and MS-TCNN. A study proposed an innovative learning method that introduces a fundamental inductive bias for weakly supervised temporal action detection tasks by comparing the differences between sequences in context, which helps to accurately identify coherent action instances. Specifically, through a differentiable dynamic programming formula, two complementary comparison objectives are designed, namely the fine-grained sequence distance (FSD) comparison and the longest common subsequence (LCS) comparison. In the FSD comparison, the model is able to mine small differences between sequences through matching, insertion, and deletion operators. In the LCS comparison, the mining of common action patterns is achieved by finding the longest common subsequence between two videos. The two comparison modules enhance each other and share the advantages of distinguishing action-background separation, while alleviating the task gap between classification and positioning.

[0004] However, most of the existing temporal action detection methods are based on RGB data, and extract features from image sequences by designing appearance and optical flow models to distinguish target action instances from the background. However, these methods perform poorly on the task of fine-grained sports action detection due to the changes in human scale and appearance, the high similarity between sub-action categories, and the presence of uniform background. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a fine-grained sports detection method and device based on deep skeleton feature learning to improve the above-mentioned problems.

[0006] The embodiment of the present invention provides a fine-grained sports detection method based on deep skeleton feature learning, which includes:

[0007] S101, reading a skeleton graph for presenting a topological structure of human joints, and extracting deep posture features for sub-action instance detection from the skeleton graph using a graph convolutional network; the deep posture features include a timestamp;

[0008] S102, inputting the deep gesture feature into a temporal boundary classifier to predict the boundary probability and action probability of each timestamp, and determining possible start and end positions according to the boundary probability and the action probability, and then combining all selected start and end positions together to generate a candidate nomination set;

[0009] S103, for the candidate nomination set, re-evaluate the possibility of each candidate nomination by extracting the time relationship of the boundary position, and obtain a prediction confidence value of each candidate nomination;

[0010] S104, according to the predicted confidence value of each candidate nomination, redundant candidate nominations are removed by non-maximum suppression to obtain a final nomination set;

[0011] S105, inputting the final nomination set into a trained nomination classification module to estimate the sub-action category of each nomination, thereby achieving fine-grained sports detection according to the sub-action category.

[0012] Preferably, step S101 specifically includes:

[0013] Use a 1×1 convolutional layer to embed the input skeleton graph tensor into a higher feature dimension to obtain the feature tensor ; Where C is the original feature dimension, corresponding to the 2D position coordinates and confidence scores of each joint, M is the dimension of the joint channel, corresponding to the number of joints, and T is the time channel dimension, corresponding to the time length of each skeleton sequence;

[0014] The feature tensor is subjected to feature extraction by a posture feature extractor to obtain a deep posture feature ; Among them, the posture feature extractor includes multiple stacked processing blocks, each processing block includes a spatial convolution layer and a temporal convolution layer, so as to obtain deep posture features by performing graph convolution operations on the feature tensor in the spatial and temporal dimensions through the topological relationship defined based on the adjacency matrix; the output of each spatial convolution layer and temporal convolution layer is processed by batch normalization and ReLU activation function, and a residual connection structure is adopted to combine the output of ReLU activation with the input feature and feed it to the next processing block.

[0015] Preferably, the extraction process of the posture feature extractor is expressed as:

[0016]

[0017] in, Representation pose feature extractor The learnable parameters of Represents the output deep pose feature.

[0018] Preferably, step S102 specifically includes:

[0019] The deep posture feature As input, it passes through a 3-layer Conv1D network; each layer of the Conv1D network performs a 1D convolution with a kernel size of 3 along the time dimension, followed by a ReLU activation operation, and passes the obtained features to the next layer. The number of nodes in the last layer is set to 3, and the output features are activated by the Sigmoid function to predict the start, end, and action probabilities of each timestamp. ;in, are the start probability, end probability and action probability of each timestamp respectively;

[0020] Based on the start probability, end probability and action probability, a filtering strategy is used to determine possible start and end positions;

[0021] Combine all the determined start and end positions to generate a candidate nomination set .

[0022] Preferably, the filtering strategy is: by setting a boundary probability threshold, determining the time position where the boundary probability is greater than the threshold as the possible start or end position; or,

[0023] By selecting a time position with a boundary probability peak, if the boundary probability of a time position is simultaneously greater than its adjacent previous and next time positions, it is determined to be a possible boundary position.

[0024] Preferably, step S103 specifically includes:

[0025] Expand the start and end timestamps of each candidate nomination , ,in is the length of time for candidate nominations, It is the extension part. t s is the position of the initial start timestamp, t e The position of the initial end timestamp;

[0026] According to the original deep pose features Extended start timestamp and end timestamp , resample the posture feature of each candidate nomination to obtain the extended posture feature;

[0027] The extended posture features are input into the nomination refinement module; wherein, in the nomination refinement module, an average pooling operation is first performed along the joint channel to convert the extended posture features into a two-dimensional feature matrix, then the temporal relationship is encoded by three layers of one-dimensional convolution along the time dimension to capture higher-level semantic information, and then a maximum pooling process is performed along the time channel to obtain the final feature vector, and finally, the FC network is used to predict the confidence score of each candidate nomination; the calculation process of the nomination refinement module is defined as follows:

[0028]

[0029] in, represents the extended posture feature of the kth candidate nomination, and represents average pooling and maximum pooling operations, represents three stacked convolutional layers, represents FC network and Sigmoid activation, represents the confidence score of the kth candidate nomination prediction.

[0030] Preferably, in step S104, a Soft-NMS algorithm is used to suppress redundant results; wherein:

[0031] First, sort all the results according to their corresponding confidence scores, and calculate the t-IoU overlap between the nomination with the highest confidence score and all the remaining nominations;

[0032] Then, the confidence scores of the results with high t-IoU overlap are attenuated, and this operation is repeated for all remaining nomination cycles to obtain the final re-ranked candidate nomination set: , where N is the final number of nominations used for retrieval.

[0033] Preferably, step S105 specifically includes:

[0034] Adaptive pooling is used along the temporal channel to align the different temporal lengths of pose features in the training batch;

[0035] Flatten the joint channel and time channel, and perform 1D convolution on the feature channel;

[0036] The likelihood of the sub-action category is estimated through the FC network and Softmax activation to estimate the sub-action category of each nomination; the computational process of sub-action classification is defined as:

[0037]

[0038] in, is from The pose features of each nomination extracted from; Represents adaptive pooling and flattening operations; refers to the convolution operator; Represents the FC layer and softmax activation, which is used to predict the sub-action category probability.

[0039] Preferably, for the nomination classification module, three losses are used to optimize its learning parameters, including nomination generation loss , confidence prediction loss and the nomination classification loss ; Nomination generation loss Loss of motion , Start time positioning loss and end time positioning loss Composition; its expression is:

[0040]

[0041] Among them, the binary logistic regression loss function For all three loss components, is a hyperparameter used to balance the impact of positive and negative samples during training;

[0042] Confidence prediction loss Defined as:

[0043]

[0044] in Indicates confidence scores of the predicted sub-action nominations; Indicated by The calculated t-IoU score between a predicted sub-action nomination and its nearest true action instance; is the number of sub-action instances used for training;

[0045] Nomination classification loss It is expressed as:

[0046]

[0047] in and Represents the true sub-action category and predicted sub-action category of the i-th action instance; is the number of sub-action instances used for classifier training.

[0048] The embodiment of the present invention also provides a fine-grained sports detection device based on deep skeleton feature learning, which includes:

[0049] A feature extraction unit, configured to read a skeleton graph for presenting a topological structure of a human joint, and extract deep posture features for sub-action instance detection from the skeleton graph using a graph convolutional network; the deep posture features include a timestamp;

[0050] a candidate nomination set generating unit, configured to input the deep posture feature into a temporal boundary classifier to predict the boundary probability and the action probability of each timestamp, determine possible start and end positions according to the boundary probability and the action probability, and then combine all selected start and end positions to generate a candidate nomination set;

[0051] A prediction confidence value calculation unit, used to re-evaluate the possibility of each candidate nomination by extracting the time relationship of the boundary position of the candidate nomination set, and obtain a prediction confidence value of each candidate nomination;

[0052] A final nomination set generating unit is used to remove redundant candidate nominations through non-maximum suppression according to the predicted confidence value of each candidate nomination, so as to obtain a final nomination set;

[0053] The sub-action classification unit is used to input the final nomination set into the trained nomination classification module to estimate the sub-action category of each nomination, and then realize fine-grained sports detection according to the sub-action category.

[0054] Compared with the prior art, the present invention has at least the following advantages:

[0055] 1) This invention overcomes the confusion of RGB features mixed with a large amount of background information, directly extracts the feature representation of posture changes from the human skeleton sequence, and can more effectively solve the recognition problem between highly similar sports movements;

[0056] 2) This paper designs and implements an effective deep posture feature extractor, which captures the motion change information implicit in the human skeleton sequence, extracts important posture change patterns, and effectively characterizes the categories and boundaries of sports movements, thereby solving the problem of fine-grained sports movement detection;

[0057] 3) The present invention designs and implements a reasonable nomination refinement module, which effectively filters out background segment instances and transition segment instances in the candidate nomination set by extracting important intrinsic patterns from the skeleton sequence and performing temporal saliency evaluation, thereby improving the nomination generation quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A schematic diagram of a process flow of a fine-grained sports detection method based on deep skeleton feature learning provided by the first embodiment of the present invention;

[0060] Figure 2 A specific principle diagram of a fine-grained sports detection method based on deep skeleton feature learning provided by the first embodiment of the present invention;

[0061] Figure 3 This is the overall structure diagram of the posture feature extractor;

[0062] Figure 4 A structural diagram of the module for nomination refinement;

[0063] Figure 5 A structural diagram of the nomination classification module;

[0064] Figure 6 A schematic diagram of the structure of a fine-grained sports detection device based on deep skeleton feature learning provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] See also Figure 1 and Figure 2 The embodiment of the present invention provides a fine-grained sports detection method based on deep skeleton feature learning, which includes:

[0067] S101, reading a skeleton graph for presenting a topological structure of human body joints, and using a graph convolutional network to extract deep posture features for sub-action instance detection from the skeleton graph; the deep posture features include a timestamp.

[0068] In this embodiment, in order to fully describe the dynamic changes of human motion, a skeleton graph is first established to present the topological structure of human joints, and a graph convolutional network is used to extract deep posture features for sub-action instance detection. Among them, in the spatial dimension, human joints and their connected joint points are regarded as vertices and edges. In the time dimension, two vertices representing the same joints in adjacent frames are connected by edges. By performing graph convolution operations in both spatial and temporal dimensions, deep structural features and motion patterns can be learned from the trajectories of these joints, making the representation of dynamic changes in human motion richer and more effective.

[0069] The specific implementation is:

[0070] First, a 1×1 convolutional layer is used to embed the input skeleton image tensor into a higher feature dimension to obtain the feature tensor ; Where C is the original feature dimension, usually 3, corresponding to the 2D position coordinates and confidence scores of each joint, M is the dimension of the joint channel, corresponding to the number of joints, usually 24, and T is the time channel dimension, corresponding to the time length of each skeleton sequence.

[0071] Then, the feature tensor is extracted by the posture feature extractor to obtain the deep posture feature ; Among them, the posture feature extractor includes multiple (such as ten) stacked processing blocks, each processing block includes a spatial convolution layer and a temporal convolution layer, so as to obtain deep posture features by performing graph convolution operations on the feature tensor in the spatial and temporal dimensions through the topological relationship defined based on the adjacency matrix; the output of each spatial convolution layer and temporal convolution layer is processed by batch normalization and ReLU activation function. In order to stabilize the training, a residual connection structure is adopted to combine the output of ReLU activation with the input feature and feed it to the next processing block.

[0072] The structure of the posture feature extractor based on skeleton sequence is as follows Figure 3 As shown, the extraction process can be expressed as:

[0073]

[0074] in, Representation pose feature extractor The learnable parameters of Represents the output deep pose feature.

[0075] S102, inputting the deep gesture features into a temporal boundary classifier to predict the boundary probability and action probability of each timestamp, and determining possible start and end positions according to the boundary probability and the action probability, and then combining all selected start and end positions together to generate a candidate nomination set.

[0076] In this embodiment, a time boundary classifier is constructed to identify the boundary probability and action probability of each timestamp.

[0077] Specifically, firstly, the deep posture features of the previous step As input, it passes through a 3-layer Conv1D network. Each layer of the Conv1D network performs a 1D convolution with a kernel size of 3 along the time dimension, followed by a ReLU activation operation, and passes the obtained features to the next layer. The number of nodes in the last layer is set to 3, and the output features are activated by the Sigmoid function to predict the start, end, and action probabilities of each timestamp. ;in, are the start probability, end probability and action probability of the i-th timestamp respectively;

[0078] Then, based on the predicted boundary probability and action probability, a filtering strategy is used to determine the possible start and end positions. The filtering strategy is: by setting a boundary probability threshold, the time position with a boundary probability greater than the threshold is determined as the possible start or end position; or by selecting a time position with a boundary probability peak, if the boundary probability of the time position is greater than its adjacent previous and next time positions at the same time, it is determined as a possible boundary position.

[0079] Finally, all selected start and end positions are combined to generate a candidate nomination set .

[0080] S103 , for the candidate nomination set, re-evaluate the possibility of each candidate nomination by extracting the time relationship of the boundary position, and obtain a prediction confidence value of each candidate nomination.

[0081] In this embodiment, in order to further explore and optimize the evolution of action instances in the entire video, a nomination refinement module is designed. This module re-evaluates the possibility of candidate nominations by extracting the temporal relationship of boundary positions. The process of the nomination refinement module is as follows: Figure 4 The specific implementation is as follows:

[0082] First, expand the start and end timestamps of each candidate nomination , ,in is the length of time for candidate nominations, It is the extension part. t s is the position of the initial start timestamp, t e The position of the initial end timestamp;

[0083] Then, according to the original deep pose features Extended start timestamp and end timestamp , resample the posture feature of each candidate nomination to obtain the extended posture feature;

[0084] Finally, the extended posture features are input into the nomination refinement module; wherein, in the nomination refinement module, firstly, an average pooling operation is performed along the joint channel to convert the extended posture features into a two-dimensional feature matrix, then, the temporal relationship is encoded by three layers of one-dimensional convolution along the time dimension to capture higher-level semantic information, and then a maximum pooling process is performed along the time channel to obtain the final feature vector, and finally, the FC network is used to predict the confidence score of each candidate nomination; the calculation process of the nomination refinement module is defined as follows:

[0085]

[0086] in, represents the extended posture feature of the kth candidate nomination, and represents average pooling and maximum pooling operations, represents three stacked convolutional layers, represents FC network and Sigmoid activation, represents the confidence score of the kth candidate nomination prediction.

[0087] S104: According to the predicted confidence value of each candidate nomination, redundant candidate nominations are removed by non-maximum suppression to obtain a final nomination set.

[0088] In this embodiment, redundant nominations may be removed by non-maximum suppression. The goal of suppressing redundant nominations is to achieve a higher recall rate with fewer nominations.

[0089] Specifically, the Soft-NMS algorithm can be used to suppress redundant results.

[0090] The specific implementation method is: first, sort all the results according to the corresponding confidence scores, and calculate the t-IoU overlap between the nomination with the highest confidence score and all the remaining nominations. After that, the Soft-NMS algorithm attenuates the scores of the results with high t-IoU overlap, and repeats this operation for all the remaining nominations. The final re-ranked candidate nomination set can be obtained as , where N is the final number of nominations used for retrieval.

[0091] S105, inputting the final nomination set into a trained nomination classification module to estimate the sub-action category of each nomination, thereby achieving fine-grained sports detection according to the sub-action category.

[0092] Since the background information of different sub-action instances is very similar, the distinction between different sub-action instances depends on the movement of joints or body parts. Therefore, during the training of the sub-action classifier, we also only use the posture features of the sub-action instances as input to learn the classification model.

[0093] The specific implementation method is: First, adaptive pooling is used along the time channel to align the different time lengths of the posture features in the training batch. Then the joint channel and the time channel are flattened, and the feature channel is 1D convolved. Finally, the possibility of the sub-action category is estimated through the FC network and Softmax activation. The calculation process of the sub-action nomination classification can be defined as:

[0094]

[0095] in, is from The pose features of each nomination are extracted from . Represents adaptive pooling and flattening operations. It refers to the convolution operator. represents the FC layer and softmax activation, which is used to predict the probability of sub-action categories. The module framework of the sub-action classifier is as follows Figure 5 shown.

[0096] In this embodiment, in order to improve the classification accuracy of the sub-action classifier, three losses can be used to optimize the learning parameters of the method. It includes nomination generation loss , confidence prediction loss and the nomination classification loss . Nomination generation loss It consists of action loss, start time positioning loss and end time positioning loss. It can be expressed as:

[0097]

[0098] Among them, the binary logistic regression loss function For all three loss components, is a hyperparameter used to balance the impact of positive and negative samples during training.

[0099] Confidence prediction loss It is to evaluate the error between the predicted confidence value and the true confidence value of each sub-action nomination, where the true confidence value is calculated by the intersection between each predicted nomination and its sub-action instance. It can be defined as:

[0100]

[0101] in Indicates The confidence score of each predicted sub-action nomination. Indicated by The t-IoU score is calculated between the predicted sub-action nominations and their nearest true action instance. is the number of sub-action instances used for training.

[0102] To train the nomination classification module, we use the nomination classification loss Optimize the parameters of the nomination classification network. It can be expressed as:

[0103]

[0104] in and represents the true sub-action category and predicted sub-action category of the i-th action instance. is the number of sub-action instances used for classifier training.

[0105] In the inference phase, the top K high-quality nominations are selected from the obtained nominations according to their confidence scores. Finally, the trained nomination classification module is used to estimate its sub-action category.

[0106] In summary, compared with the prior art, the embodiments of the present invention have at least the following advantages:

[0107] 1) This invention overcomes the confusion of RGB features mixed with a large amount of background information, directly extracts the feature representation of posture changes from the human skeleton sequence, and can more effectively solve the recognition problem between highly similar sports movements;

[0108] 2) This paper designs and implements an effective deep posture feature extractor, which captures the motion change information implicit in the human skeleton sequence, extracts important posture change patterns, and effectively characterizes the categories and boundaries of sports movements, thereby solving the problem of fine-grained sports movement detection;

[0109] 3) The present invention designs and implements a reasonable nomination refinement module, which effectively filters out background segment instances and transition segment instances in the candidate nomination set by extracting important intrinsic patterns from the skeleton sequence and performing temporal saliency evaluation, thereby improving the nomination generation quality.

[0110] See also Figure 6 The second embodiment of the present invention further provides a fine-grained sports detection device based on deep skeleton feature learning, which includes:

[0111] A feature extraction unit 210 is used to read a skeleton graph for presenting a topological structure of a human body joint, and extract deep posture features for sub-action instance detection from the skeleton graph using a graph convolutional network; the deep posture features include a timestamp;

[0112] A candidate nomination set generation unit 220 is used to input the deep posture feature into a temporal boundary classifier to predict the boundary probability and action probability of each timestamp, and determine possible start and end positions according to the boundary probability and the action probability, and then combine all selected start and end positions together to generate a candidate nomination set;

[0113] A prediction confidence value calculation unit 230 is used to re-evaluate the possibility of each candidate nomination by extracting the time relationship of the boundary position of the candidate nomination set to obtain a prediction confidence value of each candidate nomination;

[0114] A final nomination set generating unit 240 is used to remove redundant candidate nominations by non-maximum suppression according to the predicted confidence value of each candidate nomination, so as to obtain a final nomination set;

[0115] The sub-action classification unit 250 is used to input the final nomination set into the trained nomination classification module to estimate the sub-action category of each nomination, and then realize fine-grained sports detection according to the sub-action category.

[0116] The third embodiment of the present invention also provides a fine-grained sports motion detection based on deep skeleton feature learning, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the fine-grained sports motion detection method based on deep skeleton feature learning as described above.

[0117] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0119] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, electronic device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fine-grained sports detection method based on deep skeleton feature learning, characterized in that: include: S101, reading a skeleton graph for presenting a topological structure of human joints, and extracting deep posture features for sub-action instance detection from the skeleton graph using a graph convolutional network; The deep posture feature includes a timestamp; wherein step S101 specifically includes: using a 1×1 convolutional layer to embed the tensor of the input skeleton image into a higher feature dimension to obtain a feature tensor ; Where C is the original feature dimension, corresponding to the 2D position coordinates and confidence scores of each joint, M is the dimension of the joint channel, corresponding to the number of joints, and T is the time channel dimension, corresponding to the time length of each skeleton sequence; the feature tensor is extracted by the posture feature extractor to obtain the deep posture feature ; Among them, the posture feature extractor includes multiple stacked processing blocks, each of which includes a spatial convolution layer and a temporal convolution layer, so as to obtain deep posture features by performing graph convolution operations on feature tensors in spatial and temporal dimensions through topological relationships defined based on adjacency matrices; the output of each spatial convolution layer and temporal convolution layer is processed by batch normalization and ReLU activation function, and a residual connection structure is adopted to combine the output of ReLU activation with the input feature and feed it to the next processing block; S102, inputting the deep gesture feature into a temporal boundary classifier to predict the boundary probability and action probability of each timestamp, and determining possible start and end positions according to the boundary probability and the action probability, and then combining all selected start and end positions together to generate a candidate nomination set; S103, for the candidate nomination set, re-evaluate the possibility of each candidate nomination by extracting the time relationship of the boundary position, and obtain a prediction confidence value of each candidate nomination; S104, according to the predicted confidence value of each candidate nomination, redundant candidate nominations are removed by non-maximum suppression to obtain a final nomination set; S105, inputting the final nomination set into a trained nomination classification module to estimate the sub-action category of each nomination, thereby achieving fine-grained sports detection according to the sub-action category.

2. The fine-grained sports detection method based on deep skeleton feature learning according to claim 1 is characterized in that: The extraction process of the posture feature extractor is expressed as: in, Representation pose feature extractor The learnable parameters of Represents the output deep pose feature.

3. The fine-grained sports detection method based on deep skeleton feature learning according to claim 1 is characterized in that: Step S102 specifically includes: The deep posture feature As input, it passes through a 3-layer Conv1D network; each layer of the Conv1D network performs a 1D convolution with a kernel size of 3 along the time dimension, followed by a ReLU activation operation, and passes the obtained features to the next layer. The number of nodes in the last layer is set to 3, and the output features are activated by the Sigmoid function to predict the start, end, and action probabilities of each timestamp. ;in, are the start probability, end probability and action probability of the i-th timestamp respectively; Based on the start probability, end probability and action probability, a filtering strategy is used to determine possible start and end positions; Combine all the determined start and end positions to generate a candidate nomination set .

4. The fine-grained sports detection method based on deep skeleton feature learning according to claim 3 is characterized in that: The filtering strategy is: by setting a boundary probability threshold, the time position where the boundary probability is greater than the threshold is determined as a possible start or end position; or, By selecting a time position with a boundary probability peak, if the boundary probability of a time position is simultaneously greater than its adjacent previous and next time positions, it is determined to be a possible boundary position.

5. The fine-grained sports detection method based on deep skeleton feature learning according to claim 1 is characterized in that: Step S103 specifically includes: Expand the start and end timestamps of each candidate nomination , ,in is the length of time for candidate nominations, It is the extension part. t s is the position of the initial start timestamp, t e The position of the initial end timestamp; According to the original deep pose features Extended start timestamp and end timestamp , resample the posture feature of each candidate nomination to obtain the extended posture feature; The extended posture features are input into the nomination refinement module; wherein, in the nomination refinement module, an average pooling operation is first performed along the joint channel to convert the extended posture features into a two-dimensional feature matrix, then the temporal relationship is encoded by three layers of one-dimensional convolution along the time dimension to capture higher-level semantic information, and then a maximum pooling process is performed along the time channel to obtain the final feature vector, and finally, the FC network is used to predict the confidence score of each candidate nomination; the calculation process of the nomination refinement module is defined as follows: in, represents the extended posture feature of the kth candidate nomination, and represents average pooling and maximum pooling operations, represents three stacked convolutional layers, represents FC network and Sigmoid activation, represents the confidence score of the k-th candidate nomination prediction.

6. The fine-grained sports detection method based on deep skeleton feature learning according to claim 5 is characterized in that: In step S104, the Soft-NMS algorithm is used to suppress redundant results; wherein: First, sort all the results according to their corresponding confidence scores, and calculate the t-IoU overlap between the nomination with the highest confidence score and all the remaining nominations; Then, the confidence scores of the results with high t-IoU overlap are attenuated, and this operation is repeated for all remaining nomination cycles to obtain the final re-ranked candidate nomination set: , where N is the final number of nominations used for retrieval.

7. The fine-grained sports detection method based on deep skeleton feature learning according to claim 1 is characterized in that: Step S105 specifically includes: Adaptive pooling is used along the temporal channel to align the different temporal lengths of pose features in the training batch; Flatten the joint channel and time channel, and perform 1D convolution on the feature channel; The likelihood of the sub-action category is estimated through the FC network and Softmax activation to estimate the sub-action category of each nomination; the computational process of sub-action classification is defined as: in, is from The pose features of each nomination extracted from; Represents adaptive pooling and flattening operations; refers to the convolution operator; Represents the FC layer and softmax activation, which is used to predict the sub-action category probability.

8. The fine-grained sports detection method based on deep skeleton feature learning according to claim 1 is characterized in that: For the nomination classification module, three losses are used to optimize its learning parameters, including nomination generation loss , confidence prediction loss and the nomination classification loss ; Nomination generation loss Loss of motion , Start time positioning loss and end time positioning loss Composition; its expression is: Among them, the binary logistic regression loss function For all three loss components, is a hyperparameter used to balance the impact of positive and negative samples during training; Confidence prediction loss Defined as: in Indicates confidence scores of the predicted sub-action nominations; Indicated by The computed t-IoU score between a predicted sub-action nomination and its nearest true action instance; is the number of sub-action instances used for training; Nomination classification loss It is expressed as: in and Represents the true sub-action category and predicted sub-action category of the i-th action instance; is the number of sub-action instances used for classifier training.

9. A fine-grained sports detection device based on deep skeleton feature learning, characterized in that: include: A feature extraction unit, configured to read a skeleton graph for presenting a topological structure of human joints, and extract deep posture features for sub-action instance detection from the skeleton graph using a graph convolutional network; The deep posture feature includes a timestamp; wherein the feature extraction unit is specifically used to: use a 1×1 convolution layer to embed the tensor of the input skeleton image into a higher feature dimension to obtain a feature tensor ; Where C is the original feature dimension, corresponding to the 2D position coordinates and confidence scores of each joint, M is the dimension of the joint channel, corresponding to the number of joints, and T is the time channel dimension, corresponding to the time length of each skeleton sequence; the feature tensor is extracted by the posture feature extractor to obtain the deep posture feature ; Among them, the posture feature extractor includes multiple stacked processing blocks, each of which includes a spatial convolution layer and a temporal convolution layer, so as to obtain deep posture features by performing graph convolution operations on feature tensors in spatial and temporal dimensions through topological relationships defined based on adjacency matrices; the output of each spatial convolution layer and temporal convolution layer is processed by batch normalization and ReLU activation function, and a residual connection structure is adopted to combine the output of ReLU activation with the input feature and feed it to the next processing block; a candidate nomination set generating unit, configured to input the deep posture feature into a temporal boundary classifier to predict the boundary probability and the action probability of each timestamp, determine possible start and end positions according to the boundary probability and the action probability, and then combine all selected start and end positions to generate a candidate nomination set; A prediction confidence value calculation unit, used to re-evaluate the possibility of each candidate nomination by extracting the time relationship of the boundary position of the candidate nomination set, and obtain a prediction confidence value of each candidate nomination; A final nomination set generating unit is used to remove redundant candidate nominations through non-maximum suppression according to the predicted confidence value of each candidate nomination, so as to obtain a final nomination set; The sub-action classification unit is used to input the final nomination set into the trained nomination classification module to estimate the sub-action category of each nomination, and then realize fine-grained sports detection according to the sub-action category.

Images