An action evaluation method based on temporal dynamic adaptation and hierarchical comparison

Abstract

The present invention relates to an action evaluation method based on temporal dynamic adaptation and hierarchical comparison, comprising the following steps: step S1: acquiring and preprocessing a data set, and training a key point detection algorithm based on computer vision; step S2: using a modified key point detection algorithm to identify and extract skeletal key points from an input video image, and outputting the information; step S3: using a limb neatness algorithm based on cosine similarity to process the information of the key points and calculate limb neatness; step S4: using a joint neatness algorithm based on distance measurement to calculate joint neatness, and calculating comprehensive neatness based on limb and joint neatness to complete the final evaluation.

Description

Technical Field

[0001] The present invention relates to the fields of image and video processing and computer vision technology, and in particular to an action evaluation method based on temporal dynamic adaptation and hierarchical comparison. Background Art

[0002] With the rapid development of artificial intelligence and computer vision in recent years, researchers have been focusing on how to apply these cutting-edge technologies to real-world scenarios. Using machines to automatically assess human motion is a research direction with high application value and a wide range of scenarios. However, in many specialized scenarios, human motion variations are extremely subtle, resulting in poor intra-class discrimination, which significantly limits the effectiveness of network models. Effectively mining fine-grained motion variations has become a key research focus and challenge in human motion recognition and assessment.

[0003] Although some methods are dedicated to studying segmentation and optimizing fine-grained scene features, existing methods are limited to analyzing the global features of action videos. Although global features can provide overall motion changes of the human body, they cannot fully capture the subtle changes and local details of each time step in the motion sequence. Especially in videos with highly consistent motion types, the differences between human bodies often exist in the temporal changes of a small part of the sub-action sequence and the motion details of the local space. This requires attention to the temporal relationship of the action and the changes in local details. Action videos contain rich multi-level feature information, such as global, local, temporal, spatial and other dimensions, including low-level preliminary actions and high-level action categories. This information is conducive to achieving more comprehensive and accurate motion evaluation and analysis. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an action evaluation method based on temporal dynamic adaptation and hierarchical comparison, which can effectively analyze and optimize the hierarchical features of the human body in human action videos, focus on important action nodes, align the temporal actions of two videos, and improve the accuracy of action difference comparison and evaluation analysis.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: an action evaluation method based on temporal dynamic adaptation and hierarchical comparison, comprising the following steps:

[0006] Step S1: Obtain image sequences or videos of scenes related to action evaluation, annotate the action categories and quality scores contained therein to construct a dataset;

[0007] Step S2: Construct X3D, which is pre-trained on a large-scale action dataset, as the backbone network to extract the spatiotemporal features of paired videos, and design a temporal alignment video encoder to dynamically align the spatiotemporal features;

[0008] Step S3: Perform multi-dimensional pooling on the spatiotemporal features of step S2 to obtain multi-level features, and design a hierarchical perception feature aggregation network to enhance the features of each level;

[0009] Step S4: Design a Transformer-based motion difference comparison decoder to analyze the feature differences at each level of the paired videos and output the final analysis and evaluation results.

[0010] In a preferred embodiment, step S1 specifically includes the following steps:

[0011] Step S11: Obtain coherent pictures or videos of application scenarios related to human motion assessment from the Internet to initially form a data set;

[0012] Step S12: Clean the data set, label the categories and quality scores of human actions based on the goals of the current application scenario on the existing data labels, and complete the construction of the data set;

[0013] Step S13: Divide the constructed data set into a training set and a test set according to an appropriate predetermined ratio.

[0014] In a preferred embodiment, step S2 specifically includes the following steps:

[0015] Step S21: For an input query video X, randomly select sample videos Z with the same frame length l from the training set to form input video pairs, and split them into N segments containing 16 frames;

[0016] Step S22: Pre-train the X3D backbone network based on the large-scale human motion dataset Kinetics-400 to obtain excellent motion information capture capabilities, and then use the network to extract segment-level spatiotemporal features of the input video pair Where T, H×W, and D represent the time step, spatial block size, and feature size, respectively;

[0017] Step S23: Design a temporal alignment video encoder W. The temporal alignment video encoder E first uses a projection block containing two fully connected layers with batch normalization and a ReLU activation function to project F into an intermediate embedding of size T×H×W×D', where D' is the encoding feature size. Then, sine and cosine position encoding is added to each block of the intermediate embedding and a 3-layer Transformer encoder is used to model the spatiotemporal contextual relationship between all blocks. Finally, a temporal transformer consisting of a localization network L and a temporal transformer T is used to align the temporal actions between the query video and the sample video. The conversion process is expressed as follows:

[0018] f'=T Ψ (f),Ψ=(ω,b)=L(f)

[0019] Where f is the feature to be converted, f' represents the timing alignment result, and Ψ = (ω, b) represents the generated perturbation parameter;

[0020] Step S24: Introducing timing cycle consistency loss To optimize the timing alignment coding described in step S23, for each segment First calculate its soft nearest neighbor Then trace the location Nearest neighbor Feature pairs It should meet the temporal consistency, that is, The above loss learning process is expressed as follows:

[0021]

[0022]

[0023]

[0024] Among them, f' X and f' Z That is, the features of the query video and the sample video after temporal alignment, To calculate the similarity factor of the nearest neighbor, g(·) represents the calculation of cosine similarity, and represents the j-th and k-th segments of the sample video Z, represents the kth segment of the query video X, and N is the number of segments in step S21.

[0025] In a preferred embodiment, step S3 specifically includes the following steps:

[0026] Step S31: Obtain block-shaped spatiotemporal features after dynamic alignment in step S2 In the space-time dimension and space dimension Perform global average pooling to obtain global features and time series characteristics

[0027] Step S32: Constructing a hierarchical perception feature aggregation network In order to explore the importance of semantic information at each level and enhance the focus on actions, the network first performs self-attention enhancement on features at each level, and applies it to the global, temporal and spatial dimensions to capture contextual information of different situations. in

[0028]

[0029] Among them, W1, W2 and W3 are weight matrices, is the normalization factor, and x' is the enhanced feature;

[0030] Step S33: Next, the hierarchical perception feature aggregation network Contains an innovative feature optimization module that uses global features to guide the optimization of temporal and spatial features. It uses a two-layer MLP as a projection network p(·) to project the enhanced hierarchical features x' to the potential embedding P = p(x'), and then temporal embedding and spatial embedding Subtract the global embedding In order to reduce the redundancy and influence of global common information, the fully connected layer is finally used to further optimize the local features and the three types of features are spliced ​​to obtain the multi-level semantic features of the input video pair.

[0031] In a preferred embodiment, step S4 specifically includes the following steps:

[0032] Step S41: Design an action difference comparison decoder based on Transformer, and use the powerful ability of Transformer to process sequences to compare the multi-level action feature differences between the query and sample videos. The decoder will and As "query" and "key-value pair" respectively, the learning process is as follows:

[0033]

[0034]

[0035] Among them, W q , W k and W v is the weight matrix, is the normalization factor. The MLP module contains two linear layers with GELU activation function, and finally generates a new relative representation

[0036] Step S42: Based on the task requirements, use a classifier or regressor to analyze the relative representation obtained in step S41. Perform multi-level perceptual motion evaluation and analysis, fully considering the multi-granularity differences between query and sample videos, such as global motion trends, temporal sub-motion changes, and local motion details;

[0037] Step S43: Based on the task requirements, use the corresponding loss function, such as cross entropy loss, mean square error loss, etc. to optimize the multi-level perceptual action evaluation and analysis network to obtain more accurate difference attributes between the query and sample videos, and combine the real label of the sample video to obtain the final evaluation result of the query video.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. Human motion comparative assessment methods often rely solely on global video information for analysis and comparison, often overlooking subtle differences in temporal sub-movements and local motion details. This present invention proposes a method for extracting and analyzing multi-level human motion features in human motion video scenes with highly consistent categorical information, and for comparative assessment of information at all levels of the human body. This method effectively extracts multi-level semantic information from motion videos, effectively utilizing a combination of coarse-grained and fine-grained information to achieve more comprehensive and refined comparative assessment of motion.

[0040] 2. In order to achieve more accurate local action comparison, the present invention designs a time alignment video encoder that uses dynamic temporal adaptation to align the actions in two videos, more accurately compare local subtle changes, and mine key information.

[0041] 3. The present invention designs a hierarchical perceptual feature aggregation network, further optimizes the features at each level, captures contextual information in different semantic dimensions, effectively focuses on actions that are more important for evaluation and analysis, and innovatively proposes a feature optimization module that uses global features to guide the optimization of temporal and spatial features, reducing the redundancy and impact of global common information and avoiding the incoherence of local features. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of a method implementation according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0045] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0046] like Figure 1 As shown, the present invention provides an action evaluation method based on temporal dynamic adaptation and hierarchical comparison, comprising the following steps:

[0047] Step S1: Obtain image sequences or videos of scenes related to action evaluation, annotate the action categories and quality scores contained therein to construct a dataset. Specifically, the following steps are included:

[0048] Step S11: Obtain coherent pictures or videos of application scenarios related to human motion assessment from the Internet to initially form a data set;

[0049] Step S12: Clean the data set, label the categories and quality scores of human actions based on the goals of the current application scenario on the existing data labels, and complete the construction of the data set;

[0050] Step S13: Divide the constructed data set into a training set and a test set according to an appropriate predetermined ratio.

[0051] Step S2: Build X3D, pre-trained on a large-scale action dataset, as the backbone network to extract spatiotemporal features of paired videos, and design a temporal alignment video encoder to dynamically align spatiotemporal features. This includes the following steps:

[0052] Step S21: For an input query video X, randomly select sample videos Z with the same frame length l from the training set to form input video pairs, and split them into N segments containing 16 frames;

[0053] Step S22: Pre-train the X3D backbone network based on the large-scale human motion dataset Kinetics-400 to obtain excellent motion information capture capabilities, and then use the network to extract segment-level spatiotemporal features of the input video pair Where T, H×W, and D represent the time step, spatial block size, and feature size, respectively;

[0054] Step S23: Design a temporal alignment video encoder E. The encoder first uses a projection block containing two fully connected layers with batch normalization and ReLU activation function to project F into an intermediate embedding of size T×H×W×D', where D' is the encoding feature size. Then, sine and cosine position encoding is added to each block of the intermediate embedding and a 3-layer Transformer encoder is used to model the spatiotemporal contextual relationship between all blocks. Finally, a temporal transformer consisting of a localization network L and a temporal transformer T is used to align the temporal actions between the query video and the sample video. The conversion process is expressed as follows:

[0055] f'=T Ψ (f),Ψ=(ω,b)=L(f)

[0056] Where f is the feature to be converted, f' represents the timing alignment result, and Ψ = (ω, b) represents the generated perturbation parameter;

[0057] Step S24: Introducing timing cycle consistency loss To optimize the timing alignment coding described in step S23, for each segment First calculate its soft nearest neighbor Then trace the location Nearest neighbor Feature pairs It should meet the temporal consistency, that is, The above loss learning process is expressed as follows:

[0058]

[0059]

[0060]

[0061] Among them, f' X and f' Z That is, the features of the query video and the sample video after temporal alignment, To calculate the similarity factor of the nearest neighbor, g(·) represents the calculation of cosine similarity, and represents the j-th and k-th segments of the sample video Z, represents the kth segment of the query video X, and N is the number of segments in step S21.

[0062] Step S3: Perform multi-dimensional pooling on the spatiotemporal features from step S2 to obtain multi-level features, and design a hierarchical perception feature aggregation network to enhance the features at each level. This specifically includes the following steps:

[0063] Step S31: Obtain block-shaped spatiotemporal features after dynamic alignment in step S2 In the space-time dimension and space dimension Perform global average pooling to obtain global features and time series characteristics

[0064] Step S32: Constructing a hierarchical perception feature aggregation network In order to explore the importance of semantic information at each level and enhance the focus on actions, the network first performs self-attention enhancement on features at each level, and applies it to the global, temporal and spatial dimensions to capture contextual information of different situations. in

[0065]

[0066] Among them, W1, W2 and W3 are weight matrices, is the normalization factor, and x' is the enhanced feature;

[0067] Step S33: Next, the hierarchical perception feature aggregation network Contains an innovative feature optimization module that uses global features to guide the optimization of temporal and spatial features. It uses a two-layer MLP as a projection network p(·) to project the enhanced hierarchical features x' to the potential embedding P = p(x'), and then temporal embedding and spatial embedding Subtract the global embedding In order to reduce the redundancy and influence of global common information, the fully connected layer is finally used to further optimize the local features and the three types of features are spliced ​​to obtain the multi-level semantic features of the input video pair.

[0068] Step S4: Design a Transformer-based motion difference comparison decoder to analyze the feature differences at each level of the paired videos and output the final analysis and evaluation results. This includes the following steps:

[0069] Step S41: Design an action difference comparison decoder based on Transformer, and use the powerful ability of Transformer to process sequences to compare the multi-level action feature differences between the query and sample videos. The decoder will and As "query" and "key-value pair" respectively, the learning process is as follows:

[0070]

[0071]

[0072] Among them, W q , W k and W v is the weight matrix, is the normalization factor. The MLP module contains two linear layers with GELU activation function, and finally generates a new relative representation

[0073] Step S42: Based on the task requirements, use a classifier or regressor to analyze the relative representation obtained in step S41. Perform multi-level perceptual motion evaluation and analysis, fully considering the multi-granularity differences between query and sample videos, such as global motion trends, temporal sub-motion changes, and local motion details;

[0074] Step S43: Based on the task requirements, use the corresponding loss function, such as cross entropy loss, mean square error loss, etc. to optimize the multi-level perceptual action evaluation and analysis network to obtain more accurate difference attributes between the query and sample videos, and combine the real label of the sample video to obtain the final evaluation result of the query video.

[0075] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions and effects do not exceed the scope of the technical solution of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. An action evaluation method based on temporal dynamic adaptation and hierarchical comparison, characterized in that: The following steps are involved: Step S1: Obtain image sequences or videos of scenes related to action evaluation, annotate the action categories and quality scores contained therein to construct a dataset; Step S2: Construct X3D, which is pre-trained on a large-scale action dataset, as the backbone network to extract the spatiotemporal features of paired videos, and design a temporal alignment video encoder to dynamically align the spatiotemporal features; Step S3: Perform multi-dimensional pooling on the spatiotemporal features of step S2 to obtain multi-level features, and design a hierarchical perception feature aggregation network to enhance the features of each level; Step S4: Design a Transformer-based motion difference comparison decoder to analyze the feature differences at each level of the paired videos and output the final analysis and evaluation results; The step S2 specifically includes the following steps: Step S21: For an input query video X, randomly select sample videos Z with the same frame length l from the training set to form an input video pair, and split the input video pair into N segments containing 16 frames; Step S22: Pre-train the X3D backbone network based on the large-scale human action dataset Kinetics-400, and then use the network to extract segment-level spatiotemporal features of the input video pair Where T, H×W, and D represent the time step, spatial block size, and feature size, respectively; Step S23: Design a temporal alignment video encoder E. The temporal alignment video encoder E first uses a projection block containing two fully connected layers with batch normalization and a ReLU activation function to project F into an intermediate embedding of size T×H×W×D′, where D′ is the encoding feature size. Then, sine and cosine position encoding is added to each block of the intermediate embedding and a 3-layer Transformer encoder is used to model the spatiotemporal contextual relationship between all blocks. Finally, a temporal transformer consisting of a localization network L and a temporal transformer T′ is used to align the temporal actions between the query video and the sample video. The conversion process is expressed as follows: f′=T′ Ψ (f), Ψ=(ω, b)=L(f) Where f is the feature to be converted, f′ represents the timing alignment result, and Ψ = (ω, b) represents the generated perturbation parameter; Step S24: Introducing timing cycle consistency loss To optimize step S23 timing alignment coding, for each segment First calculate its soft nearest neighbor Then trace the location Nearest neighbor Feature pairs Comply with temporal consistency, that is The above loss learning process is expressed as follows: Among them, f′ X and f′ Z That is, the features of the query video and the sample video after temporal alignment, To calculate the similarity factor of the nearest neighbor, g(·) represents the calculation of cosine similarity, and represents the j-th and k-th segments of the sample video Z, represents the kth segment of the query video X, where N is the number of segments in step S21; The step S3 specifically includes the following steps: Step S31: Obtain block-shaped spatiotemporal features after dynamic alignment in step S2 In the space-time dimension and space dimension Perform global average pooling to obtain global features and time series characteristics Step S32: Constructing a hierarchical perception feature aggregation network In order to explore the importance of semantic information at each level and enhance the focus on actions, the network first performs self-attention enhancement on features at each level, and applies it to the global, temporal and spatial dimensions to capture contextual information of different situations. in Among them, W1, W2 and W3 are weight matrices, is the normalization factor, and x′ is the enhanced feature; Step S33: Next, the hierarchical perception feature aggregation network Contains an innovative feature optimization module that uses global features to guide the optimization of temporal and spatial features. It uses a two-layer MLP as a projection network p(·) to project the enhanced hierarchical features x′ to the potential embedding P = p(x′), and then temporal embedding and spatial embedding Subtract the global embedding In order to reduce the redundancy and influence of global common information, the fully connected layer is finally used to further optimize the local features and the three types of features are spliced ​​to obtain the multi-level semantic features of the input video pair.

2. The action evaluation method based on temporal dynamic adaptation and hierarchical comparison according to claim 1, characterized in that: The step S1 specifically includes the following steps: Step S11: Obtain coherent pictures or videos of application scenarios related to human motion assessment from the Internet to initially form a data set; Step S12: Clean the data set, label the categories and quality scores of human actions based on the goals of the current application scenario on the existing data labels, and complete the construction of the data set; Step S13: Divide the constructed data set into a training set and a test set according to a predetermined ratio.

3. The action evaluation method based on temporal dynamic adaptation and hierarchical comparison according to claim 1, characterized in that: The step S4 specifically includes the following steps: Step S41: Design an action difference comparison decoder based on Transformer, and use the powerful ability of Transformer to process sequences to compare the multi-level action feature differences between the query and sample videos. The decoder will and As "query" and "key-value pair" respectively, the learning process is as follows: Among them, W q , W k and W v is the weight matrix, is the normalization factor. The MLP module contains two linear layers with GELU activation function, and finally generates a new relative representation Step S42: Based on the task requirements, use the classifier or regressor to analyze the relative representation obtained in step S41 Perform multi-level perceptual motion evaluation and analysis, fully considering the multi-granularity differences between query and sample videos, including global motion trends, temporal sub-motion changes, and local motion details; Step S43: Based on the task requirements, the corresponding loss function is used, and the cross entropy loss and mean square error loss are used to optimize the multi-level perceptual action evaluation and analysis network to obtain more accurate difference attributes between the query and sample videos, and the final evaluation result of the query video is obtained by combining the real label of the sample video.

Images