Action quality assessment method, device and equipment based on critical moment perception

Abstract

Embodiments of the present invention provide a method, apparatus, and device for assessing motion quality based on critical moment perception. The method includes extracting features from a target motion video and a sample video in a backbone network to obtain first and second segment-level features arranged in time sequence; inputting the features into a critical moment perception network to obtain a critical moment perception result; inputting the critical moment perception result into a weight adjustment module to obtain enhanced segment-level features; concatenating the enhanced segment-level features and inputting them into a comparative regression network to output a relative quality score; and adding the relative quality score to the quality score of the sample video to obtain a quality score for the action in the target motion video. In this way, the impact of common features between videos on motion quality assessment can be reduced, thereby effectively improving the accuracy of motion quality assessment.

Description

Technical Field

[0001] The present invention generally relates to the technical field of computer vision action recognition, and more particularly to a method, apparatus and device for evaluating action quality based on key moment perception. Background Art

[0002] As the demand for video data analysis continues to increase across various fields, the importance of motion recognition and video understanding technologies has become increasingly prominent. As core components of intelligent video analysis systems, they have been widely used in a variety of fields, including surveillance security, entertainment media, sports training, and behavior recognition.

[0003] Existing techniques for evaluating movement quality can be divided into two categories, depending on the objective: regression-based methods and ranking-based methods. When evaluating movement quality in sports, regression-based methods can quantitatively analyze and provide a specific score. In the analysis of movements for skill learning and fitness training, ranking-based methods can qualitatively assess the correctness of movements.

[0004] Although existing technologies can still work in simple motion quality assessment scenarios, they perform poorly in actions with little difference in motion quality. There is a technical problem that the differences between the overall features of the actions are not obvious, while similar features in the overall evaluation strategy lead to the influence of subtle differences. Summary of the Invention

[0005] According to an embodiment of the present invention, a motion quality assessment solution based on key moment perception is provided. This solution can solve the problem that current motion quality assessment methods using overall feature assessment strategies are insufficient in perceiving subtle motion differences and thus have poor assessment results.

[0006] In a first aspect of the present invention, a method for evaluating action quality based on key moment perception is provided. The method comprises:

[0007] In the backbone network, features are extracted from the target action video and the example video to obtain the first segment-level features and the second segment-level features arranged in time sequence.

[0008] The first segment-level features and the second segment-level features are input into a key moment perception network, and the key moments in the action are perceived by the key moment perception network based on the overall time scale of the video and the adjacent segment-level features, thereby obtaining a first key moment perception result and a second key moment perception result.

[0009] The first key moment perception result and the second key moment perception result are input into the weight adjustment module for compression and weight extraction, and the weight of the segment-level feature is adjusted to obtain the enhanced first segment-level feature and the enhanced second segment-level feature.

[0010] The enhanced first segment-level features and the enhanced second segment-level features are spliced ​​and input into the contrastive regression network to output a relative quality score; the relative quality score is added to the quality score of the example video to obtain the quality score of the action in the target action video.

[0011] In the backbone network, features of the target action video are extracted to obtain segment-level features arranged in time sequence, including: the target action video enters the backbone network in the form of N consecutive image frames, and the backbone network extracts a feature for every L image frames with an interval of E, for a total of S features, each feature representing the spatiotemporal information contained in the L image frames; where N, E, L, and S are positive integers.

[0012] The critical moment perception network includes a global information perception module and a local information perception module; the global information perception module and the local information perception module both adopt a residual connection method, the input of the global information perception module serves as the input of the critical moment perception network, and the output of the local information perception module serves as the output of the critical moment perception network.

[0013] The global information perception module is based on a multi-head self-attention network, which includes a fully connected unit, a scaled dot product attention module, a concat layer and a second fully connected layer in sequence.

[0014] The fully connected unit includes three groups of fully connected modules, each group of fully connected modules includes several first fully connected layers, and the number of fully connected layers in each group of fully connected modules is consistent with the number of heads of the multi-head self-attention; the input of the fully connected unit serves as the input of the global information perception module; the output of the fully connected unit serves as the input of the scaled dot product attention layer.

[0015] The scaled dot product attention module includes several scaled dot product attention layers, and the number of scaled dot product attention layers is consistent with the number of heads of the multi-head self-attention; the output of the scaled dot product attention module is used as the input of the concat layer.

[0016] The concat layer is used to splice the outputs of the scaled dot product attention module and output the splicing result to the second fully connected layer.

[0017] The second fully connected layer is used to take the output of the concat layer as input, and the output of the second fully connected layer is added to the input of the global information perception module to obtain the output result of the global information perception module.

[0018] The local information perception module includes a third fully connected layer, a 1D convolutional layer and a fourth fully connected layer in sequence.

[0019] The third fully connected layer takes the output of the global information perception module as input, and the output of the third fully connected layer serves as the input of the 1D convolutional layer.

[0020] The 1D convolution layer is used to search for key moments in a time domain with the same size as the convolution kernel, and the output of the 1D convolution layer serves as the input of the fourth fully connected layer.

[0021] The output of the fourth fully connected layer is added to the input of the local information perception module to obtain the output result of the local information perception module.

[0022] The weight adjustment module includes an average pooling layer, a fifth fully connected layer, a first nonlinear mapping layer, a sixth fully connected layer and a first Sigmoid layer.

[0023] The segment-level features are compressed by the average pooling layer to obtain the initial weight coefficient of each segment-level feature.

[0024] The weight of each segment-level feature is adjusted according to the loss function and the adjusted weight is assigned to each segment-level feature to obtain a weight-adjusted segment-level feature.

[0025] The loss function is:

[0026]

[0027] Among them, L weight is the loss function; L rank is the ranking loss; L norm is the normalization loss; λ is the ranking loss L rank The weight of γ is the normalized loss L norm The weight of W i and W j are the weight values ​​of the i-th and j-th moments respectively; K is the number of key moments, and T is the total number of moments.

[0028] The contrast regression network is a branch network composed of a classification network and a regression network; the outputs of the classification network and the regression network are multiplied together as the output result of the contrast regression network.

[0029] The classification network sequentially includes a seventh fully connected layer, a second nonlinear mapping layer, and an eighth fully connected layer; the input of the classification network is the result of splicing the output of the weight adjustment module.

[0030] The regression network includes a ninth fully connected layer, a third nonlinear mapping layer, a tenth fully connected layer, and a second Sigmoid layer in sequence; the input of the regression network is the result of splicing the outputs of the weight adjustment module.

[0031] In a second aspect of the present invention, a device for evaluating action quality based on key moment perception is provided. The device comprises:

[0032] The extraction module is used to extract features from the target action video and the example video in the backbone network to obtain first segment-level features and second segment-level features arranged in time sequence.

[0033] The perception module is used to input the first segment-level features and the second segment-level features into a key moment perception network, and the key moments in the action are perceived by the key moment perception network based on the overall time scale of the video and the adjacent segment-level features, thereby obtaining a first key moment perception result and a second key moment perception result.

[0034] An adjustment module is used to input the first key moment perception result and the second key moment perception result into a weight adjustment module for compression and weight extraction, and to adjust the weight of the segment-level features to obtain enhanced first segment-level features and enhanced second segment-level features.

[0035] The evaluation module is used to splice the enhanced first segment-level features and the enhanced second segment-level features and input them into the contrast regression network to output a relative quality score; the relative quality score is added to the quality score of the example video to obtain the quality score of the action in the target action video.

[0036] In a third aspect of the present invention, an electronic device is provided. The electronic device comprises at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of the first aspect of the present invention.

[0037] In a fourth aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the first aspect of the present invention.

[0038] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0040] Figure 1A flowchart of a method for evaluating motion quality based on key moment perception according to an embodiment of the present invention is shown.

[0041] Figure 2 A diagram showing a network structure of action quality assessment based on key moment perception according to an embodiment of the present invention is shown.

[0042] Figure 3 A diagram showing a critical moment awareness network structure according to an embodiment of the present invention is shown.

[0043] Figure 4 A structural diagram of a weight adjustment module according to an embodiment of the present invention is shown.

[0044] Figure 5 A comparative regression network structure diagram according to an embodiment of the present invention is shown.

[0045] Figure 6 FIG2 shows a block diagram of an apparatus for evaluating action quality based on key moment perception according to an embodiment of the present invention;

[0046] Figure 7 shows a block diagram of an exemplary electronic device capable of implementing embodiments of the present invention;

[0047] Among them, 700 is an electronic device, 701 is a computing unit, 702 is a ROM, 703 is a RAM, 704 is a bus, 705 is an I / O interface, 706 is an input unit, 707 is an output unit, 708 is a storage unit, and 709 is a communication unit. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0050] Figure 1 A flowchart of a method for evaluating action quality based on key moment perception according to an embodiment of the present invention is shown. The method includes:

[0051] S101. In the backbone network, feature extraction is performed on the target action video and the example video to obtain first segment-level features and second segment-level features arranged in time sequence.

[0052] like Figure 2 As shown, the first segment-level feature is obtained by extracting features from the target action video, and the second segment-level feature is obtained by extracting features from the example video.

[0053] In this embodiment, the example video is the feature that the current perception network focuses on recognizing, and the target action video is other action features that need to be recognized.

[0054] Furthermore, the I3D network is used as the backbone network for feature extraction, and the weights pre-trained on the Kinetics-400 dataset are used to assign new weights to the backbone network.

[0055] By extending the 2D convolutional network to 3D, the I3D network can simultaneously capture the spatial information and temporal dynamic information in the video frame, and can better understand the video content than a simple 2D convolutional network.

[0056] Kinetics-400 is a large-scale human action recognition dataset. The pre-trained model on this dataset has learned rich video dynamic features, which can provide a very good initialization effect for the backbone network, accelerate model training and improve the final performance.

[0057] In this embodiment, in the backbone network, features of the target action video are extracted to obtain segment-level features arranged in time sequence, including: the target action video enters the backbone network in the form of N continuous image frames, and the backbone network extracts a feature for every L image frames with an interval of E, and a total of S features are extracted, each feature represents the spatiotemporal information contained in the L image frames; where N, E, L, and S are positive integers.

[0058] In one embodiment, when you want to extract 20 segment-level features from each video, you can set N to 103, L to 8, and E to 5. By using the backbone network for feature extraction, you can extract 20 segment-level features arranged in time sequence.

[0059] The backbone network extracts a feature for every L image frames with an interval of E. For example, the first feature can be extracted from the eight images "1-8", and the second feature can be extracted from the eight images "6-13", until 20 segment-level features arranged in time sequence are extracted.

[0060] By extracting features over a time series, the temporal continuity of the video is preserved, helping to capture the dynamic changes and continuity of the action. Processing L consecutive image frames at a time ensures that each feature covers sufficient motion information, facilitating more accurate identification of the target action in the video. By extracting features every E frames, information from different time points can be captured while ensuring a certain degree of overlap between features, increasing feature diversity.

[0061] S102. Input the first segment-level features and the second segment-level features into a key moment perception network. The key moment perception network perceives the key moments in the action based on the overall time scale of the video and the adjacent segment-level features, respectively, to obtain a first key moment perception result and a second key moment perception result.

[0062] like Figure 3 As shown, the critical moment perception network includes a global information perception module and a local information perception module.

[0063] Among them, the global information perception module is used to perceive the key moments in the action on a larger time scale, and the local information perception module is used to perceive the key moments in the action on adjacent segment-level features.

[0064] Among them, the global information perception module and the local information perception module both adopt the residual connection method. The characteristic of the residual connection is that the input of a certain layer of network is added to the output of the layer of network as the final output result of the layer of network and sent to the next layer of network.

[0065] In this embodiment, a residual connection is used to connect the input of the global information perception module to the input of the key moment perception network, and the output of the local information perception module to the output of the key moment perception network. This means that the extracted segment-level features are fed into the global information perception module while the input segment-level features are added to the output of the global information perception module. The segment-level features output by the global information perception module are not only fed into the local information perception module but also added to its output.

[0066] The use of residual connections helps alleviate the vanishing gradient problem in deep network training, enabling the network to better learn parameters and improving model training efficiency and stability. By directly passing input features to subsequent layers, this ensures direct information flow, enhances the model's sensitivity to input features, and improves the expressiveness of features. This structural design enables the network to more accurately capture global and local information in the video, further improving the perception of key moments and optimizing the performance of video understanding tasks.

[0067] In this embodiment, in order to enable the global information perception module to perceive the key moments in the action on a larger time scale, a multi-head self-attention network is used for construction.

[0068] Among them, the global information perception module is based on a multi-head self-attention network, which includes a fully connected unit, a scaled dot product attention module, a concat layer and a second fully connected layer in sequence.

[0069] Furthermore, a multi-head self-attention mechanism is used to find key moments on a global scale. In this mechanism, the input sequence is projected into multiple subspaces (heads), and each head learns to focus on different parts of the input. This allows the global information perception module to focus on different aspects of the input sequence in parallel, thereby better capturing the relationship between the inputs.

[0070] In this embodiment, the number of heads can be set to a number of integers, that is, one main action is decomposed into several sub-actions.

[0071] By segmenting the input sequence and processing it simultaneously in multiple subspaces, the multi-head self-attention network learns a more comprehensive and richer feature representation. This approach allows the network to concurrently focus on different parts of the input sequence in different subspaces, effectively capturing complex patterns and inherent connections in the data. This parallel processing mechanism not only improves computational efficiency but also enables the network to understand data from multiple perspectives and dimensions, enhancing the model's deep understanding and ability to express sequential data.

[0072] like Figure 3 As shown, the fully connected unit includes three groups of fully connected modules, each group of fully connected modules includes several first fully connected layers (Linear), and the number of first fully connected layers in each group of fully connected modules is consistent with the number of heads of multi-head self-attention; the input of the fully connected unit serves as the input of the global information perception module; the output of the fully connected unit serves as the input of the scaled dot product attention layer.

[0073] The scaled dot product attention module includes several scaled dot product attention layers (Scaled Dot-ProductAttention), and the number of scaled dot product attention layers is consistent with the number of heads of the multi-head self-attention; the output of the scaled dot product attention module is used as the input of the concat layer.

[0074] The concat layer is used to splice the output of the scaled dot product attention module and output the spliced ​​result to Figure 3 The second fully connected layer (Linear) is shown.

[0075] In one embodiment, a network of three groups of fully connected modules is used to form a fully connected unit, each group of fully connected modules includes 4 first fully connected layers, and 4 groups of scaled dot product attention layers are used to form a scaled dot product attention module, which is finally spliced ​​with 1 concat layer to form a global perception module. After the fragment-level features are input into the global perception module, they pass through the fully connected unit and the scaled dot product attention module in sequence, and the output is spliced ​​through the concat layer and sent to the second fully connected layer.

[0076] The second fully connected layer is used to take the output of the concat layer as input, and the output of the second fully connected layer is added to the input of the global information perception module to obtain the output result of the global information perception module.

[0077] The fully connected unit can perform preliminary transformation and abstraction on the input features, providing a more suitable input representation for the subsequent self-attention module. The multi-head self-attention module processes multiple heads in parallel, capturing different aspects of the input sequence from different subspaces, which helps to fully understand the video content. The scaled dot product attention module can effectively calculate the correlation between different positions in the sequence, allowing the model to focus on more important information and improve the efficiency of information processing. The global perception module constructed in this way not only has strong processing capabilities and deep understanding of video features, but also enhances the flexibility and stability of the model through modular design.

[0078] In this embodiment, in order to enable the local information perception module to perceive the key moments of the action based on adjacent segment-level features, a convolutional neural network is mainly used to construct the local information perception module.

[0079] Among them, such as Figure 3 As shown, the local information perception module includes a third fully connected layer (Linear), a 1D convolutional layer (Conv1D) and a fourth fully connected layer (Linear) in sequence.

[0080] The third fully connected layer takes the output of the global information perception module as input, and the output of the third fully connected layer serves as the input of the 1D convolutional layer.

[0081] The 1D convolution layer is used to search for key moments in a time domain with the same size as the convolution kernel, and the output of the 1D convolution layer serves as the input of the fourth fully connected layer.

[0082] In one embodiment, the size of the convolution kernel may be set to 3, corresponding to the current moment and adjacent previous and next moments respectively.

[0083] The output of the fourth fully connected layer is added to the input of the local information perception module to obtain the output result of the local information perception module.

[0084] The convolution kernel design of the 1D convolutional layer enables the model to effectively identify features of the current moment and its adjacent moments, focusing on the local relationships between adjacent segment-level features, thereby more precisely capturing key moments in local actions. By adjusting the size and number of convolution kernels in the 1D convolutional layer, the module's perception range and sensitivity to local information can be flexibly adjusted. The output of the fourth fully connected layer is added to the input of the local information perception module via a residual connection, allowing the local information perception module to combine the output of the global information perception module and better utilize contextual information to identify key moments.

[0085] S103: Input the first key moment perception result and the second key moment perception result into a weight adjustment module for compression and weight extraction, and perform weight adjustment on the segment-level features to obtain enhanced first segment-level features and enhanced second segment-level features.

[0086] In this embodiment, the weight adjustment module mainly uses the channel attention mechanism to compress and extract weights of the segment-level features output by S102 through the adjustment module, and re-assigns the weights of the segment-level features that were pre-trained on the original Kinetics-400 dataset.

[0087] Further, such as Figure 4 As shown in FIG, the weight adjustment module includes an average pooling layer (AvgPool1d), a fifth fully connected layer (Linear), a first nonlinear mapping layer (ReLU), a sixth fully connected layer (Linear), and a first Sigmoid layer.

[0088] Among them, the average pooling layer is used to compress the segment-level features, the first nonlinear mapping layer is used to increase the nonlinear ability of the entire network and improve the expression ability, and the first Sigmoid layer is used to process specific types of tasks and perform nonlinear transformations.

[0089] Furthermore, the segment-level features are compressed by the average pooling layer to obtain the initial weight coefficient of each segment-level feature.

[0090] In one embodiment, each feature size of the fragment-level feature is 1024. After the fragment-level feature compression operation is performed by the average pooling layer, each feature size becomes 1, that is, 1 number, and the initial weight coefficient of each fragment-level feature is obtained.

[0091] The weight of each segment-level feature is adjusted according to the loss function and the adjusted weight is assigned to each segment-level feature to obtain a weight-adjusted segment-level feature.

[0092] The loss function (Ranking Loss) is:

[0093]

[0094] Among them, L weight is the loss function; L rank is the ranking loss; L norm is the normalization loss; λ is the ranking loss L rank The weight of γ is the normalized loss L norm The weight of W i and W j are the weight values ​​of the i-th and j-th moments respectively; K is the number of key moments, and T is the total number of moments.

[0095] Among them, the loss function is used to adjust the weights of different segments, and the normalized loss function is used to avoid excessive weight growth at critical moments.

[0096] Furthermore, through training, the loss function L weight After obtaining the minimum value, the weight value at the i-th moment can be adjusted.

[0097] The average pooling layer compresses segment-level features to effectively extract representative weight coefficients for each segment, helping to reduce feature dimensionality while retaining important information. By adjusting these weights, the model enhances the representation of key moments, thereby improving the accuracy of identifying key moments in action videos.

[0098] S104. Concatenate the enhanced first segment-level features and the enhanced second segment-level features and input them into a comparative regression network to output a relative quality score; add the relative quality score to the quality score of the example video to obtain the quality score of the action in the target action video.

[0099] The quality score of the example video is a known and determined value.

[0100] In this embodiment, the highest and lowest possible quality score differences between the target action video and the example video are first subtracted, and then divided into several intervals. The enhanced first segment-level features and the enhanced second segment-level features are then spliced ​​and input into the comparative regression network.

[0101] The contrast regression network is a network composed of a classification network and a regression network; the outputs of the classification network and the regression network are multiplied together to serve as the output result of the contrast regression network.

[0102] like Figure 5 As shown, the classification network includes the seventh fully connected layer (Linear), the second nonlinear mapping layer (ReLU) and the eighth fully connected layer (Linear) in sequence; the input of the classification network is the result of splicing the output of the weight adjustment module.

[0103] Among them, the output result of the classification network represents the corresponding interval of the difference in quality scores between the final action target video and the example video.

[0104] The output of the classification network is designed to locate the range of quality score differences. Because the quality score difference between two videos can be significant, directly performing a regression prediction on the entire video can result in high prediction errors. By dividing this large score difference into smaller ranges, the accuracy of the subsequent regression analysis can be significantly improved, resulting in a more detailed and accurate quality score assessment.

[0105] like Figure 5 As shown, the regression network includes a ninth fully connected layer (Linear), a third nonlinear mapping layer (ReLU), a tenth fully connected layer (Linear), and a second Sigmoid layer in sequence; the input of the regression network is the result of concatenating the output of the weight adjustment module.

[0106] Among them, the output result of the regression network represents the specific value within this range.

[0107] The regression network significantly improves the accuracy and reliability of video quality assessment through its sophisticated feature processing and stable numerical prediction capabilities.

[0108] Furthermore, the output of the contrastive regression network is the difference between the scores of the target action video and the example video.

[0109] In this embodiment, the quality score of the action in the target action video is obtained by adding the result output by the comparative regression network and the quality score of the example video.

[0110] In this embodiment, the final output result is the score of the action in the video. The output score varies depending on the type of training and action to be evaluated.

[0111] In one embodiment, a diving video is used to train a network, and the trained model is used to evaluate the diving motion. The output result is the diving motion score.

[0112] This application uses a key moment perception network to find key moments in actions. By enhancing the weight of these key moments in the overall features, the key difference information between different videos is made more obvious, and the impact of common features between videos on action quality assessment is reduced, thereby effectively improving the accuracy of action quality assessment.

[0113] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0114] The above is an introduction to the method embodiment. The following is a further explanation of the solution of the present invention through an apparatus embodiment having the same inventive concept as the method in the aforementioned embodiment.

[0115] like Figure 6 As shown, the apparatus 600 includes:

[0116] An extraction module 610 is configured to perform feature extraction on the target action video and the example video in the backbone network to obtain first segment-level features and second segment-level features arranged in time sequence;

[0117] A perception module 620 is configured to input the first segment-level features and the second segment-level features into a key moment perception network, and use the key moment perception network to perceive key moments in the action based on the overall time scale of the video and adjacent segment-level features, thereby obtaining a first key moment perception result and a second key moment perception result.

[0118] An adjustment module 630 is configured to input the first key moment perception result and the second key moment perception result into a weight adjustment module for compression and weight extraction, and perform weight adjustment on the segment-level features to obtain enhanced first segment-level features and enhanced second segment-level features;

[0119] The evaluation module 640 is used to splice the enhanced first segment-level features and the enhanced second segment-level features and input them into the contrast regression network to output a relative quality score; the relative quality score is added to the quality score of the example video to obtain the quality score of the action in the target action video.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0121] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0122] According to an embodiment of the present invention, the present invention further provides an electronic device and a readable storage medium.

[0123] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0124] The electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0125] Multiple components in the electronic device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0126] The computing unit 701 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as methods S101 to S104. For example, in some embodiments, methods S101 to S104 can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the methods S101 to S104 described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute methods S101 to S104 in any other appropriate manner (eg, by means of firmware).

[0127] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0129] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0131] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0132] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0133] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0134] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for evaluating motion quality based on critical moment perception, characterized in that: include: In the backbone network, feature extraction is performed on the target action video and the example video to obtain the first segment-level features and the second segment-level features arranged in time sequence; Inputting the first segment-level features and the second segment-level features into a key moment perception network, wherein the key moment perception network perceives the key moments in the action based on the overall time scale of the video and the adjacent segment-level features, thereby obtaining a first key moment perception result and a second key moment perception result; Inputting the first key moment perception result and the second key moment perception result into a weight adjustment module for compression and weight extraction, and performing weight adjustment on the segment-level features to obtain enhanced first segment-level features and enhanced second segment-level features; The enhanced first segment-level features and the enhanced second segment-level features are concatenated and input into a contrastive regression network to output a relative quality score; the relative quality score is added to the quality score of the example video to obtain the quality score of the action in the target action video; The critical moment perception network includes a global information perception module and a local information perception module; the global information perception module and the local information perception module both adopt a residual connection method; the global information perception module is based on a multi-head self-attention network, and includes a fully connected unit, a scaled dot product attention module, a concat layer and a second fully connected layer in sequence; the local information perception module includes a third fully connected layer, a 1D convolution layer and a fourth fully connected layer in sequence; the weight adjustment module includes an average pooling layer, a fifth fully connected layer, a first nonlinear mapping layer, a sixth fully connected layer and a first Sigmoid layer; the fragment-level features are compressed by the average pooling layer to obtain the initial weight coefficient of each fragment-level feature; the weight of each fragment-level feature is adjusted according to the loss function and the adjusted weight is assigned to each fragment-level feature to obtain the fragment-level feature after weight adjustment.

2. The method according to claim 1, characterized in that In the backbone network, feature extraction is performed on the target action video to obtain segment-level features arranged in time sequence, including: The target action video enters the backbone network in the form of N consecutive image frames. The backbone network extracts a feature for every L image frames with an interval of E, and extracts S features in total. Each feature represents the spatiotemporal information contained in the L image frames; where N, E, L, and S are positive integers.

3. The method according to claim 1, characterized in that The input of the global information perception module serves as the input of the critical moment perception network, and the output of the local information perception module serves as the output of the critical moment perception network.

4. The method according to claim 3, characterized in that The fully connected unit includes three groups of fully connected modules, each group of fully connected modules includes several first fully connected layers, and the number of first fully connected layers in each group of fully connected modules is consistent with the number of heads of the multi-head self-attention; the input of the fully connected unit serves as the input of the global information perception module; the output of the fully connected unit serves as the input of the scaled dot product attention layer; The scaled dot product attention module includes several scaled dot product attention layers, and the number of scaled dot product attention layers is consistent with the number of heads of the multi-head self-attention; the output of the scaled dot product attention module is used as the input of the concat layer; The concat layer is used to splice the outputs of the scaled dot product attention module and output the splicing result to the second fully connected layer; The second fully connected layer is used to take the output of the concat layer as input, and the output of the second fully connected layer is added to the input of the global information perception module to obtain the output result of the global information perception module.

5. The method according to claim 3, characterized in that The third fully connected layer takes the output of the global information perception module as input, and the output of the third fully connected layer serves as the input of the 1D convolutional layer; The 1D convolution layer is used to search for key moments in a time domain with the same size as the convolution kernel, and the output of the 1D convolution layer serves as the input of the fourth fully connected layer; The output of the fourth fully connected layer is added to the input of the local information perception module to obtain the output result of the local information perception module.

6. The method according to claim 1, characterized in that The loss function is: ； ； ； in, is the loss function; is the ranking loss; is the normalized loss; is the ranking loss The weight of is the normalized loss The weight of and are the weight values ​​at the i-th and j-th moments respectively; is the number of critical moments, is the total number of moments.

7. The method according to claim 1, characterized in that The comparative regression network is a network composed of a classification network and a regression network; the outputs of the classification network and the regression network are multiplied together as the output result of the comparative regression network; The classification network includes a seventh fully connected layer, a second nonlinear mapping layer, and an eighth fully connected layer in sequence; the input of the classification network is the result of splicing the output of the weight adjustment module; The regression network includes a ninth fully connected layer, a third nonlinear mapping layer, a tenth fully connected layer, and a second Sigmoid layer in sequence; the input of the regression network is the result of splicing the outputs of the weight adjustment module.

8. An action quality assessment device based on critical moment perception, characterized in that: include: An extraction module is used to extract features from the target action video and the example video in the backbone network to obtain first segment-level features and second segment-level features arranged in time sequence; a perception module, configured to input the first segment-level features and the second segment-level features into a key moment perception network, and use the key moment perception network to perceive the key moments in the action based on the overall time scale of the video and the adjacent segment-level features, thereby obtaining a first key moment perception result and a second key moment perception result; An adjustment module is configured to input the first key moment perception result and the second key moment perception result into a weight adjustment module for compression and weight extraction, and perform weight adjustment on the segment-level features to obtain enhanced first segment-level features and enhanced second segment-level features; An evaluation module is configured to concatenate the enhanced first segment-level features and the enhanced second segment-level features and input the concatenated features into a contrastive regression network to output a relative quality score; and to add the relative quality score to the quality score of the example video to obtain a quality score of the action in the target action video. The critical moment perception network includes a global information perception module and a local information perception module; the global information perception module and the local information perception module both adopt a residual connection method; the global information perception module is based on a multi-head self-attention network, and includes a fully connected unit, a scaled dot product attention module, a concat layer and a second fully connected layer in sequence; the local information perception module includes a third fully connected layer, a 1D convolution layer and a fourth fully connected layer in sequence; the weight adjustment module includes an average pooling layer, a fifth fully connected layer, a first nonlinear mapping layer, a sixth fully connected layer and a first Sigmoid layer; the fragment-level features are compressed by the average pooling layer to obtain the initial weight coefficient of each fragment-level feature; the weight of each fragment-level feature is adjusted according to the loss function and the adjusted weight is assigned to each fragment-level feature to obtain the fragment-level feature after weight adjustment.

9. An electronic device comprising at least one processor; and a memory communicatively connected to the at least one processor; characterized in that: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images