An action quality evaluation dataset assisted labeling system based on interframe difference mode waveform diagram

Abstract

The application discloses a motion quality evaluation dataset auxiliary labeling system based on an interframe difference mode waveform diagram, and comprises a motion quality evaluation dataset auxiliary labeling system framework, a motion segment start and end time labeling module and a motion segment quality labeling module, wherein the motion segment start and end time labeling module is used for loading a video to be labeled and playing, and after motion start and end time is found with the aid of an interframe difference mode waveform diagram, the start and end time is labeled to obtain a segmented motion segment; the motion segment quality labeling module is used for loading the segmented motion segment and playing, and after comparison with a standard motion, quality labeling is performed. The application can conveniently complete segmentation and labeling of a motion quality evaluation dataset with the aid of a simple user interface and an interframe difference mode waveform diagram, and improves labeling efficiency.

Description

Technical Field

[0001] This invention relates to a dataset-assisted annotation system, and more particularly to a dataset-assisted annotation system for motion quality assessment based on inter-frame differential waveform diagrams. Background Technology

[0002] Deep learning is a machine learning method that simulates the structure and function of the human brain's neural networks to achieve automatic learning and abstract representation of data. In deep learning, datasets play a crucial role, serving as the foundation for training and evaluating deep learning models. Dataset labeling refers to assigning labels or annotations to samples in a dataset to indicate the category, attribute, or related information of the samples. Dataset labeling is essential for many deep learning tasks, such as image classification, object detection, semantic segmentation, and natural language processing.

[0003] The purpose of dataset annotation is to provide accurate and detailed information about the samples so that deep learning models can learn and generalize from them. In many cases, deep learning models require a large amount of labeled data to achieve high performance. Annotated data can be provided by human annotation teams, professional annotation service providers, or crowdsourcing platforms.

[0004] Deep learning dataset annotation can be applied to various fields and tasks. In computer vision, annotated image datasets can be used for tasks such as image classification, object detection, and image segmentation. For example, in image classification tasks, each image needs to be labeled as belonging to different categories, such as cat, dog, and car. In object detection and image segmentation tasks, a bounding box or segmentation mask needs to be assigned to each object or pixel in the image.

[0005] For motion quality assessment tasks in the field of computer vision, it is necessary to annotate the start and end times of all motion segments in a motion video, and then annotate the quality of the segmented motion segments. Motions such as fitness exercises, Tai Chi, and Baduanjin involve several basic motion combinations repeated several times. Annotating such videos requires frequently dragging the progress bar to find the start and end times of the motion segments, resulting in low annotation efficiency. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide an auxiliary annotation system for motion quality assessment dataset based on inter-frame differential waveform diagram, which addresses the shortcomings of the existing technology.

[0007] To address the aforementioned technical problems, this invention discloses an auxiliary annotation system for motion quality assessment datasets based on inter-frame differential waveform diagrams. The system includes: a framework for an auxiliary annotation system for motion quality assessment datasets, a motion segment start and end time annotation module, and a motion segment quality annotation module, wherein:

[0008] The motion quality assessment dataset auxiliary annotation system framework includes: an annotation system front-end module and a back-end processing and calculation module. The annotation system front-end module is used to perform operations such as loading the video to be annotated and the annotation, calculating the inter-frame differential waveform, and saving and modifying the annotation. The back-end processing and calculation module is used to perform the above operations on the server side.

[0009] The action segment start and end time annotation module is used to load the video to be annotated and play it. After finding the start and end times of the action with the help of the inter-frame differential waveform, the start and end times are annotated to obtain the segmented action segments.

[0010] The motion segment quality annotation module is used to load the segmented motion segments, play them, and then perform quality annotations after comparing them with standard motion segments.

[0011] Furthermore, the action segment start and end time annotation module specifically includes: a video loading module, a video playback module, an action start and end time auxiliary search module, and an action segment start and end time annotation, saving, and loading module, wherein:

[0012] The video loading module is used to call the video loading function in the backend processing and calculation module to obtain a list of videos to be labeled and display it on the frontend interface.

[0013] The video playback module is used to play videos from the list of videos to be labeled in the front-end interface.

[0014] The action start and end time auxiliary search module is used to call the inter-frame differential waveform calculation function in the back-end processing and calculation module to calculate the inter-frame differential waveform of the current video to be labeled and display it in the front-end interface to assist in the labeling of the start and end times of action segments.

[0015] The action segment start and end time annotation, saving and loading module is used to call the annotation saving and modification function in the backend processing and calculation module to annotate the current video to be annotated and save the annotated action segment start and end time information to the server, or call the annotation loading function to display the saved annotation information on the front end interface.

[0016] Furthermore, the calculation function in the back-end processing and calculation module, namely, calculating the inter-frame differential waveform, specifically includes:

[0017] Step 1: Read the video F to be labeled, which has N frames.

[0018] Step 2: Convert the read video frames to be labeled into grayscale images. The specific method is as follows:

[0019]

[0020] in, and Let A, B, and C be the blue, green, and red channel values ​​of the pixel at position (x, y) in the i-th frame of video F, respectively, and let A, B, and C be the sensitivity weights. These values ​​are converted into a single grayscale channel value by weighted summation.

[0021] Step 3: Calculate the inter-frame difference of the video after conversion to grayscale. The specific method is as follows:

[0022]

[0023] in, and The grayscale images of frame i+1 and frame i are represented respectively. The difference between them yields the inter-frame difference D of frame i. i ;

[0024] Step 4: Calculate the magnitude of the video frame difference, using the following method;

[0025] Norm i =||D i |||2

[0026] Where ||||2 represents finding the 2-norm, Norm i The modulus representing the difference between video frames;

[0027] Step 5, calculate the modulus of the inter-frame difference in the video. i The specific method for performing moving average filtering is as follows:

[0028]

[0029] Where M is the size of the moving average window, k is the index within the window, and the filtered signal is y. i ;

[0030] Step 6, based on the filtered signal y from Step 5 i Draw a waveform diagram of the magnitude of the video frame difference as a function of the video frame.

[0031] Step 7, find the segmentation time S1,…,S of the action segment. k ;

[0032] Step 8, divide the time intervals S1, ..., S k Plot the waveform of the differential mode change between the video frames and return it.

[0033] Furthermore, the segmentation time S1,…,S described in step 7 for finding action segments… kSpecific methods include:

[0034] Step 7-1: Calculate the filtered signal y i The x-axis values ​​for peaks and valleys;

[0035] Step 7-2, the filtered signal y i The start and end x-coordinates, peak x-coordinates, and trough x-coordinates are combined together, as follows:

[0036] peaks_valleys={0}∪peaks∪valleys∪{N-1}

[0037] Where ∪ represents finding the union of sets, and peaks_valleys is the set of horizontal coordinates after the union;

[0038] Step 7-3, based on the filtered signal y i The change magnitude at the x-axis of peaks_valleys is used to remove the x-axis of peaks_valleys with smaller change magnitudes to obtain the candidate segmentation time set candidate;

[0039] Step 7-4: Use the DBSCAN clustering algorithm to select candidate segmentation times. i and the filtered signal at the corresponding coordinates The set of point pairs formed Clustering to obtain each candidate i Corresponding category label i ;

[0040] Step 7-5: Calculate the maximum and minimum values ​​of all candidate split times for each category label. and The specific method is as follows:

[0041]

[0042]

[0043] in and These represent the maximum and minimum split times for candidate segments with category label j, respectively.

[0044] Step 7-6: Merge the category labels with overlapping intervals formed by the minimum and maximum values ​​of all candidate segmentation times. The specific method is as follows:

[0045] Enumerate the label pairs (i,j), if Then change all the labels j in the label to i, and finally there will be K category labels remaining;

[0046] Step 7-7: Calculate the maximum and minimum values ​​of all candidate segmentation times for each category label to obtain the action segmentation time S. DBSCAN ;

[0047] Steps 7-8: Heuristically calculate the action segmentation time S using a genetic algorithm. genetic ;

[0048] Steps 7-9: Combine the action segmentation times calculated by the DBSCAN algorithm and the genetic algorithm to obtain the final segmentation set {S1,…,S}. k The specific method is as follows:

[0049] S = S DBSCAN ∪S genetic ={S1, ...,S k}

[0050] Where S represents the final partition set {S1,…,S} k}

[0051] Furthermore, the specific method for obtaining the candidate segmentation time set (candidate) as described in step 7-3 is as follows:

[0052]

[0053] Among them, by using peaks_valleys as the x-axis i+1 and peaks_valleys i Signal at the location and Take the absolute value of the difference and retain those changes that are greater than a threshold of D times. and The x-coordinate of the maximum value in peaks_valleys i .

[0054] Furthermore, the step 7-7 describes obtaining the action segment segmentation time S. DBSCAN The specific method is as follows:

[0055]

[0056] Among them, S DBSCAN This represents the union of the maximum and minimum values ​​of the candidate split times for each category label.

[0057] Furthermore, the step 7-8 describes using a genetic algorithm to heuristically solve for the action segmentation time S. genetic The specific method is as follows:

[0058] Let S′1,…,S′k The parameters to be optimized are solved using a genetic algorithm:

[0059]

[0060] in, This represents the segmentation time S′1,…,S′ that minimizes the objective function. k Thus, S is obtained. genetic ={S′1,…,S′ k}

[0061] Furthermore, the motion segment quality annotation module includes: a video segment loading module for motion quality to be annotated, a video segment playback module for motion quality to be annotated and standard motion video, and a motion segment quality annotation, saving, and loading module, wherein:

[0062] The video clip loading module for the action quality to be labeled is used to call the video loading function in the backend processing and calculation module to obtain a list of action clips to be labeled and display it on the front-end interface.

[0063] The video clip to be labeled and the standard video playback module are used to play the video clip to be labeled and the standard video corresponding to the video clip in the front-end interface.

[0064] The motion segment quality annotation, saving, and loading module is used to call the annotation saving and modification function in the backend processing and calculation module to annotate the current motion segment and save the annotated motion segment quality information to the server, or call the annotation loading function to display the saved annotation information on the front-end interface.

[0065] Furthermore, the sensitivity weights mentioned in step 2, which assign weight values ​​based on the human eye's sensitivity to different colors, have the optimal weight values ​​set as follows:

[0066] A = 0.114

[0067] B = 0.587

[0068] C = 0.299

[0069] Furthermore, the threshold D mentioned in step 7-3 is set to 0.1.

[0070] Beneficial effects:

[0071] 1. By using the present invention, the approximate start and end times of an action segment can be found with the assistance of the inter-frame differential waveform diagram. Therefore, compared with the existing annotation system, the system described in the present invention can greatly reduce the number of times and time of manually dragging the progress bar, and improve the efficiency of annotation.

[0072] 2. When annotating the quality of a motion segment, the current motion segment to be annotated can be compared with the corresponding standard motion, which helps to improve the accuracy of motion quality annotation. Attached Figure Description

[0073] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0074] Figure 1 This is a schematic diagram of the front-end interface of the action segment start and end time annotation module of an action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagram, which is part of the embodiment of the present invention.

[0075] Figure 2 This is a schematic diagram of the front-end interface of the action segment quality annotation module of an action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagram, which is part of the embodiments of the present invention.

[0076] Figure 3 This is a flowchart of user operations when annotating the start and end times of action segments in an action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams, as provided in part of an embodiment of the present invention.

[0077] Figure 4 This is a flowchart illustrating the user's operation when performing motion segment quality annotation in an action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams, as provided in part of an embodiment of the present invention.

[0078] Figure 5 This is a schematic diagram of the overall system architecture of the present invention. Detailed Implementation

[0079] This invention provides an auxiliary annotation system for motion quality assessment datasets based on inter-frame differential waveform diagrams, which solves the problem of low annotation efficiency when annotating motion quality assessment datasets. With the help of a simple user interface and inter-frame differential waveform diagrams, the system can easily complete the segmentation and annotation of motion quality assessment datasets, thereby improving annotation efficiency.

[0080] This invention discloses an auxiliary annotation system for motion quality assessment datasets based on inter-frame differential waveform diagrams. The system includes: a framework for auxiliary annotation of motion quality assessment datasets, a motion segment start and end time annotation module, and a motion segment quality annotation module.

[0081] The action quality assessment dataset auxiliary annotation system framework includes an annotation system front-end module and a back-end processing and calculation module. When the system front-end module needs to perform operations such as loading, calculating, and saving modifications, it calls the back-end processing and calculation module to complete the above loading, calculation, and saving modification operations on the server side.

[0082] The action segment start and end time annotation module includes a video loading module, a video playback module, an action start and end time auxiliary search module, and an action segment start and end time annotation, saving, and loading module. It can load the video to be annotated and then play it, and use the inter-frame differential waveform diagram to help find the action start and end time before annotating the start and end time.

[0083] The motion segment quality annotation module includes a video loading module, a video and standard motion video playback module, and a motion segment quality annotation, saving, and loading module. It can load and play the segmented motion segments and perform quality annotation after comparing them with standard motion.

[0084] Furthermore, in one implementation, when the action segment start and end time annotation module and the action segment quality annotation module perform action segment start and end time annotation and action segment quality annotation, the action quality evaluation dataset auxiliary annotation system framework provides a unified implementation method for the functions of the system front-end module and the back-end processing and calculation module, including:

[0085] Define the form of the backend interface called by the system frontend module, and call the backend processing and computing modules according to the form of the backend interface;

[0086] Define a unified implementation method for the functions of the backend processing and calculation modules, and implement the functions of the action quality assessment dataset auxiliary annotation system framework according to the backend interface form. The functions include loading, calculation and saving / modification functions.

[0087] Furthermore, in one implementation, the action segment start and end time annotation module includes the video to be annotated loading module, the video to be annotated playback module, the action start and end time auxiliary search module, and the action segment start and end time annotation, saving, and loading module;

[0088] The video loading module is used to call the loading function in the backend processing and calculation module to obtain a list of videos to be labeled and display it on the frontend interface.

[0089] The video playback module is used to play the video to be labeled in the front-end interface;

[0090] The action start and end time auxiliary search module is used to call the calculation function in the back-end processing and calculation module to calculate the inter-frame differential waveform of the current video to be labeled and display it in the front-end interface to assist in the labeling of the start and end times of the action segments.

[0091] The action segment start and end time annotation, saving and loading module is used to call the save and modify function in the backend processing and calculation module to save the annotated action segment start and end time information to the server. It can also call the loading function to display the saved annotation information to the front end.

[0092] Furthermore, in one implementation, the calculation function in the back-end processing and calculation module is an algorithm for calculating inter-frame differential waveforms, including:

[0093] Step 1: Read the video F to be labeled, which has N frames.

[0094] Step 2: Convert the read video frames to be labeled into grayscale images. The specific method is as follows:

[0095]

[0096] in, and Let A, B, and C be the blue, green, and red channel values ​​of the pixel at position (x, y) in the i-th frame of video F, respectively, and let A, B, and C be the sensitivity weights. These values ​​are converted into a single grayscale channel value by weighted summation.

[0097] Step 3: Calculate the inter-frame difference of the video after conversion to grayscale. The specific method is as follows:

[0098]

[0099] in, and The grayscale images of frame i+1 and frame i are represented respectively. The difference between them yields the inter-frame difference D of frame i. i ;

[0100] Step 4: Calculate the magnitude of the video frame difference, using the following method;

[0101] Norm i =||D i ||2

[0102] Where ||||2 represents finding the 2-norm, Norm i The modulus representing the difference between video frames;

[0103] Step 5, calculate the modulus of the inter-frame difference in the video. iThe specific method for performing moving average filtering is as follows:

[0104]

[0105] Where M is the size of the moving average window, k is the index within the window, and the filtered signal is y. i ;

[0106] Step 6, based on the filtered signal y from Step 5 i Draw a waveform diagram of the magnitude of the video frame difference as a function of the video frame.

[0107] Step 7, find the segmentation time S1,…,S of the action segment. k The specific method is as follows:

[0108]

[0109] Where argmin represents the segmentation time S1,…,S that minimizes the objective function. k ;

[0110] Step 8, divide the time intervals S1, ..., S k Plot the waveform of the differential mode change between the video frames and return it.

[0111] Furthermore, in step 7, the segmentation time S1,…,S for finding action segments is… k Specific methods include:

[0112] Step 1, calculate the filtered signal y i The x-axis values ​​for peaks and valleys;

[0113] Step 2, the filtered signal y i The start and end x-coordinates, peak x-coordinates, and trough x-coordinates are combined together, as follows:

[0114] peaks_valleys={0}∪peaks∪valleys∪{N-1}

[0115] Where ∪ represents finding the union of sets, and peaks_valleys is the set of horizontal coordinates after the union;

[0116] Step 3, based on the filtered signal y i To determine the magnitude of change at the x-axis of peaks_valleys, remove the x-axis values ​​of peaks_valleys with smaller changes. The specific method is as follows:

[0117]

[0118] Among them, by using peaks_valleys as the x-axis i+1 and peaks_valleys i Signal at the location and Take the absolute value of the difference and retain those changes that are greater than a threshold of D times. and The x-coordinate of the maximum value in peaks_valleys i This yields the candidate segmentation time set, candidate;

[0119] Step 4: Use the DBSCAN clustering algorithm to segment the candidate time. i and the filtered signal at the corresponding coordinates The set of point pairs formed Clustering to obtain each candidate i Corresponding category label i ;

[0120] Step 5: Calculate the maximum and minimum values ​​of all candidate segmentation times for each category label. The specific method is as follows:

[0121]

[0122]

[0123] in and These represent the maximum and minimum split times for candidate segments with category label j, respectively.

[0124] Step 6: Merge the category labels with overlapping intervals formed by the minimum and maximum values ​​of all candidate segmentation times. The specific method is as follows:

[0125] Enumerate the label pairs (i,j), if Then change all the labels j in the label to i, and finally there will be K category labels remaining;

[0126] Step 6: Calculate the maximum and minimum values ​​of all candidate segmentation times for each category label to obtain the action segmentation time S. DBSCAN The specific method is as follows:

[0127]

[0128] Where S DBSCAN The union of the maximum and minimum values ​​of the candidate split times for each category label;

[0129] Step 7: Heuristically solve the action segmentation time S using a genetic algorithm. geneticThe specific method is as follows:

[0130] Let S′1,…,S′ k As parameters to be optimized, a genetic algorithm is used for optimization.

[0131]

[0132] Get S genetic ={S′1,…,S′ k};

[0133] Step 8: Combine the action segmentation times calculated by the DBSCAN algorithm and the genetic algorithm to obtain the final segmentation set {S1,…,S}. k The specific method is as follows:

[0134] S = S DBSCAN ∪S genetic ={S1,...,S k}

[0135] Furthermore, in one implementation, the motion segment quality annotation module includes a video loading module to be annotated, a video to be annotated and standard motion video playback module, and a motion segment quality annotation, saving and loading module;

[0136] The video loading module is used to call the loading function in the backend processing and calculation module to obtain a list of videos to be labeled and display it on the frontend interface.

[0137] The video to be annotated and standard action video playback module is used to play the video to be annotated and the standard action video corresponding to the action in the front-end interface.

[0138] The motion segment quality annotation, saving, and loading module is used to call the save and modify function in the backend processing and calculation module to save the annotated motion segment quality information to the server. It can also call the loading function to display the saved annotation information to the front end.

[0139] Example:

[0140] This invention discloses an auxiliary annotation system for motion quality assessment datasets based on inter-frame differential waveform diagrams. With a simple user interface and the assistance of inter-frame differential waveform diagrams, the system facilitates the segmentation and annotation of motion quality assessment datasets, improving annotation efficiency. For example, when applied to annotating the Baduanjin (Eight Pieces of Brocade) motion quality assessment dataset, this system can calculate the inter-frame differential waveform diagram of the Baduanjin video to be annotated. From the waveform diagram, the start and end times of each movement in Baduanjin can be obtained, greatly reducing the number of times and time spent manually dragging the progress bar, thus improving annotation efficiency. After segmenting the motion segments, the quality of the motion segments can be annotated based on their differences from standard movements.

[0141] The present invention provides an auxiliary annotation system for motion quality assessment datasets based on inter-frame differential waveform diagrams, comprising: a framework for an auxiliary annotation system for motion quality assessment datasets, a motion segment start and end time annotation module, and a motion segment quality annotation module.

[0142] The action quality assessment dataset auxiliary annotation system framework includes an annotation system front-end module and a back-end processing and calculation module. When the system front-end module needs to perform operations such as loading, calculating, and saving modifications, it calls the back-end processing and calculation module to complete the above loading, calculation, and saving modification operations on the server side.

[0143] The action segment start and end time annotation module includes a video loading module, a video playback module, an action start and end time auxiliary search module, and an action segment start and end time annotation, saving, and loading module. It can load the video to be annotated and then play it, and use the inter-frame differential waveform diagram to help find the action start and end time before annotating the start and end time.

[0144] The motion segment quality annotation module includes a video loading module, a video and standard motion video playback module, and a motion segment quality annotation, saving, and loading module. It can load and play the segmented motion segments and perform quality annotation after comparing them with standard motion.

[0145] When the action segment start and end time annotation module and the action segment quality annotation module described in this invention perform action segment start and end time annotation and action segment quality annotation, the action quality evaluation dataset auxiliary annotation system framework provides a unified implementation method for the function functions of the system front-end module and the back-end processing and calculation module, including:

[0146] Define the form of the backend interface called by the system frontend module, and call the backend processing and computing modules according to the form of the backend interface;

[0147] Define a unified implementation method for the functions of the backend processing and calculation modules, and implement the functions of the action quality assessment dataset auxiliary annotation system framework according to the backend interface form. The functions include loading, calculation and saving / modification functions.

[0148] The action segment start and end time annotation module described in this invention includes a video loading module, a video playback module, an action start and end time auxiliary search module, and an action segment start and end time annotation, saving, and loading module.

[0149] The video loading module is used to call the loading function in the backend processing and calculation module to obtain a list of videos to be labeled and display it on the frontend interface. In this example, the video of Baduanjin (Eight Pieces of Brocade) to be labeled can be loaded.

[0150] The video playback module is used to play the video to be labeled on the front-end interface. In this example, the video of the Eight-Section Brocade exercise to be labeled can be played.

[0151] The action start and end time auxiliary search module is used to call the calculation function in the backend processing and calculation module to calculate the inter-frame differential waveform of the current video to be labeled and display it on the front-end interface to assist in the labeling of the start and end times of action segments. In this example, the inter-frame differential waveform can be used to assist in finding the start and end times of each segment of the Eight-Section Brocade exercise.

[0152] The action segment start and end time annotation, saving, and loading module is used to call the save and modify function in the backend processing and calculation module to save the annotated action segment start and end time information to the server. It can also call the loading function to display the saved annotation information on the front end. In this example, the start and end times of each segment of the Eight-Section Brocade exercise can be annotated and saved to the server, or the annotated Eight-Section Brocade action segment start and end times can be loaded and displayed on the front end.

[0153] like Figure 1 and Figure 3 As shown in the figure, this invention provides a flowchart illustrating the motion segment start and end time annotation module in the context of motion segment start and end time annotation in the Eight-Section Brocade motion quality assessment dataset, including the following steps:

[0154] Step 101: Click the Load button to request the backend to return the file names of all videos to be labeled, which will then be displayed in the frontend list.

[0155] Step 102: Double-click the file name in the file list, select the video of the Eight Pieces of Brocade exercise to be annotated, and request the backend to return the address of the Eight Pieces of Brocade exercise video;

[0156] Step 103: Play the selected Baduanjin exercise video;

[0157] Step 104: Request the backend to query whether there is any start and end time annotation information for the existing action segments in the current Baduanjin action video to be annotated. If so, return the information and display it on the frontend.

[0158] Step 105: Click the "Calculate Waveform" button to request the backend to calculate the inter-frame differential waveform of the current Baduanjin movement video to be annotated. After the backend returns the waveform, the frontend displays it.

[0159] Step 106: Drag the two sliders on the progress bar at the bottom of the video to mark the start and end times of the action. Click the button to the right of the sliders to add the marks to the mark list on the right.

[0160] Step 107: After completing the annotation, click the save button to request the backend to save the annotation information to the server;

[0161] Step 108: If there is another video that needs to be annotated, click the "Next" button to jump to step 102;

[0162] Step 109: Complete the annotation.

[0163] The calculation function in the back-end processing and calculation module described in this invention example is an algorithm for calculating inter-frame differential waveforms, including:

[0164] Step 201: Read the video F to be labeled, which has N frames.

[0165] Step 202: Convert the read video frames to be labeled into grayscale images.

[0166]

[0167] in and Let be the blue, green, and red channel values ​​of the pixel at position (x, y) in the i-th frame of video F, respectively. Convert them into a single grayscale channel value by weighted summation.

[0168] Step 203: Calculate the inter-frame difference of the video after conversion to grayscale:

[0169]

[0170] in and Let represent the grayscale images of frame i+1 and frame i respectively, and then subtract them to obtain D;

[0171] Step 204: Calculate the modulus of the video frame difference;

[0172] Norm i =||Di ||2

[0173] Where ||||2 represents finding the 2-norm.

[0174] Step 205, apply a moving average filter to the Norm:

[0175]

[0176] Where M is the size of the moving average window, k is the index of the window, and the formula adds the values ​​of Norm within the window and divides them by the window size to obtain the filtered signal y;

[0177] Step 206, based on the filtered signal y from step 5 i Draw a waveform diagram of the magnitude of the video frame difference as a function of the video frame.

[0178] Step 207, find the segmentation time S1,…,S of the action segment. k The specific method is as follows:

[0179]

[0180] Where argmin represents the segmentation time S1,…,S that minimizes the objective function. k ;

[0181] Step 208, divide the time intervals S1, ..., S k Plot the waveform of the differential mode change between the video frames and return it.

[0182] In this embodiment, the segmentation time S1,…,S for finding action segments k Specific methods include:

[0183] Step 1, calculate the filtered signal y i The x-axis values ​​for peaks and valleys;

[0184] Step 2, the filtered signal y i The start and end x-coordinates, peak x-coordinates, and trough x-coordinates are combined together, as follows:

[0185] peaks_valleys={0}∪peaks∪valleys∪{N-1}

[0186] Where ∪ represents finding the union of sets, and peaks_valleys is the set of horizontal coordinates after the union;

[0187] Step 3, based on the filtered signal y iTo determine the magnitude of change at the x-axis of peaks_valleys, remove the x-axis values ​​of peaks_valleys with smaller changes. The specific method is as follows:

[0188]

[0189] Among them, by using peaks_valleys as the x-axis i+1 and peaks_valleys i Signal at the location and Take the absolute value of the difference and retain those with a change greater than 0.1. and The x-coordinate of the maximum value in peaks_valleys i This yields the candidate segmentation time set, candidate;

[0190] Step 4: Use the DBSCAN clustering algorithm to segment the candidate time. i and the filtered signal at the corresponding coordinates The set of point pairs formed Clustering to obtain each candidate i Corresponding category label i ;

[0191] Step 5: Calculate the maximum and minimum values ​​of all candidate segmentation times for each category label. The specific method is as follows:

[0192]

[0193]

[0194] in and These represent the maximum and minimum split times for candidate segments with category label j, respectively.

[0195] Step 6: Merge the category labels with overlapping intervals formed by the minimum and maximum values ​​of all candidate segmentation times. The specific method is as follows:

[0196] Enumerate the label pairs (i,j), if Then change all the labels j in the label to i, and finally there will be K category labels remaining;

[0197] Step 6: Calculate the maximum and minimum values ​​of all candidate segmentation times for each category label to obtain the action segmentation time S. DBSCAN The specific method is as follows:

[0198]

[0199] Where S DBSCAN The union of the maximum and minimum values ​​of the candidate split times for each category label;

[0200] Step 7: Heuristically solve the action segmentation time S using a genetic algorithm. genetic The specific method is as follows:

[0201] Let S′1,…,S′ k As parameters to be optimized, a genetic algorithm is used for optimization.

[0202]

[0203] Get S genetic ={S′1,…,S′ k};

[0204] Step 8: Combine the action segmentation times calculated by the DBSCAN algorithm and the genetic algorithm to obtain the final segmentation set {S1,…,S}. k The specific method is as follows:

[0205] S = S DBSCAN ∪S genetic ={S1,...,S k}

[0206] The motion segment quality annotation module described in this invention includes a video loading module, a video and standard motion video playback module, and a motion segment quality annotation, saving, and loading module.

[0207] The video loading module is used to call the loading function in the backend processing and calculation module to obtain a list of videos to be labeled and display it on the front-end interface. In this example, it can load the Eight-Section Brocade movement segments whose movement quality needs to be labeled;

[0208] The video to be labeled and standard action video playback module is used to play the video to be labeled and the standard action video corresponding to the action on the front-end interface. In this example, it can play the Eight-Section Brocade movement segment with the quality of the action to be labeled and the standard action video corresponding to the action;

[0209] The motion segment quality annotation, saving, and loading module is used to call the save and modification function in the backend processing and calculation module to save the annotated motion segment quality information to the server, and can also call the loading function to display the saved annotation information on the front end. In this example, the annotated Baduanjin motion segment quality information can be saved to the server, or it can be loaded and displayed on the front end.

[0210] like Figure 2 and Figure 4 As shown, this invention provides a flowchart illustrating the motion segment quality annotation module in the motion quality annotation scenario of the Baduanjin motion quality assessment dataset, including the following steps:

[0211] Step 301: Click the Load button to request the backend to return the file names of all videos to be labeled, which will then be displayed in the frontend list.

[0212] Step 302: Double-click the file name in the file list, select the video clip of the Eight Pieces of Brocade exercise to be annotated, and request the backend to return the address of the video clip.

[0213] Step 303: Play the selected Eight Pieces of Brocade movement clip video and the corresponding standard movement video;

[0214] Step 304: Request the backend to query whether there is existing quality annotation information for the current Baduanjin movement segment video to be annotated. If so, return the information and display it on the frontend.

[0215] Step 305: Click the five-pointed star to mark the quality of the Eight Pieces of Brocade movement segment;

[0216] Step 306: After completing the annotation, click the save button to request the backend to save the annotation information to the server;

[0217] Step 307: If there is another video that needs to be annotated, click the "Next" button to jump to step 302;

[0218] Step 308: Complete the annotation.

[0219] Using this invention, the approximate start and end times of an action segment can be found with the assistance of inter-frame differential waveform diagrams. Therefore, compared with existing annotation systems, the system described in this invention can significantly reduce the number of times and time spent manually dragging the progress bar, thus improving annotation efficiency. Furthermore, during action segment quality annotation, the current action segment to be annotated can be compared with the corresponding standard action, which helps improve the accuracy of action quality annotation.

[0220] Using this invention, the approximate start and end times of an action segment can be found with the assistance of inter-frame differential waveform diagrams. Therefore, compared with existing annotation systems, the system described in this invention can significantly reduce the number of times and time spent manually dragging the progress bar, thus improving annotation efficiency. Furthermore, during action segment quality annotation, the current action segment to be annotated can be compared with the corresponding standard action, which helps improve the accuracy of action quality annotation.

[0221] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content and some or all of the steps in various embodiments of the action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams provided by this invention. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0222] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MCU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0223] This invention provides an idea and method for an auxiliary annotation system for motion quality assessment datasets based on inter-frame differential waveform diagrams. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A dataset-aided annotation system for motion quality assessment based on inter-frame differential waveform diagrams, characterized in that, The system includes: a framework for an auxiliary annotation system for motion quality assessment datasets, a module for annotating the start and end times of motion segments, and a module for annotating the quality of motion segments, wherein: The motion quality assessment dataset auxiliary annotation system framework includes: an annotation system front-end module and a back-end processing and calculation module. The annotation system front-end module is used to perform operations such as loading the video to be annotated and the annotation, calculating the inter-frame differential waveform, and saving and modifying the annotation. The back-end processing and calculation module is used to perform the above operations on the server side. The action segment start and end time annotation module is used to load the video to be annotated and play it. After finding the start and end times of the action with the help of the inter-frame differential waveform, the start and end times are annotated to obtain the segmented action segments. The action segment start and end time annotation module includes: an action start and end time auxiliary finding module, which is used to call the inter-frame differential waveform calculation function in the back-end processing and calculation module to calculate the inter-frame differential waveform of the current video to be annotated and display it in the front-end interface to assist in the annotation of the start and end times of the action segments. The motion segment quality annotation module is used to load the segmented motion segments, play them, and perform quality annotation after comparing them with standard motion segments. The calculation function in the back-end processing and calculation module, namely, calculating the inter-frame differential waveform, specifically includes: Step 1, Read the video to be labeled There are a total of Frame image; Step 2: Convert the read video frames to be labeled into grayscale images; Step 3: Calculate the inter-frame difference of the video after it has been converted to grayscale; Step 4: Calculate the modulus of the video frame difference; Step 5: Perform a moving average filter on the magnitude of the difference between video frames; Step 6, based on the filtered signal from Step 5 Draw a waveform diagram of the magnitude of the video frame difference as a function of the video frame. Step 7, find the segmentation time of the action segment ; Step 8, divide the time Plot the waveform of the inter-frame differential mode variation and return it; Step 7 describes finding the segmentation time of the action segment. Specific methods include: Step 7-1: Calculate the filtered signal. The x-coordinate of the peak and the valley ; Step 7-2, the filtered signal The start and end x-coordinates, peak x-coordinates, and trough x-coordinates are combined together, as follows: ； in, Represents finding the union of sets. The merged set of x-coordinates; Step 7-3, based on the filtered signal exist The magnitude of change at the x-axis will be less than a threshold. Remove the x-axis to obtain the candidate segmentation time set. ; Step 7-4: Use the DBSCAN clustering algorithm to select candidate segmentation times. and the filtered signal at the corresponding coordinates The set of point pairs formed Clustering, to obtain each Corresponding category tags ; Step 7-5: Calculate the maximum and minimum values ​​of all candidate split times for each category label. and The specific method is as follows: ； ； in and These represent category labels respectively. The maximum and minimum values ​​of the candidate segmentation time; Step 7-6: Merge the category labels with overlapping intervals formed by the minimum and maximum values ​​of all candidate segmentation times. The specific method is as follows: Enumerate label pairs ,if Then All tags in Change to The final remaining Category labels; Step 7-7: Calculate the maximum and minimum values ​​of all candidate segmentation times for each category label to obtain the action segmentation time. ; Steps 7-8: Heuristically solve the action segmentation time using a genetic algorithm. ; Steps 7-9: Combine the action segmentation times calculated by the DBSCAN algorithm and the genetic algorithm to obtain the final segmentation set. The specific method is as follows: ； in, That is, the final split set .

2. The action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams according to claim 1, characterized in that, The action segment start and end time annotation module further includes: a video loading module, a video playback module, and an action segment start and end time annotation, saving, and loading module, wherein: The video loading module is used to call the video loading function in the backend processing and calculation module to obtain a list of videos to be labeled and display it on the frontend interface. The video playback module is used to play videos from the list of videos to be labeled in the front-end interface. The action segment start and end time annotation, saving and loading module is used to call the annotation saving and modification function in the backend processing and calculation module to annotate the current video to be annotated and save the annotated action segment start and end time information to the server, or call the annotation loading function to display the saved annotation information on the front end interface.

3. The action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams as described in claim 2, characterized in that, The calculation of the inter-frame differential waveform specifically includes: Step 1, Read the video to be labeled There are a total of Frame image; Step 2: Convert the read video frames to be labeled into grayscale images. The specific method is as follows: ； in, , and Videos No. Frame located The blue, green, and red channel values ​​of the pixel at the location. , and As sensitivity weights, they are converted into single grayscale channel values ​​through a weighted summation. ; Step 3: Calculate the inter-frame difference of the video after conversion to grayscale. The specific method is as follows: ；； in, and Representing the first Frame and the The grayscale images of the frames are subtracted to obtain the first frame. Frame-to-frame difference ; Step 4: Calculate the magnitude of the video frame difference, using the following method; ； in, To find the L2 norm, The modulus representing the difference between video frames; Step 5, modulate the inter-frame difference of the video. The specific method for performing moving average filtering is as follows: ； in, The size of the moving average window. The index within the window is the filtered signal. ; Step 6, based on the filtered signal from Step 5 Draw a waveform diagram of the magnitude of the video frame difference as a function of the video frame. Step 7, find the segmentation time of the action segment ; Step 8, divide the time Plot the waveform of the differential mode change between the video frames and return it.

4. The action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams according to claim 3, characterized in that, The process of obtaining the candidate segmentation time set as described in step 7-3 The specific method is as follows: ； Among them, by using the x-coordinate as and Signal at the location and Take the absolute value of the difference and retain those with a change greater than the threshold. times and x-coordinate of the maximum value .

5. The action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams according to claim 4, characterized in that, The process of obtaining the action segment segmentation time described in step 7-7 The specific method is as follows: ； in, This represents the union of the maximum and minimum values ​​of the candidate split times for each category label.

6. The action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams according to claim 5, characterized in that, Steps 7-8 describe the heuristic solution of action segmentation time using a genetic algorithm. The specific method is as follows: Will The parameters to be optimized are solved using a genetic algorithm: ； in, This represents the segmentation time required to minimize the objective function. Thus obtain .

7. The action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams according to claim 6, characterized in that, The motion segment quality annotation module includes: a video segment loading module for motion quality to be annotated, a video segment playback module for motion quality to be annotated and standard motion video, and a motion segment quality annotation, saving, and loading module, wherein: The video clip loading module for the action quality to be labeled is used to call the video loading function in the backend processing and calculation module to obtain a list of action clips to be labeled and display it on the front-end interface. The video clip to be labeled and the standard video playback module are used to play the video clip to be labeled and the standard video corresponding to the video clip in the front-end interface. The motion segment quality annotation, saving, and loading module is used to call the annotation saving and modification function in the backend processing and calculation module to annotate the current motion segment and save the annotated motion segment quality information to the server, or call the annotation loading function to display the saved annotation information on the front-end interface.

8. The action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams according to claim 7, characterized in that, The sensitivity weights mentioned in step 2 are weight values ​​assigned based on the human eye's sensitivity to different colors. The optimal weight values ​​are set as follows: 。 9. The action quality assessment dataset auxiliary annotation system based on inter-frame differential waveform diagrams according to claim 8, characterized in that... The threshold mentioned in step 7-3 The value is 0.1.

Images