Accuracy assessment method and system for automatically generated video slices

Abstract

The invention discloses an accuracy evaluation method and system for automatically generated video slices. The method comprises the following steps: receiving system slice division information of a target video automatically generated by a slice division system and manual slice division information of the target video manually; the slice division information comprises slice category information and slice starting and ending time information; according to the system slice division information and the manual slice division information, calculating a quantity difference ratio, a time overlap ratio ratio and a category consistency ratio of the system division slices and the manual division slices; and calculating a total score of system slice division information accuracy evaluation through a weighted average method according to a preset weight ratio of the slice number difference ratio, the time overlap ratio ratio and the slice category consistency ratio. Through the method, the evaluation efficiency of automatically generating the slices by the system can be improved, and the accuracy of an evaluation result is improved.

Description

Technical Field

[0001] This application relates to the technical field of video processing, and in particular, to a method and system for automatically evaluating the accuracy of video slices. Background Art

[0002] In today's digital age, with the explosive growth of video data volume, how to efficiently manage and utilize these resources has become a key issue. Video slicing technology has emerged. It refers to dividing a long video into multiple independent segments according to dimensions such as time, content, or semantics, for the convenience of content retrieval, analysis, and reuse. This technology can not only help users quickly locate the video segments they are interested in, but also provide basic support for in-depth mining and personalized recommendation of video content. For example, in intelligent education, classroom content slicing is an important basis for summarizing the entire class, evaluating teachers' behaviors, and students' knowledge consolidation after class. Classroom content slicing, that is, using artificial intelligence technology and combining relevant standards, divides classroom teaching content into interactive segments, discussion segments, blank segments, preview learning segments, homework arrangement segments, and other segments.

[0003] With the explosive growth of video content, automatic video slicing systems have emerged and become an important tool for video content management and reuse. These systems can automatically extract exciting segments from long videos or slice according to semantics through artificial intelligence and machine learning technologies, greatly improving the efficiency of video processing.

[0004] However, there are still many deficiencies in the accuracy evaluation of these automatic slicing systems. Traditional evaluation methods mainly rely on manual comparison of the slices generated by the system with the manually labeled slices one by one. This method is not only time-consuming and laborious, but also easily affected by subjective factors, resulting in inaccurate and objective evaluation results. In addition, manual comparison is difficult to handle large-scale video data, with low efficiency and unable to meet the requirements of fast and accurate evaluation in practical applications. Summary of the Invention

[0005] Based on this, in view of the above technical problems, a method and system for automatically evaluating the accuracy of video slices are provided to solve the problems of low evaluation efficiency and low evaluation accuracy of the existing technology for the slice division results of the system.

[0006] In a first aspect, a method for automatically evaluating the accuracy of video slices, the method includes:

[0007] Receiving the system slice division information of the target video automatically generated by the slice division system and the manual slice division information of the target video by humans; the slice division information includes slice category information and slice start and end time information;

[0008] Calculate the ratio of the difference in the number of slices between the system - divided slices and the manually - divided slices according to the system slice division information and the manual slice division information;

[0009] Calculate the ratio of the time overlap between the system - divided slices and the manually - divided slices according to the start - end time of the system slices and the start - end time information of the manual slices;

[0010] Calculate the ratio of the consistency of the slice categories between the slices generated by the system and the slice categories of the manual slices according to the system slice division information and the manual slice division information;

[0011] Calculate the total score for evaluating the accuracy of the system slice division information by weighted - averaging the ratio of the difference in the number of slices, the ratio of the time overlap, and the ratio of the consistency of the slice categories according to the preset weight ratio.

[0012] In the above solution, optionally, the calculation of the ratio of the difference in the number of slices between the system - divided slices and the manually - divided slices includes:

[0013] Calculate the number of slices of the target video divided by the system and the number of slices of the target video divided manually;

[0014] Calculate the ratio of the difference in the number of slices by dividing the smaller number of slices between the number of slices of the target video divided by the system and the number of slices of the target video divided manually by the larger number of slices.

[0015] In the above solution, optionally, the calculation of the ratio of the time overlap between the system - divided slices and the manually - divided slices includes:

[0016] Calculate the absolute time - difference duration between each time endpoint of the system - divided slices and the nearest time endpoint of the manually - divided slices;

[0017] Add up the absolute time - difference durations between all time endpoints of the system - divided slices and the nearest time endpoints of the manually - divided slices to obtain the total difference duration;

[0018] Subtract the total difference duration from the total absolute duration of the target video, and then divide by the total absolute duration of the target video to obtain the ratio of the time overlap.

[0019] In the above solution, optionally, the calculation of the ratio of the consistency of the slice categories between the slices generated by the system and the slice categories of the manual slices further includes:

[0020] Obtain the time period and category information of each slice generated by the system;

[0021] Within the time period of each system slice, obtain the category of the manually - divided slices, calculate the number of slices of the same category in the manual division, and take the category with the largest number as the dominant category of the manual division during this period;

[0022] If the number of all categories is the same, calculate the slice duration corresponding to each category manually divided during the time period of each system slice, and take the category with the maximum duration as the dominant category of manual division for this period of time;

[0023] Compare each slice category generated by the system with the dominant category of manual division corresponding to the same time period. If they are the same, increment the count by 1, and calculate the total number of slice categories generated by the system that are consistent with the dominant category of manual division corresponding to the same time period;

[0024] Divide the total number of slice categories generated by the system that are consistent with the dominant category of manual division corresponding to the same time period by the total number of slices generated by the system to obtain the slice category consistency ratio.

[0025] In the above solution, further optionally, after calculating the slice duration corresponding to each category manually divided during the time period of each system slice and taking the category with the maximum duration as the dominant category of manual division for this period of time, it further includes:

[0026] If the durations of all categories manually divided during the time period of each slice generated by the system are equal, then take the multiple categories with equal durations as the dominant categories of manual division for this period of time respectively;

[0027] Compare each slice category generated by the system with the multiple dominant categories of manual division corresponding to the same time period. If one of them is the same, increment the count by 1.

[0028] In the above solution, optionally, the preset weight ratio is the quantity difference ratio: the time overlap ratio: the slice category consistency ratio = 1:1:3.

[0029] In a second aspect, a system for evaluating the accuracy of automatically generated video slices, the system includes:

[0030] A division information receiving module: used to receive the system slice division information of the target video automatically generated by the slice division system and the manual slice division information of the target video by humans; the slice division information includes slice category information and slice start and end time information;

[0031] A quantity difference ratio calculation module: used to calculate the quantity difference ratio between the system-divided slices and the manually-divided slices according to the system slice division information and the manual slice division information;

[0032] A time overlap ratio calculation module: used to calculate the time overlap ratio between the system-divided slices and the manually-divided slices according to the start and end times of the system slices and the start and end time information of the manual slices;

[0033] Category consistency ratio calculation module: used to calculate the slice category consistency ratio between the slice categories of the slices generated by the system and the slice categories of the manually sliced categories according to the system slice division information and the manual slice division information;

[0034] Total score calculation module for evaluating the accuracy of slice information: used to calculate the total score for evaluating the accuracy of the system slice division information through a weighted average method according to the preset weight ratio of the slice quantity difference ratio, time coincidence ratio, and slice category consistency ratio.

[0035] This application has at least the following beneficial effects:

[0036] This application obtains the system slice division information of the target video automatically generated by the slice division system and the manual slice division information of the target video by a human, calculates the slice quantity difference ratio, time coincidence ratio, and slice category consistency ratio between the slices divided by the system and the manually divided slices; calculates the total score for evaluating the accuracy of the system divided slice information through a weighted average method according to the preset weight ratio of the slice quantity difference ratio, time coincidence ratio, and slice category consistency ratio. The evaluation of the system slice division is carried out from multiple dimensions, and finally the various evaluation results are weighted and summed, thereby realizing automatic evaluation, improving the evaluation efficiency, and at the same time improving the evaluation accuracy of the system slice division result. Description of the Drawings

[0037] Figure 1 It is a schematic flowchart of a method for evaluating the accuracy of automatically generated video slices provided by an embodiment of this application;

[0038] Figure 2 It is a comparison diagram of inference slice information and annotation slice information provided by an embodiment of this application;

[0039] Figure 3 It is a comparison diagram of inference slice information and annotation slice information provided by an embodiment of this application;

[0040] Figure 4 It is a comparison diagram of inference slice information and annotation slice information provided by an embodiment of this application;

[0041] Figure 5 It is a comparison diagram of inference slice information and annotation slice information provided by an embodiment of this application;

[0042] Figure 6 It is a comparison diagram of inference slice information and annotation slice information provided by an embodiment of this application;

[0043] Figure 7 It is a specific schematic flowchart of the schematic flowchart of a method for evaluating the accuracy of automatically generated video slices in an embodiment of this application. Detailed implementation manners

[0044] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The tags in the video slices refer to the keywords or descriptive texts attached to the video segments, which are used to identify the content, theme or features of the video segments. These tags can help users quickly understand the core content of the video segments. In intelligent education, the classroom content slices are an important basis for summarizing the entire class, evaluating the teacher's behavior, and helping students consolidate knowledge after class. The classroom content slices are obtained by using artificial intelligence technology and combining relevant standards to divide the classroom teaching content into interaction segments, discussion segments, blank segments, preview learning segments, homework assignment segments, and other segments. The specific categories and descriptions are shown in Table 1:

[0046] Table 1

[0047]

[0048]

[0049] Current drawbacks of evaluating the accuracy of classroom content slices:

[0050] (1) Manual evaluation is time-consuming and laborious, and there is no unified measurement standard. After working for a long time, due to human subjectivity, subjective judgments will be formed on the classroom content slices, and the results of the model cannot be objectively evaluated.

[0051] (2) Common slicing systems only perform slicing division, but do not evaluate the slicing division results and have no user feedback mechanism.

[0052] In one embodiment, as Figure 1 shown, a method for automatically evaluating the accuracy of video slices is provided, and the method includes:

[0053] Step S1: Receive the system slice division information of the target video automatically generated by the slice division system and the manual slice division information of the target video by humans; the slice division information includes slice category information and slice start and end time information.

[0054] Step S2: Calculate the quantity difference ratio between the system-divided slices and the manually-divided slices according to the system slice division information and the manual slice division information.

[0055] Specifically, step S2 specifically includes: calculating the number of target video slices divided by the system and the number of slices of the target video manually divided;

[0056] Using the smaller number of slices among the number of target video slices divided by the system and the number of slices of the target video manually divided, divide it by the larger number of slices to calculate the quantity difference ratio.

[0057] The quantity difference ratio (slice count ratio, abbreviated as scr) is one of the important indicators for evaluating the automatic slicing effect of the system. It mainly measures the degree of difference between the number of classroom content slices automatically generated by the system and the number of classroom content slices manually marked by experts. This indicator can intuitively reflect whether the granularity of the system slices is appropriate. If the difference in the number of the two is large, it may mean that the system slices are too fine or too coarse and need to be optimized and adjusted accordingly. By comparing these two quantities, the accuracy of the system slices can be initially judged. The quantity difference ratio ranges between 0 and 1. The closer it is to 1, the smaller the difference between the number of slices generated by the system and the number of slices manually marked.

[0058] The specific calculation formula is as follows:

[0059]

[0060] Note: Use the maximum value of the number of classroom content slices manually marked and system-inferred as the denominator, and the minimum value as the numerator. In this calculation formula, the larger the value, the closer the manual marking result is to the system automatic slicing result, and vice versa.

[0061] Step S3: According to the start and end times of the system slices and the start and end time information of the manual slices, calculate the time overlap ratio of the system-divided slices and the manually-divided slices.

[0062] Specifically, step S3 includes: calculating the absolute time difference of each time endpoint of the system-divided slices from the nearest time endpoint of the manually-divided slices;

[0063] Add up the absolute time differences of all time endpoints of the system-divided slices from the nearest time endpoint of the manually-divided slices to obtain the total difference duration;

[0064] Subtract the total difference duration from the total absolute duration of the target video, and then divide it by the total absolute duration of the target video to obtain the time overlap ratio.

[0065] The slice time overlap ratio (STOR) is an important quantitative metric for evaluating slice accuracy. It is evaluated by calculating the proportion of the overlapping part between the slice time period automatically predicted by the system and the slice time period marked by human experts. Specifically, this metric calculates the ratio of the sum of the absolute values of the differences between the endpoints of the automatically sliced video by the system and the endpoints marked by human experts subtracted from the total video duration, and the total video duration. The value range is between 0 and 1. The higher the overlap ratio, the closer the predicted time boundary by the system is to the human annotation, and the better the slicing effect of the system. This metric can intuitively reflect the accuracy of the system's slicing in the time dimension.

[0066] The specific calculation formula is as follows:

[0067] 1: Marked slice time period:

[0068] 2: Inferred slice time period:

[0069] 3: Calculate the closest endpoint between the endpoints of the inferred segment and the endpoints of the marked segment (ε is a very small number;

[0070] 4:

[0071] Step S4: According to the system slice division information and the manual slice division information, calculate the slice category consistency ratio between the slice categories generated by the system and the slice categories of the manual slices.

[0072] Specifically, step S4 includes: obtaining the time period and category information of each slice generated by the system;

[0073] Within the time period of each of the system slices, obtain the category of the manually divided slices, calculate the number of slices in the same category manually divided, and take the category with the largest number as the dominant category of the manual division for this period of time;

[0074] If the number of all categories is the same, calculate the slice duration corresponding to each category manually divided within the time period of each of the system slices, and take the category with the largest duration as the dominant category of the manual division for this period of time;

[0075] Compare each slice category generated by the system with the dominant category of the manual division corresponding to the same time period. If they are the same, increment the count, and calculate the total number of slice categories generated by the system that are consistent with the dominant category of the manual division corresponding to the same time period;

[0076] If, in the time period of each slice generated by the system, the durations of all categories manually divided are equal, then the multiple categories with equal durations are respectively used as the dominant categories manually divided for this period of time;

[0077] Compare each slice category generated by the system with the multiple dominant categories manually divided corresponding to the same time period. If one of them is consistent, the count is incremented by 1.

[0078] Divide the total number of slice categories generated by the system that are consistent with the dominant categories manually divided corresponding to the same time period by the total number of all slices generated by the system to obtain the slice category consistency ratio.

[0079] The slice category consistency ratio (abbreviated as slcr) is a key indicator for evaluating the quality of slices, mainly used to measure the matching degree between the slice labels generated by the system and the manually annotated labels.

[0080] This indicator is calculated through the following methods: First is statistical verification, which conducts statistical analysis on the labels generated by the system and the manually annotated labels; second is category statistics, which calculates the number of categories of the manually annotated labels corresponding to a single system slice time period, and takes the label with the largest number of categories as the result of the manually annotated for this period of time. If the slice category generated by the system is consistent with the slice category manually annotated, the number of consistent slice segments is incremented by 1, otherwise it remains unchanged;

[0081] If the numbers of multiple categories are the same, the judgment is made through the following steps:

[0082] ①. Take out the inference time period segment by segment;

[0083] ②. In this time period, calculate the sum of the durations of the category labels of the annotated data to obtain the label corresponding to the maximum value;

[0084] ③. If the obtained label is consistent with the label of this inference time period, increment by 1.

[0085] Finally, calculate the ratio of the number of slice segments with consistent system-generated labels and manually annotated labels to the total number of system-generated slice segments to obtain the label consistency ratio score. Through the above methods, the accuracy of slice labels can be comprehensively evaluated, providing an important reference basis for optimizing the system performance. This indicator not only reflects the accuracy of the system in semantic understanding but also reflects the reliability of the system's automatic slicing results.

[0086] The specific calculation formula is as follows:

[0087]

[0088] Step S5: Calculate the total score for evaluating the accuracy of the system slice division information by using the weighted average method according to the preset weight ratio for the slice quantity difference ratio, time coincidence ratio, and slice category consistency ratio.

[0089] In step S5, the preset weight ratio is slice quantity difference ratio ∶ time coincidence ratio ∶ slice category consistency ratio = 1 ∶ 1 ∶ 3. Calculate the total score of the automatic slice generation system by using the weighted average method.

[0090] The specific calculation formula is as follows:

[0091] Score = 0.2 * scr + 0.2 * stor + 0.6 * slcr.

[0092] As Figure 2 shown, for the better case (quantity, time, and label are all close), the corresponding calculation is:

[0093]

[0094] As Figure 3 shown, for the medium case (quantity, time, and label have medium differences), the corresponding calculation is:

[0095]

[0096] As Figure 4 shown, for the medium case (quantity, time, and label have medium differences), the corresponding calculation is:

[0097]

[0098] As Figure 5 shown, for the worse case (quantity, time, and label all have large differences), the corresponding calculation is:

[0099]

[0100] As Figure 6 , for the worse case (quantity, time, and label all have large differences), the corresponding calculation is:

[0101]

[0102] Figures 2 - 6 In , a, b, and c represent categories. Specifically, a is preview, b is others, and c is homework arrangement.

[0103] In the above method for automatically evaluating the accuracy of video slices, by obtaining the system slice division information of the target video automatically generated by the slice division system and the manual slice division information of the target video by humans, the quantity difference ratio, time coincidence ratio, and slice category consistency ratio between the slices divided by the system and the manually divided slices are calculated; the quantity difference ratio, time coincidence ratio, and slice category consistency ratio are weighted and averaged according to a preset weight ratio to calculate the total score for evaluating the accuracy of the system slice division information. The evaluation of the system slice division is carried out from multiple dimensions, and finally the weighted sum of various evaluation results is used to achieve automatic evaluation, improve the evaluation efficiency, and at the same time improve the accuracy of the evaluation of the system slice division results.

[0104] As Figure 7 shown, in the evaluation of the accuracy of slice content, it is to conduct a multi-dimensional and all-round evaluation of the slice results of the existing content automatic slicing system, calculate the scores of the evaluation indicators in each dimension, and finally obtain the total score by the method of weighted average, and objectively evaluate the content slice results according to the total score.

[0105] In one embodiment, there is provided an automatically generated video slice accuracy evaluation system, characterized in that the system includes:

[0106] A division information receiving module: used to receive the system slice division information of the target video automatically generated by the slice division system and the manual slice division information of the target video by humans; the slice division information includes slice category information and slice start and end time information;

[0107] A quantity difference ratio calculation module: used to calculate the quantity difference ratio between the slices divided by the system and the manually divided slices according to the system slice division information and the manual slice division information;

[0108] A time coincidence ratio calculation module: used to calculate the time coincidence ratio between the slices divided by the system and the manually divided slices according to the start and end times of the system slices and the start and end time information of the manual slices;

[0109] A category consistency ratio calculation module: used to calculate the slice category consistency ratio between the slice categories generated by the system and the slice categories of the manual slices according to the system slice division information and the manual slice division information;

[0110] A total score calculation module for evaluating the accuracy of slice information: used to calculate the total score for evaluating the accuracy of the system slice division information by weighted average according to a preset weight ratio for the slice quantity difference ratio, time coincidence ratio, and slice category consistency ratio.

[0111] For the specific limitations of an automatically generated video slice accuracy evaluation system, reference may be made to the limitations of an automatically generated video slice accuracy evaluation method described above, which will not be elaborated here. Each module in the above automatically generated video slice accuracy evaluation system can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0112] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0113] The above-described embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A method for evaluating the accuracy of automatically generated video slices, characterized in that: The method comprises: Receive system slicing division information of a target video automatically generated by a slicing division system, and manual slicing division information of the target video; the slicing division information includes slicing category information and slicing start and end time information; Calculating the quantity difference ratio of the system divided slices and the manual divided slices according to the system divided slice information and the manual divided slice information; Calculate the time overlap ratio between the system-divided slices and the manual-divided slices according to the start and end time information of the system-divided slices and the start and end time information of the manual-divided slices; Calculating the slice category consistency ratio between the slice category generated by the system and the manual slice category according to the system slice division information and the manual slice division information; The slice quantity difference ratio, time overlap ratio and slice category consistency ratio are calculated according to the preset weight ratio through the weighted average method to obtain the total score of the system slice division information accuracy assessment.

2. The method for automatically generating video slice accuracy assessment according to claim 1, characterized in that: The difference ratio between the number of slices divided by the computing system and the number of slices divided manually includes: The number of target video slices divided by the system and the number of target video slices divided manually are calculated; the number difference ratio is calculated by dividing the smaller number of slices of the target video slices divided by the smaller number of slices of the target video slices divided by the manually divided number of slices.

3. The method for automatically generating video slice accuracy assessment according to claim 1, characterized in that: The time overlap ratio between the computing system dividing the slices and the manual dividing the slices includes: Calculate the absolute time difference between each time endpoint of the slice divided by the system and the time endpoint of the closest slice divided manually; Add the absolute duration differences of all time endpoints of the system-divided slices and the time endpoint of the closest manually divided slice to obtain the total difference duration; The total difference duration is subtracted from the total absolute duration of the target video, and the result is divided by the total absolute duration of the target video to obtain the time overlap ratio.

4. The method for automatically generating video slice accuracy assessment according to claim 1, characterized in that: The slice category consistency ratio between the slice category generated by the computing system and the manual slice category also includes: Get the time period and category information of each slice generated by the system; In each time period of the system slices, the categories of the manually divided slices are obtained, and the number of slices of the same category that are manually divided is calculated, and the category with the largest number is taken as the dominant category of the manual division during this period; If the number of all categories is the same, then the slice duration corresponding to each manually divided category in the time period of each system slice is calculated, and the category with the largest duration is taken as the manually divided dominant category in this period; Compare each slice category generated by the system with the dominant category manually divided in the same time period. If they are consistent, the count is increased by 1. The total number of slice categories generated by the system that are consistent with the dominant category manually divided in the same time period is calculated. The total number of slice categories generated by the system that are consistent with the dominant categories manually divided in the same time period is divided by the number of all slices generated by the system to obtain the slice category consistency ratio.

5. The method for automatically generating video slice accuracy assessment according to claim 4, characterized in that: Calculating the slice duration corresponding to each manually divided category in the time period of each system slice, taking the category with the largest duration as the manually divided dominant category for this period of time also includes: If in the time period of each slice generated by the system, the durations of all manually divided categories are equal, then multiple categories with equal durations are respectively used as the dominant categories of the manually divided time period; Each slice category generated by the system is compared with multiple manually divided dominant categories corresponding to the same time period. If one of them is consistent, the count is increased by 1.

6. The method for automatically generating video slice accuracy assessment according to claim 1, characterized in that: The preset weight ratio is the ratio of quantity difference: the ratio of time overlap: the ratio of slice category consistency = 1:1:

3.

7. An automatically generated video slice accuracy assessment system, characterized in that: The system comprises: A partitioning information receiving module: used to receive the system slice partitioning information of the target video automatically generated by the slice partitioning system, and the manual slice partitioning information of the target video; the slice partitioning information includes slice category information and slice start and end time information; A quantity difference ratio calculation module is used to calculate the quantity difference ratio of the system divided slices and the manual divided slices according to the system slice division information and the manual slice division information; Time overlap ratio calculation module: used to calculate the time overlap ratio between the system-divided slices and the manual-divided slices according to the start and end time of the system slices and the start and end time information of the manual slices; Category consistency ratio calculation module: used to calculate the slice category consistency ratio between the slice category generated by the system and the manual slice category according to the system slice division information and the manual slice division information; Slice information accuracy assessment total score calculation module: used to calculate the total score of the system slice division information accuracy assessment by weighted averaging the slice quantity difference ratio, time overlap ratio and slice category consistency ratio according to the preset weight ratio.

Images