A nonlinear operation experimental video action intelligent scoring method and device

Abstract

The present application relates to a method and device for scoring an action in an experimental video of a nonlinear operation, and relates to the technical field of video action recognition. The method comprises: obtaining experimental video data, performing multi-dimensional analysis on the experimental video data through a fine-tuning model to obtain a temporal relationship of experimental actions in the experimental video data, performing action classification prediction through sliding window prediction based on the temporal relationship and the experimental video data to obtain a set of initial action intervals with consistency, confidence information, and action categories, performing action merging by eliminating breakpoints on the initial action intervals based on the confidence information and action categories through the fine-tuning model to obtain a target action interval with consistency, and scoring each target action interval according to a flexible scoring strategy to obtain an experimental action scoring result, thereby achieving accurate action interval positioning, being able to quickly find and focus on the video clip where the scoring point is located, and improving the accuracy and reliability of the scoring through a flexible scoring method.

Description

Technical Field

[0001] The present application relates to the technical field of video action recognition, and in particular to a method and device for intelligently scoring experimental video actions for nonlinear operations. Background Art

[0002] The rapid development of artificial intelligence (AI) technology has profoundly impacted the field of education, and has been applied to various grading processes such as grading of physics, chemistry, and biology laboratory exams. At present, AI technology is widely used in the education community at home and abroad to automatically evaluate laboratory exam videos. This method accurately scores each scoring point in the exam by analyzing the experimental operation videos recorded in the examination room. The AI ​​grading system not only improves the efficiency of grading, but also ensures the fairness and accuracy of grading through the teacher's review process. However, how to further improve the intelligence level of the grading system and how to optimize the grading process to make it more efficient and accurate are technical problems that need to be solved urgently.

[0003] Existing AI scoring algorithms mainly utilize a preset fixed-order scoring mechanism. Under standard circumstances, the scoring points are determined for the experimental actions in the video according to a preset specific order, and then the scoring points are scored. However, this rigid sequential scoring method cannot effectively cope with the complex situations in actual exams. For example, students may repeat the operation after completing a certain scoring point, or complete the experimental steps in a non-standard order (i.e., non-linear experimental steps). Existing technologies can only terminate the scoring process after identifying a preset number of scoring points, ignoring various auxiliary situations that may occur, such as repeated operations or changes in order, and have major limitations, which causes the system to miss additional valid operations, affecting the comprehensiveness and fairness of the scoring. Summary of the Invention

[0004] The present application provides a method and device for intelligent scoring of experimental video actions of nonlinear operations. Through innovative initial action interval positioning and flexible scoring methods, accurate action analysis and initial action interval positioning are achieved. Taking into full consideration the diversity and nonlinear characteristics of experimental operations, the method can quickly find and focus on the video clips where the scoring points are located. A more flexible and intelligent scoring method is used to adapt to the uncertainty of experimental operations, accurately capture scoring points, and avoid scoring errors caused by algorithm limitations. That is, the accuracy, reliability, comprehensiveness and fairness of scoring are improved through flexible scoring methods. It can handle various situations such as repeated operations and sequence adjustments that may occur in students during the experiment, and solve the problems of missed judgments and wrong judgments caused by the inability of existing technologies to be applicable to complex and changeable experimental operations.

[0005] In a first aspect, the present application provides a method for intelligently scoring experimental video actions for nonlinear operations, comprising:

[0006] Obtain experimental video data to be predicted;

[0007] Performing a multi-dimensional analysis on the experimental video data using a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data;

[0008] Based on the temporal relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain an initial action interval set containing the complete experimental action, confidence information, and action category, wherein the initial action interval set includes at least one initial action interval related to the experimental action;

[0009] By using the fine-tuning model, based on the confidence information and the action category, the initial action interval set is subjected to action merging by eliminating breakpoints, so as to obtain a target action interval with coherence corresponding to the experimental action;

[0010] According to a preset flexible scoring strategy, the target experimental action corresponding to each target action interval is scored to obtain an experimental action scoring result.

[0011] Optionally, performing a multi-dimensional analysis on the experimental video data using a preset fine-tuning model to obtain a temporal relationship of experimental actions in the experimental video data includes:

[0012] Inputting the experimental video data into the fine-tuning model;

[0013] The experimental video data is subjected to spatial dimension analysis and temporal dimension analysis by the fine-tuning model to obtain the temporal relationship between the contexts of the experimental actions in the experimental video data.

[0014] Optionally, performing action classification prediction through sliding window prediction based on the temporal relationship and the experimental video data to obtain an initial action interval set containing a complete experimental action, confidence information, and action category includes:

[0015] Using a sliding window technique, the experimental video data is scanned for video content according to a preset scanning method to obtain a sliding window set;

[0016] Based on the temporal relationship, a sliding window prediction method is used to predict the experimental actions in the sliding window set through the fine-tuning model to obtain at least one initial action interval related to the experimental action and an action category corresponding to the initial action interval;

[0017] For the action category, performing confidence prediction on the initial action interval using the fine-tuning model to obtain confidence information of the initial action interval;

[0018] The confidence information is used to characterize the correlation between the initial action interval and the action category.

[0019] Optionally, the performing prediction processing on the experimental actions in the sliding window set by the fine-tuning model to obtain at least one initial action interval related to the experimental action includes:

[0020] Performing sliding window prediction on the experimental actions in the sliding window set using the fine-tuning model to obtain at least one target sliding window containing the experimental actions;

[0021] Interval acquisition is performed based on the target sliding window to obtain at least one initial action interval corresponding to the experimental action, which includes a start frame and an end frame.

[0022] Optionally, the fine-tuning model is used to perform action merging by eliminating breakpoints on the initial action interval set based on the confidence information and the action category to obtain a coherent target action interval corresponding to the experimental action, including:

[0023] Get the preset confidence threshold;

[0024] For each confidence information corresponding to the initial action interval, when the confidence information is not lower than the confidence threshold, performing breakpoint elimination and merging of the corresponding initial action interval to obtain a coherent target action interval corresponding to the experimental action;

[0025] When the confidence information is lower than the confidence threshold, the initial action interval corresponding to the confidence information below the confidence threshold is filtered through a preset post-processing module, and the initial action interval corresponding to the confidence information not lower than the confidence threshold is merged with breakpoints eliminated to obtain a target action interval with continuity corresponding to the experimental action.

[0026] Optionally, performing interval filtering on the initial action intervals corresponding to the confidence information below the confidence threshold by a preset post-processing module includes:

[0027] According to formula C filtered =Filter(C,θ)=(l,[s,e])∈C|Conf([l,s,e])≥θ, the initial action interval corresponding to the confidence information below the confidence threshold is filtered, and the initial action interval whose confidence information is not lower than the confidence threshold is obtained according to the formula

[0028] Merge(C filtered , δ) = (l, [s′, e′]), l∈L, s′=min i∈I (si ), e′=max i∈I (e i ), the initial action interval C after filtering filtered Merge and use the time interval threshold δ for the merge operation;

[0029] Among them, θ is the preset confidence threshold, C filtered is the initial action interval after filtering, l is the experimental action, s is the starting frame corresponding to the experimental action, e is the ending frame corresponding to the experimental action, (l, [s, e]) is the initial action interval of the experimental action l, I satisfies l i = l, and for index i and j, the initial action interval [s i , e i ] and [s j , e j ]Close in time.

[0030] Optionally, scoring the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result includes:

[0031] When the flexible scoring strategy is a parallel scoring strategy, score identification is performed on all experimental actions corresponding to the target action interval according to the parallel scoring strategy to obtain a score action point corresponding to each experimental action;

[0032] At the same time, all scoring action points are judged in parallel to obtain the action score corresponding to each scoring action point;

[0033] An experimental action scoring result is determined according to the action scoring.

[0034] Optionally, scoring the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result includes:

[0035] When the flexible scoring strategy is a nonlinear scoring strategy, identifying nonlinear operations in the target action interval;

[0036] Performing score point identification on the nonlinear operation to obtain duplicate score points and / or jump score points, wherein the duplicate score points are score points corresponding to repetitive nonlinear operations, and the jump score points are score points corresponding to jumpy nonlinear operations;

[0037] Scoring and judging are performed based on the duplicate removal scoring points and / or jump scoring points to obtain the experimental action scoring result.

[0038] Optionally, scoring the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result includes:

[0039] When the flexible scoring strategy is a combined scoring strategy, a correlation analysis is performed on the target experimental actions corresponding to the target action intervals to obtain a target action interval combination with correlation.

[0040] In a second aspect, the present application provides an intelligent scoring device for experimental video actions of nonlinear operations, comprising:

[0041] An experimental video data acquisition module is used to acquire the experimental video data to be predicted;

[0042] A multi-dimensional analysis module, configured to perform a multi-dimensional analysis on the experimental video data using a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data;

[0043] An action classification prediction module is used to perform action classification prediction through sliding window prediction based on the temporal relationship and the experimental video data, and obtain an initial action interval set containing a complete experimental action, confidence information, and action category;

[0044] an action merging module, configured to perform action merging by eliminating breakpoints on the initial action interval set based on the confidence information and the action category using the fine-tuning model, to obtain a target action interval with coherence corresponding to the experimental action;

[0045] The scoring module is used to score the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result.

[0046] In summary, the embodiment of the present application obtains the experimental video data to be predicted, performs multi-dimensional analysis on the experimental video data through a fine-tuning model, obtains the temporal relationship of the experimental actions in the experimental video data, and uses the temporal relationship to characterize the temporal sequence between the contexts of the experimental actions of the nonlinear operations in the experimental video data. Subsequently, based on the temporal relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain a set of initial action intervals with consistency, confidence information and action categories. Then, through the fine-tuning model, the initial action intervals are merged by eliminating breakpoints based on the confidence information and action categories to obtain a target action interval with consistency corresponding to the experimental action. According to the flexible scoring strategy, the target experimental action corresponding to the target action interval is scored to obtain the experimental action scoring result, thereby achieving accurate initial action interval positioning, being able to quickly find and focus on the video clip where the scoring point is located, and improving the accuracy and reliability of the scoring through a flexible scoring method. It can be seen that the present application realizes accurate motion analysis and initial motion interval positioning through innovative initial motion interval positioning and flexible scoring methods, fully takes into account the diversity and nonlinear characteristics of experimental operations, can quickly find and focus on the video clips where the scoring points are located, and uses more flexible and intelligent scoring methods to adapt to the uncertainty of experimental operations, accurately capture scoring points, and avoid scoring errors caused by algorithm limitations. That is, through flexible scoring methods, the accuracy, reliability, comprehensiveness and fairness of scoring are improved, and the problems of missed judgments and wrong judgments caused by the inability of existing technologies to be applicable to complex and changeable experimental operations are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A flowchart of an intelligent scoring method for experimental video actions of nonlinear operations provided in an embodiment of the present application;

[0050] Figure 2 This is a schematic flow chart of the steps of a method for intelligent scoring of experimental video actions of non-linear operations provided by an optional embodiment of the present application;

[0051] Figure 3 This is a predictive reasoning diagram provided by an optional example of this application;

[0052] Figure 4 This is a structural block diagram of an intelligent scoring device for experimental video actions of nonlinear operations provided in an embodiment of the present application;

[0053] Figure 5 This is a structural diagram of an electronic device provided as an optional example of the present application. DETAILED DESCRIPTION

[0054] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] There are two main techniques for locating scoring points in experimental videos: 1. Single-frame object detection methods, which analyze the interaction of detection frames to determine actions, have significant shortcomings in practical applications. Due to the lack of holistic analysis of continuous frame sequences, single-frame detection struggles to accurately capture the start and end moments of actions, especially for subtle or rapidly changing operations. This can lead to the system incorrectly demarcating the initial action interval and failing to precisely locate the positions of certain key scoring points, thus affecting scoring accuracy. 2. Using a pre-set, fixed-order scoring mechanism. For example, under standard conditions, an experiment typically contains approximately 10 scoring points that must be completed in a specific order. Using a rigid, fixed-order scoring mechanism, only experimental operations that conform to this specific order are scored. This rigid, sequential scoring method cannot effectively cope with the complexities of actual exams. Scoring terminates only after a preset number of scoring points have been identified, ignoring the possibility that students may repeat an operation after completing a scoring point or complete experimental steps in a non-standard order. This failure to account for repeated operations or sequence changes can easily lead to missed or incorrect judgments. It can be seen that the existing experimental video action scoring methods have great limitations and cannot accurately, comprehensively and fairly perform action scoring.

[0056] To solve the above problems, the present application provides an intelligent scoring method for experimental video actions of nonlinear operations. First, a fine-tuning model is used to perform multi-dimensional analysis on the experimental video to obtain the temporal relationship between the context of the experimental actions of the nonlinear operation, fully considering the problem of changes in the order of experimental actions. Then, based on the temporal relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain the initial action interval, confidence information and action category of the coherent experimental action. The initial action interval is merged by eliminating breakpoints based on the confidence information and action category through the fine-tuning model to obtain the coherent target action interval corresponding to the experimental action. Then, according to a flexible scoring strategy, the target experimental action corresponding to the target action interval is scored to obtain the experimental action scoring result. Therefore, this application fully takes into account the diversity and nonlinearity of experimental operations and other complex characteristics, realizes accurate motion analysis and initial motion interval positioning, and fully takes into account the diversity and nonlinearity of experimental operations. It can quickly find and focus on the video clips where the scoring points are located, and uses more flexible and intelligent scoring methods to adapt to the uncertainty of experimental operations, accurately capture scoring points, and avoid scoring errors caused by algorithm limitations. That is, through flexible scoring methods, the accuracy, reliability, comprehensiveness and fairness of scoring are improved, and it can handle various situations such as repeated operations and sequence adjustments that may occur in students during the experiment, and solve the problems of missed judgments and wrong judgments caused by the inability of existing technologies to apply to complex and changeable experimental operations.

[0057] To facilitate understanding of the embodiments of the present application, further explanation will be given below in conjunction with the drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present application.

[0058] The intelligent scoring method for experimental video actions of nonlinear operations provided in this application has broad application prospects in the fields of physics, chemistry and biology experimental teaching and examinations, and can effectively promote the innovation and development of physics, chemistry and biology experimental operation evaluation. Figure 1 This is a flow chart of a method for intelligent scoring of experimental video actions for nonlinear operations provided in an embodiment of the present application. Figure 1 As shown, the method for intelligent scoring of experimental video actions of nonlinear operations provided in the embodiment of the present application may specifically include the following steps:

[0059] Step 110: Acquire experimental video data to be predicted.

[0060] In this embodiment, the experimental video data refers to experimental videos, such as videos of physics, chemistry, and biology experiments, which are not limited in this embodiment. The experimental video data can include multiple experimental actions to be identified. For example, in a physics, chemistry, and biology experiment exam, the experimental video data can be obtained by filming a series of experimental operations performed by students.

[0061] Step 120 , performing a multi-dimensional analysis on the experimental video data using a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data.

[0062] The temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data.

[0063] In a specific implementation, taking into account the complex characteristics of the experimental operation such as diversity and nonlinearity, the experimental steps in the experimental video data are not necessarily completed in strict accordance with the established order, that is, the experimental actions in the experimental video data may be in accordance with the established order, and some experimental actions may not conform to the established order (i.e., non-standard order). To solve the problem of nonlinear operation, in this embodiment, the fine-tuning model can be integrated with a visual backbone model, and the experimental video data is input into the pre-trained fine-tuning model, so that the visual backbone model in the fine-tuning model is used to perform a multi-dimensional analysis of the experimental video data including the time dimension and the space dimension, and the context relationship between each experimental action is obtained as the temporal relationship of the context. The temporal relationship of the context can be used to clarify the context relationship between the linear operation and the nonlinear operation in each experimental action, and the temporal relationship of the context of the video action can also be deeply understood. Compared with the prior art, the model adopted in this embodiment does not need to rely on additional optical flow extraction or other preprocessing steps, thereby achieving the purpose of simplifying the processing flow and improving efficiency.

[0064] Therefore, this embodiment performs multi-dimensional analysis on the experimental actions in the experimental video data. For experimental actions that conform to a predetermined sequence, experimental actions of nonlinear operations, and possible repetitive experimental actions in the experimental video data, the sequential relationship between each experimental action can be accurately analyzed, thereby facilitating subsequent classification prediction and scoring.

[0065] Step 130 : Based on the temporal relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain an initial action interval set containing complete experimental actions, confidence information, and action categories.

[0066] The initial action interval set includes at least one initial action interval related to the experimental action.

[0067] In this embodiment, after obtaining the temporal relationship of the experimental actions, the contextual relationship of each experimental action in the experimental video data can be fully understood. This embodiment uses a sliding window prediction method to first divide the experimental video data into video frames according to a certain number of video frames or according to the input requirements of the model, so as to divide the experimental video data into multiple sliding windows (simply referred to as sliding windows). Subsequently, action classification recognition / prediction is performed on the multiple sliding windows to obtain the action category of each experimental action, and for each action category, the sliding window is subjected to action prediction to obtain the initial action interval set of each experimental action.

[0068] In actual implementation, the initial action interval set of the experimental action with consistency can include video frames including the starting video frame (referred to as the starting frame) to the ending video frame (referred to as the ending frame) of the experimental action. Therefore, the initial action interval corresponding to the experimental action in this embodiment is the sliding window containing the action category. For the sliding window predicted by the model to contain the same action category, it can be used as the initial action interval. The initial action interval can contain video frames of the same action category. By aggregating all initial action intervals of the same action category, the initial action interval set of the experimental action with consistency is obtained, that is, all video frames of the complete target action interval corresponding to the action category.

[0069] In a specific implementation, for the same experimental action, if the experimental action only belongs to one sliding window, the sliding window can be used as the initial action interval. If the experimental action belongs to multiple sliding windows, the initial action interval set can include multiple initial action intervals. This embodiment thus realizes the identification of coherent experimental actions from multiple sliding windows. Confidence prediction is then performed on each initial action interval to obtain the confidence level that the initial action interval belongs to the action category. This confidence level is used as confidence information for subsequent breakpoint elimination processing.

[0070] Step 140 : Using the fine-tuning model, based on the confidence information and the action category, the initial action interval set is subjected to action merging by eliminating breakpoints, so as to obtain a target action interval with coherence corresponding to the experimental action.

[0071] In this embodiment, the target action interval with continuity can be understood as a video containing a complete experimental action, or as a set of multiple video frames including a start frame to an end frame of the experimental action.

[0072] In a specific implementation, the embodiment of the present application performs breakpoint elimination processing on the initial action interval set corresponding to each experimental action of the same action category in turn through a fine-tuning model, and determines whether to retain the initial action interval and merge it by judging whether the confidence corresponding to each initial action interval in the initial action interval set is greater than a preset confidence threshold. Specifically, the embodiment can process each initial action interval in the initial action interval set in turn or in parallel, and filter the initial action interval when the confidence corresponding to the initial action interval is less than a preset confidence threshold; retain the initial action interval when the confidence corresponding to the initial action interval is not less than the preset confidence threshold, and merge all the retained initial action intervals until each initial action interval in the initial action interval set is judged, thereby achieving breakpoint elimination and obtaining a target action interval with continuity corresponding to the experimental action. Therefore, this embodiment first ensures that the confidence levels of all initial action intervals are higher than or equal to a threshold, and then merges these high-confidence initial action intervals, thereby avoiding merging low-confidence prediction results into the final initial action interval. This helps to improve the accuracy and reliability of action recognition, and can accurately identify and divide the specific time interval of each scoring point in the video, so that the scoring points can be quickly located in subsequent processing, and then accurate action analysis can be performed.

[0073] In actual processing, for each experimental action in the experimental video data, this embodiment can perform sliding window prediction, breakpoint elimination and other processing on the experimental actions sequentially / in parallel, so as to obtain the target action interval corresponding to each experimental action in the experimental video data. Subsequently, each target action interval can be action-scored to obtain the final action scoring result of the experimental video data, that is, execute step 150.

[0074] Step 150 : According to a preset flexible scoring strategy, an experimental action scoring is performed based on each target action interval corresponding to the experimental video data to obtain an experimental action scoring result of the experimental video data.

[0075] In this embodiment, the flexible scoring strategy may include but is not limited to: parallel scoring, nonlinear scoring, and combined scoring. Among them, parallel scoring refers to judging all identified scoring point actions at the same time, so that it is not constrained by the preset order and can quickly score the target experimental actions corresponding to the target action interval; nonlinear scoring can be applied to nonlinear operations that students may perform to redo or jump to complete the scoring points. The nonlinear scoring method can be used to score experimental actions such as repeated operations or jump operations. It can respond to the complex situations in actual experimental operation exams and solve the problems of system omissions and misjudgments of additional valid operations caused by the use of a preset fixed order scoring mechanism in existing AI scoring algorithms, thereby improving the comprehensiveness and fairness of scoring; combined scoring can be applied to the situation where multiple target action intervals are combined for scoring. As needed, multiple related target action intervals can be combined for overall evaluation. Thus, the embodiment of the present application optimizes the scoring process and improves the scoring efficiency through a flexible scoring strategy, making the scoring of experimental exam videos more efficient and accurate. When teachers need to review, they can also quickly find and focus on the video clips where the scoring points are located.

[0076] In summary, the embodiment of the present application obtains the experimental video data to be predicted, performs multi-dimensional analysis on the experimental video data through a fine-tuning model, obtains the temporal relationship of the experimental actions in the experimental video data, and uses the temporal relationship to characterize the temporal sequence between the contexts of the experimental actions of the nonlinear operations in the experimental video data. Subsequently, based on the temporal relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain a set of initial action intervals with consistency, confidence information and action categories. Then, through the fine-tuning model, the initial action intervals are merged by eliminating breakpoints based on the confidence information and action categories to obtain a target action interval with consistency corresponding to the experimental action. According to the flexible scoring strategy, the target experimental action corresponding to the target action interval is scored to obtain the experimental action scoring result, thereby achieving accurate initial action interval positioning, being able to quickly find and focus on the video clip where the scoring point is located, and improving the accuracy and reliability of the scoring through a flexible scoring method. It can be seen that the present application realizes accurate motion analysis and initial motion interval positioning through innovative initial motion interval positioning and flexible scoring methods, fully takes into account the diversity and nonlinear characteristics of experimental operations, can quickly find and focus on the video clips where the scoring points are located, and uses more flexible and intelligent scoring methods to adapt to the uncertainty of experimental operations, accurately capture scoring points, and avoid scoring errors caused by algorithm limitations. That is, through flexible scoring methods, the accuracy, reliability, comprehensiveness and fairness of scoring are improved, and the problems of missed judgments and wrong judgments caused by the inability of existing technologies to be applicable to complex and changeable experimental operations are solved.

[0077] Reference Figure 2 , shows a schematic flow chart of a method for intelligently scoring nonlinear experimental video actions provided by an optional embodiment of the present application. The method may specifically include the following steps:

[0078] Step 210: Acquire experimental video data to be predicted.

[0079] For the description of step 210 , reference may be made to the above-mentioned step 110 , which will not be described in detail in this embodiment.

[0080] Step 220 , performing a multi-dimensional analysis on the experimental video data using a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data.

[0081] The temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data.

[0082] In actual implementation, this embodiment adopts an advanced pre-trained visual backbone model. For example, the visual backbone model may include but is not limited to: VideoMamba model. Before using the visual backbone model to perform multi-dimensional analysis on the experimental video data, the visual backbone model can be trained first. For example, the visual backbone model can be trained using large-scale video datasets such as Kinetics-400, so that the visual backbone model can have the ability to perform in-depth analysis of video content. After completing the model training, the visual backbone model can be integrated into the fine-tuned video classification model M, referred to as the fine-tuning model.

[0083] Exemplarily, model training for a fine-tuning model may mainly include but is not limited to: sample data preprocessing, model fine-tuning and other processes.

[0084] Regarding sample data: Take the high school chemistry experiment "Production and Testing of Carbon Dioxide" as an example. This experiment has been tested over many years, accumulating a large amount of experimental video data. This data was recorded from a unified exam room perspective and covers various equipment, lighting, and desktop backgrounds, providing sufficient environmental diversity. First, according to exam requirements, this experiment can generally be divided into eight specific scoring points: ① Connecting the catheter; ② Checking for airtightness; ③ Adding marbles to the test tube; ④ Pouring dilute hydrochloric acid; ⑤ Securing the test tube on the iron stand; ⑥ Collecting gases; ⑦ Testing for carbon dioxide; ⑧ Arranging equipment and cleaning the desktop. The specific number of scoring points and their divisions can be adjusted as needed. The video samples are marked with scoring point intervals. Specifically, using data annotation tools, the start and end frames of the scoring point actions are recorded and labeled with categories. This then forms a large-scale dataset, completing the sample data preprocessing.

[0085] For model fine-tuning: This example uses the VideoMamba model for fine-tuning (it should be noted that VideoMamba is only one of the available models, and other video backbone models can be selected as replacements). This model uses the cross-entropy loss function and optimization methods such as gradient descent to fit the model's predicted results with the actual category results, ultimately obtaining a fine-tuned model M.

[0086] In actual processing, this embodiment inputs the experimental video data into the fine-tuning model. The model can directly receive video clips such as experimental video data as input data, and conduct a comprehensive analysis of its spatial and temporal dimensions, so as to deeply understand the temporal relationship between the context of the video action. Compared with the prior art, the model adopted in this application does not need to rely on additional optical flow extraction or other preprocessing steps, which simplifies the processing flow and improves efficiency. By fine-tuning the model using pre-labeled physical, chemical and biological experimental action data, a model M is obtained as a fine-tuning model, which ensures that the model can accurately capture the key actions and scoring points in the experimental video to meet the needs of specific application scenarios.

[0087] In an optional embodiment, the embodiment of the present application performs a multi-dimensional analysis on the experimental video data through a preset fine-tuning model to obtain the temporal relationship of the experimental actions in the experimental video data, which may specifically include: inputting the experimental video data into the fine-tuning model; performing spatial dimension analysis and temporal dimension analysis on the experimental video data through the fine-tuning model to obtain the temporal relationship between the context of the experimental actions in the experimental video data.

[0088] Step 230 : Using a sliding window technique, scan the video content of the experimental video data according to a preset scanning method to obtain a sliding window set.

[0089] In this embodiment, the sliding window set may include one or more sliding windows. By adopting the sliding window technology, the fine-tuning model can scan the experimental video data frame by frame or the video content according to a set time window, thereby achieving continuous action recognition.

[0090] In actual implementation, this embodiment can use sliding window technology to divide the experimental video data into windows according to the input requirements of the fine-tuning model to obtain multiple sliding windows; of course, the window division can also be performed according to usage requirements. For example, for shorter experimental video data, only one sliding window can be set to improve the speed of action classification and recognition; for longer experimental video data, the division rules can be preset as needed, such as setting the width and step size of the sliding window, and dividing the video frames in the experimental video data according to the width and step size to obtain multiple sliding windows as a sliding window set.

[0091] For example, a set of all frame sequences contained in a certain experimental video is F, then F = f1, f2, ..., f N , where N is the total number of all video frames of the experimental video, f1 represents the first video frame of the experimental video, and f N Represents the last video frame of the experimental video. Set the width of the sliding window W to w and the step size to s. Then the j-th sliding window can be expressed as:

[0092] W j =f (j-1)s+1 , f (j-1)s+2 ,...,f js

[0093] Among them, j ranges from 1 to

[0094] Step 240 , based on the temporal relationship, using a sliding window prediction method, the fine-tuning model is used to predict the experimental actions in the sliding window set to obtain at least one initial action interval related to the experimental action and an action category corresponding to the initial action interval.

[0095] In this embodiment, the action category is generally related to the video data. For example, for an experimental video, the action category is the action related to the experiment. For example, the action category related to the experiment can refer to the above-mentioned example of "production and testing of carbon dioxide." It should be noted that the above-mentioned "production and testing of carbon dioxide" is only an example of an experiment and does not constitute a specific limitation on the action category.

[0096] In a specific implementation, the fine-tuned video classification model M (also known as the fine-tuning model) has the ability to perform in-depth analysis of video content. By obtaining the input experimental video data, the experimental actions in the experimental video data are classified and identified, and the action category corresponding to each experimental action in the experimental video data is obtained. Subsequently, the initial action interval of the experimental action can be predicted for the action category through a sliding window. Specifically, the fine-tuning model can respectively identify the sliding window where the starting frame of the experimental action is located and the sliding window where the ending frame is located by predicting the experimental action in the sliding window set. For experimental actions of the same category, if the experimental action involves multiple sliding windows, such as the starting frame and the ending frame of a certain experimental action belong to different sliding windows, then it is possible to first determine which sliding windows the starting frame to the ending frame of the experimental action involves, and then process the sliding windows involved until the sliding window corresponding to the ending frame is processed, thereby obtaining multiple initial action intervals containing the complete experimental action; if the experimental action only involves a single sliding window, the complete video frame of the experimental action can be directly identified to obtain a single initial action interval.

[0097] Optionally, the above-mentioned prediction processing of the experimental action in the sliding window set by the fine-tuning model to obtain at least one initial action interval related to the experimental action may specifically include: performing overlapping prediction on the sliding window set by the fine-tuning model to obtain window dispersion information of the experimental action in the sliding window set, wherein the window dispersion information includes non-overlapping window information and overlapping window information; when the window dispersion information is non-overlapping window information, directly extracting the sliding window containing the experimental action from the sliding window set to obtain at least one initial action interval related to the experimental action; when the window dispersion information is overlapping window information, creating an overlapping sliding window set based on the sliding window set; performing action prediction based on the overlapping sliding window set to obtain at least three initial action intervals related to the experimental action.

[0098] In actual implementation, there are three situations when the fine-tuning model predicts the same experimental action: ① The experimental action only involves a single sliding window; ② The experimental action spans two adjacent sliding windows, that is, the start frame and end frame of the experimental action are respectively located in two adjacent sliding windows; ③ The experimental action spans three or more adjacent sliding windows. For example, a certain experimental action is scattered across three adjacent sliding windows (such as sliding window A, sliding window B, and sliding window C), with the start frame and some frames after the start frame located in sliding window A, some intermediate frames located in sliding window B, and the end frame and some frames before the end frame located in sliding window C. For situations involving a single sliding window or two adjacent sliding windows, the overlapping prediction is performed based on non-overlapping window information, and there is no need to re-divide the sliding windows. The sliding window involving the experimental action can be directly used as the initial action interval. For situations involving three or more adjacent sliding windows, the overlapping prediction is performed based on overlapping window information. This embodiment can re-divide the sliding window set to create an overlapping sliding window set. By performing action prediction on the overlapping sliding window set, at least three initial action intervals related to the experimental action are obtained.

[0099] Step 250 : For the action category, confidence prediction is performed on the initial action interval using the fine-tuning model to obtain confidence information of the initial action interval.

[0100] The confidence information is used to characterize the correlation between the initial action interval and the action category.

[0101] In this embodiment, when the fine-tuning model predicts the action category for the initial action interval, it can perform confidence prediction for the predicted action category. Through the confidence prediction, all initial action intervals of the same action category can be merged / filtered later.

[0102] As an example, for each sliding window Wj , model M predicts an action category l j and confidence Conf(W j ):

[0103] M:W j →(l j , Conf(W j )

[0104] Where, C=(l1,[s1,e1],(l2,[s2,e2],...,(l k ,[s k , e k ]) are all action intervals and their categories predicted by the model, where s i and e i are the start and end frames of the i-th action, respectively, and k represents the total number of predicted intervals.

[0105] For example, for the experimental video to be predicted, taking a 100-frame video as an example (it should be noted that the number of video frames actually used can be more, and this example does not limit the specific total number of video frames), it is assumed that the fine-tuning model accepts a 16-frame sequence input.

[0106] For the case of spanning two adjacent sliding windows (non-overlapping windows): When performing sliding window prediction, a sliding window with a width of 16 frames and a step size of s = 16 is used. The sliding windows are divided into the following categories: the first sliding window: frames 1-16; the second sliding window: frames 17-32; the third sliding window: frames 33-48, and so on. Assume that a certain experimental action, the action category is connecting a catheter, and this experimental action starts from frame 11 and ends at frame 25, that is, the starting frame is 11 and the ending frame is 25. Based on the above sliding window division, the experimental action of connecting the catheter will be dispersed into two sliding windows, where the first sliding window (frames 1-16) contains frames 11-16 of the experimental action of connecting the catheter; the second sliding window (frames 17-32) contains frames 17-25 of the experimental action of connecting the catheter. After performing sliding window predictions using the model, we can use these two sliding windows as the two initial action intervals, with the first initial action interval encompassing frames 11-16 of the experimental action, and the second initial action interval encompassing frames 11-16 of the experimental action. Assume that the model predicts each initial action interval and obtains the action category and confidence level. Initial action interval 1 (frames 1-16): predicted as "connect catheter" with a confidence level of 0.75; initial action interval 2 (frames 17-32): predicted as "connect catheter" with a confidence level of 0.90. Subsequently, based on the preset confidence threshold and the confidence levels corresponding to the initial action intervals, merge and filter the predictions.

[0107] For the case of spanning three or more adjacent sliding windows (overlapping windows): Using a 16-frame window width and assuming a step size s = 8, we create overlapping sliding windows, resulting in the following window partitioning: Window 1: frames 1-16; Window 2: frames 9-24; Window 3: frames 17-32; Window 4: frames 25-40, and so on. For an experimental action starting in frame 11 and ending in frame 25, this action will be included in three overlapping sliding windows. That is, the first sliding window (frames 1-16) includes frames 11-16 of the action, the second sliding window (frames 9-24) is completely contained within the action interval, and the third sliding window (frames 17-32) includes frames 17-25 of the action. All three sliding windows serve as the initial action interval for the experimental action. The model predicts each sliding window to obtain the action category and confidence score. For example: Initial action interval 1 (1-16 frames): predicted as "connect the catheter", confidence level 0.69; initial action interval 2 (9-24 frames): predicted as "connect the catheter", confidence level 0.95; initial action interval 3 (17-32 frames): predicted as "connect the catheter", confidence level 0.85. Subsequently, the preset confidence threshold can be used in combination with the confidence level corresponding to the initial action interval for merge filtering.

[0108] Step 260: Obtain a preset confidence threshold.

[0109] Step 270 , for the confidence information corresponding to each initial action interval, when the confidence information is not lower than the confidence threshold, the corresponding initial action interval is merged by eliminating breakpoints to obtain a coherent target action interval corresponding to the experimental action.

[0110] Step 280, when the confidence information is lower than the confidence threshold, the initial action interval corresponding to the confidence information lower than the confidence threshold is filtered through a preset post-processing module, and the initial action interval corresponding to the confidence information not lower than the confidence threshold is merged to eliminate breakpoints, so as to obtain a target action interval with continuity corresponding to the experimental action.

[0111] A unified description of steps 250 to 280 is provided:

[0112] In the specific implementation, in order to overcome the limitations of the model in processing long sequences, this application introduces an efficient post-processing module, which is used to merge the continuous action categories predicted by the sliding window, that is, to merge the initial action intervals of the experimental action to eliminate the potential breakpoints caused by window division and ensure the continuity of the action. By accurately locating the experimental action in the experimental video and accurately dividing the action intervals, the accuracy of subsequent scoring is improved.

[0113] In this embodiment, the post-processing module can identify and exclude prediction results with low confidence, thereby further improving the accuracy and reliability of classification. Specifically, this embodiment can set a confidence threshold. For each initial action interval predicted by the sliding window as the same action, the fine-tuning model can predict the confidence corresponding to the initial action interval, compare the confidence threshold with the confidence corresponding to the initial action interval, and use the post-processing module to filter the initial action intervals with confidence lower than the confidence threshold, and merge the initial action intervals with confidence not lower than the confidence threshold, so as to obtain a complete and coherent target action interval of the same action category.

[0114] For example, assume that the confidence threshold is set to 0.7. Combined with the above example, for the case of spanning two adjacent sliding windows, it can be seen from the above example that the confidence of the initial action interval 1 and the initial action interval 2 of the "connect the catheter" action are both greater than the confidence threshold 0.7, so the prediction results of the initial action interval 1 and the initial action interval 2 will be retained. The post-processing module will merge the overlapping initial action intervals predicted to be the same action. In this example, the initial action interval 1 and the initial action interval 2 are both predicted to be the "connect the catheter" action, and they are adjacent, so the merged target action interval is: (1, 32). After merging, the system will identify a continuous "connect the catheter" action, starting from the 1st frame and ending at the 32nd frame. Although the range is larger than the actual action (11-25 frames), the entire action process is successfully captured.

[0115] For the case of spanning three or more adjacent sliding windows, as can be seen from the above example, the confidence of initial action interval 1 of the "connect the catheter" action is less than the confidence threshold of 0.7, and initial action interval 1 will be filtered. However, the confidence of initial action intervals 2 and 3 are both greater than the confidence threshold of 0.7, so the prediction results of initial action intervals 2 and 3 are retained. In this example, initial action intervals 2 and 3 both predict "connect the catheter" and they overlap. When merging action intervals, the post-processing module merges the overlapping sliding windows predicted to be the same action. The target action interval obtained after merging is: (9, 32). Finally, the system will identify a continuous "connect the catheter" action, starting at frame 9 and ending at frame 32. Through this merging method, even if the action spans multiple sliding windows, the post-processing module can still identify and merge it into a continuous action, solving the problem of the same action being scattered across different sliding windows.

[0116] It should be noted that the window size, step size, and whether the window is divided into overlapping parts in this example can be adjusted as needed.

[0117] As an example, see Figure 3 As shown in the figure, an input test video is input into the fine-tuned model M through a sliding window method, and the frame sequence w (the window length is the sequence length that the model can accept) is input to a classification result and confidence. After completing the prediction of all frames, a set of classification results and corresponding confidence levels will be obtained. After passing through a post-processing model, the intervals with low confidence levels are merged and filtered, and then this filtered set is scored through a flexible scoring strategy module to obtain the final score result.

[0118] Optionally, in the embodiment of the present application, the initial action interval corresponding to the confidence information below the confidence threshold is filtered by a preset post-processing module, which may specifically include: according to formula C filtered =Filter(C,θ)=(l,[s,e])∈C|Conf([l,s,e])≥θ, the initial action intervals corresponding to the confidence information lower than the confidence threshold are filtered to obtain the initial action intervals whose confidence information is not lower than the confidence threshold; according to the formula Merge(C filtered , δ) = (l, [s′, e′]), l∈L, s′=min i∈I (s i ), e′=max i∈I (e i ), the initial action interval C after filtering filtered Merge and use the time interval threshold δ to perform the merging operation; where θ is the preset confidence threshold, C filtered is the initial action interval after filtering, l is the experimental action, s is the starting frame corresponding to the experimental action, e is the ending frame corresponding to the experimental action, (l, [S, e]) is the initial action interval of the experimental action l, I satisfies l i = l, and for index i and j, the initial action interval [s i , e i ] and [s j , e j ]Close in time.

[0119] Step 290 : Scoring the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result.

[0120] In actual implementation, this embodiment can construct a scoring point analysis module, which can be composed of target detection and hand information interaction logic. The model can call the scoring point analysis module to perform accurate right and wrong analysis on the specific scoring interval, thereby assigning points to the corresponding scoring actions of the examinee, so that the judgment of the score of the experimental video will no longer be limited to the preset judgment order, and the re-completed actions can also be taken into account. Therefore, this application optimizes the scoring process, making it more efficient and accurate, and improving the efficiency of the scoring process. When the user (such as a teacher in an experimental exam) needs to review, he can also quickly find and focus on the video clip where the scoring point is located, and this embodiment can also significantly identify the repeated operations performed by students on a certain scoring point during the experiment, which can adapt to the uncertainty of the student's operation, accurately capture the scoring point, and avoid the scoring error caused by the algorithm limitation, thereby further improving the accuracy and reliability of the scoring.

[0121] In an optional embodiment, the embodiment of the present application scores the target experimental action corresponding to each of the target action intervals according to a preset flexible scoring strategy to obtain an experimental action scoring result, which may specifically include: when the flexible scoring strategy is a parallel scoring strategy, scoring and identifying all experimental actions corresponding to the target action interval according to the parallel scoring strategy to obtain a scored action point corresponding to each experimental action; simultaneously, all scored action points are evaluated in parallel to obtain an action score corresponding to each scored action point; and the experimental action scoring result is determined based on the action score.

[0122] In this embodiment, based on all pre-identified action scoring intervals, the system can simultaneously evaluate multiple scoring points. The scoring process is not constrained by a preset order; the evaluation logic for all scoring points can be executed in parallel, thus achieving parallel scoring. For example: a. The system simultaneously evaluates all identified action intervals, such as "connecting a catheter," "adding a marble to a test tube," and "pouring dilute hydrochloric acid." b. The scoring logic for each action runs independently, without waiting for the scoring of other actions to complete. This approach improves scoring efficiency, especially when processing large amounts of video data.

[0123] In an optional embodiment, the embodiment of the present application scores the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result, which may specifically include: when the flexible scoring strategy is a nonlinear scoring strategy, identifying the nonlinear operations in the target action interval; identifying scoring points for the nonlinear operations to obtain deduplicated scoring points and / or jump scoring points, the deduplicated scoring points being scoring points corresponding to repetitive nonlinear operations, and the jump scoring points being scoring points corresponding to jumpy nonlinear operations; performing scoring judgment based on the deduplicated scoring points and / or jump scoring points to obtain the experimental action scoring result.

[0124] In this embodiment, nonlinear scoring allows the system to adapt to students' potential redoes or non-standard ordering of operations while completing the experiment, while still adhering to the basic logical order of the experiment. This system also correctly handles situations where students may attempt the same action multiple times. For example: a. A student may perform an "airtightness check" immediately after "connecting the tube" and then "adding marbles to the test tube"; b. A student may attempt "connecting the tube" multiple times, but the system will still identify the attempt that ultimately succeeds; c. "Adding dilute hydrochloric acid to the test tube" and "Carbon dioxide test" are separated by other actions, but "adding clear lime water to the test tube" is a sub-step of "Carbon dioxide test." When a student performs this action, they may perform "adding clear lime water to the test tube" after "adding dilute hydrochloric acid to the test tube." The system will still correctly evaluate this non-standard but reasonable order of operations.

[0125] In an optional embodiment, the embodiment of the present application scores the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result, which may specifically include: when the flexible scoring strategy is a combination scoring strategy, performing a correlation analysis on the target experimental actions corresponding to the target action interval to obtain a target action interval combination with correlation; performing an overall evaluation on the target action interval combination to obtain an experimental action scoring result.

[0126] In this embodiment, combined scoring refers to combining multiple temporally close and logically related action intervals for a holistic assessment. For example, a. "Connecting the catheter" and "Airtightness Check" could be considered a combined action, with the system checking whether the catheter was correctly connected and subsequently checked for airtightness. b. "Collecting Gas" and "Carbon Dioxide Test" could be combined for evaluation, with the system checking the correct order and execution quality of these two actions.

[0127] In summary, the embodiment of the present application obtains experimental video data to be predicted, and uses a fine-tuning model to perform multi-dimensional analysis on the experimental video data to obtain a temporal relationship between the context of each experimental action in the experimental video data, and uses a sliding window technology to scan the experimental video data according to a preset scanning method to obtain a sliding window set. Then, based on the temporal relationship, the sliding window prediction method is used to predict the experimental actions in the sliding window set through the fine-tuning model to obtain at least one initial action interval related to the experimental action and the action category corresponding to the initial action interval. Then, based on the action category, the initial action interval is predicted by the fine-tuning model to obtain confidence information of the initial action interval. Based on the confidence information corresponding to the initial action interval, the initial action intervals whose confidence information is lower than a preset confidence threshold are filtered out through the post-processing module, and the initial action intervals whose confidence information is not lower than the confidence threshold are merged by eliminating breakpoints, thereby obtaining a target action interval with coherence. According to the preset flexible scoring strategy, the target experimental action corresponding to each target action interval is scored to obtain the experimental action scoring result. As can be seen, this application achieves accurate frame-level positioning of key actions in experimental videos through advanced sliding window prediction and post-processing optimization technology. It not only overcomes the limitations of existing technologies, but also brings a variety of breakthrough improvements, including but not limited to: ① Accurate action interval identification. Through sliding window prediction and post-processing optimization, the start and end frame numbers of each experimental action can be accurately determined. The precise positioning provides a clear and accurate time interval basis for subsequent scoring; ② Flexible scoring strategy. Breaking through the limitations of traditional sequential scoring, it can support a variety of flexible scoring methods for subsequent scoring points; ③ Adaptable scoring mechanism. It can handle various situations during the student's experiment, such as repeated operations, sequence adjustment, etc. It can support dynamic adjustment of scoring standards and easily cope with changes in different experimental requirements or scoring rules; ④ Efficient review support. When the scoring results need to be reviewed, teachers can directly access the specific action interval, which significantly improves the efficiency and accuracy of the review work, especially when processing a large number of test videos; ⑤ Comprehensive scoring coverage. It can ensure that all key operations are accurately captured and evaluated, avoiding omissions that may occur in traditional methods.

[0128] The embodiments of this application bring unprecedented accuracy and flexibility to experimental exam scoring, opening up new possibilities in the field of educational assessment and providing more comprehensive and fairer scoring results. Through this innovative action interval positioning and flexible scoring method, this application not only improves the accuracy and comprehensiveness of scoring, but also greatly enhances the adaptability and efficiency of the system when handling complex experimental scenarios, effectively solving various problems existing in the existing technology.

[0129] It should be noted that, for the purpose of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously.

[0130] like Figure 4 As shown, the embodiment of the present application further provides an intelligent scoring device 400 for experimental video actions of nonlinear operation, comprising:

[0131] The experimental video data acquisition module 410 is used to acquire the experimental video data to be predicted;

[0132] A multi-dimensional analysis module 420 is configured to perform a multi-dimensional analysis on the experimental video data using a preset fine-tuning model to obtain a temporal relationship between the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data;

[0133] An action classification prediction module 430 is configured to perform action classification prediction through sliding window prediction based on the temporal relationship and the experimental video data, and obtain an initial action interval set containing a complete experimental action, confidence information, and an action category, wherein the initial action interval set includes at least one initial action interval related to the experimental action;

[0134] An action merging module 440 is configured to perform action merging by eliminating breakpoints on the initial action interval set based on the confidence information and the action category using the fine-tuning model to obtain a coherent target action interval corresponding to the experimental action;

[0135] The scoring module 450 is configured to score the experimental action based on each target action interval corresponding to the experimental video data according to a preset flexible scoring strategy, and obtain an experimental action scoring result of the experimental video data.

[0136] Optionally, the multi-dimensional analysis module 420 is specifically used to input the experimental video data into the fine-tuning model; perform spatial dimension analysis and temporal dimension analysis on the experimental video data through the fine-tuning model to obtain the temporal relationship between the context of the experimental actions in the experimental video data.

[0137] Optionally, the action classification prediction module 430 includes:

[0138] The video content scanning submodule is used to use a sliding window technology to scan the video content of the experimental video data according to a preset scanning method to obtain a sliding window set;

[0139] A prediction processing submodule is configured to predict the experimental actions within the sliding window set using the fine-tuning model based on the temporal relationship and a sliding window prediction method, thereby obtaining at least one initial action interval related to the experimental action and an action category corresponding to the initial action interval;

[0140] The confidence prediction submodule is used to perform confidence prediction on the initial action interval according to the action category through the fine-tuning model to obtain confidence information of the initial action interval; wherein the confidence information is used to characterize the correlation between the initial action interval and the action category.

[0141] Optionally, the prediction processing submodule includes:

[0142] an overlapping prediction unit, configured to perform overlapping prediction on the sliding window set using the fine-tuning model to obtain window dispersion information of the experimental action within the sliding window set, wherein the window dispersion information includes non-overlapping window information and overlapping window information;

[0143] a first initial action interval determining unit, configured to, when the window scattered information is non-overlapping window information, directly extract a sliding window containing the experimental action from the sliding window set to obtain at least one initial action interval related to the experimental action;

[0144] The second initial action interval determination unit is used to create an overlapping sliding window set based on the sliding window set when the window dispersion information is overlapping window information; perform action prediction based on the overlapping sliding window set to obtain at least three initial action intervals related to the experimental action.

[0145] Optionally, the action merging module 440 includes:

[0146] A confidence threshold acquisition submodule is used to obtain a preset confidence threshold;

[0147] The action merging submodule is used to, for the confidence information corresponding to each initial action interval, when the confidence information is not lower than the confidence threshold, perform action merging with the corresponding initial action intervals eliminating breakpoints to obtain a target action interval with continuity corresponding to the experimental action; when the confidence information is lower than the confidence threshold, perform interval filtering on the initial action intervals corresponding to the confidence information below the confidence threshold through a preset post-processing module, and perform action merging with the initial action intervals corresponding to the confidence information not lower than the confidence threshold to obtain a target action interval with continuity corresponding to the experimental action.

[0148] Optionally, the action merging submodule is specifically configured to:filtered =Filter(C,θ)=(l,[s,e])∈C|Conf([l,s,e])≥θ, the initial action intervals corresponding to the confidence information below the confidence threshold are filtered to obtain the initial action intervals whose confidence information is not lower than the confidence threshold; according to the formula Merge(C filtered , δ) = (l, [s′, e′]), l∈L, s′=min i∈I (s i ), e′=max i∈I (e i ), the initial action interval C after filtering filtered Merge and use the time interval threshold δ to perform the merging operation; where θ is the preset confidence threshold, C filtered is the initial action interval after filtering, l is the experimental action, s is the starting frame corresponding to the experimental action, e is the ending frame corresponding to the experimental action, (l, [S, e]) is the initial action interval of the experimental action l, I satisfies l i = l, and for index i and j, the initial action interval [s i , e i ] and [s j , e j ]Close in time.

[0149] Optionally, the scoring module 450 includes:

[0150] A parallel scoring submodule is configured to, when the flexible scoring strategy is a parallel scoring strategy, score and identify the target experimental action corresponding to each target action interval according to the parallel scoring strategy to obtain a scored action point corresponding to each experimental action; simultaneously, perform parallel evaluation on all scored action points to obtain an action score corresponding to each scored action point; and determine an experimental action scoring result of the experimental video data according to the action score;

[0151] The nonlinear scoring submodule is used to identify the nonlinear operations in the target action interval when the flexible scoring strategy is a nonlinear scoring strategy; identify scoring points for the nonlinear operations to obtain duplicate scoring points and / or jump scoring points, where the duplicate scoring points are scoring points corresponding to repetitive nonlinear operations and the jump scoring points are scoring points corresponding to jumpy nonlinear operations; and perform scoring evaluation based on the duplicate scoring points and / or jump scoring points to obtain an experimental action scoring result.

[0152] The combined scoring submodule is used to perform a correlation analysis on the target experimental actions corresponding to the target action intervals when the flexible scoring strategy is a combined scoring strategy, to obtain a target action interval combination with correlation; and to perform an overall evaluation on the target action interval combination to obtain an experimental action scoring result.

[0153] It should be noted that the intelligent scoring device for experimental video actions of nonlinear operations provided in the embodiments of the present application can execute the intelligent scoring method for experimental video actions of nonlinear operations provided in any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.

[0154] In a specific implementation, the above-mentioned nonlinear operation experimental video action intelligent scoring device can be integrated into a device, so that the device can accurately classify and predict the experimental actions in the experimental video data in combination with the post-processing module by means of sliding window prediction, obtain the target action interval with coherence corresponding to the experimental action, and then score the target action interval according to the flexible scoring strategy, so as to achieve accurate initial action interval positioning, quickly find and focus on the video clip where the scoring point is located, and improve the accuracy and reliability of the scoring as an electronic device. The electronic device can be composed of two or more physical entities, or it can be composed of one physical entity, such as the electronic device can be a personal computer (PC), a computer, a server, etc., and the embodiment of the present application does not make specific restrictions on this.

[0155] like Figure 5As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114; the memory 113 is used to store computer programs; the processor 111 is used to implement the steps of the experimental video action intelligent scoring method for non-linear operations provided by any of the aforementioned method embodiments when executing the program stored in the memory 113. Exemplarily, the steps of the intelligent scoring method for experimental video actions of nonlinear operations may include the following steps: obtaining experimental video data to be predicted; performing multi-dimensional analysis on the experimental video data through a preset fine-tuning model to obtain the temporal relationship of the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data; performing action classification prediction through sliding window prediction based on the temporal relationship and the experimental video data to obtain an initial action interval set containing complete experimental actions, confidence information and action categories, wherein the initial action interval set contains at least one initial action interval related to the experimental action; performing action merging with breakpoint elimination on the initial action interval set based on the confidence information and the action category through the fine-tuning model to obtain a target action interval with continuity corresponding to the experimental action; performing experimental action scoring based on each target action interval corresponding to the experimental video data according to a preset flexible scoring strategy to obtain an experimental action scoring result of the experimental video data.

[0156] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for intelligent scoring of experimental video actions for nonlinear operations provided in any of the aforementioned method embodiments are implemented.

[0157] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0158] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A nonlinear operation experimental video action intelligent scoring method, characterized by: include: Obtain experimental video data to be predicted; The experimental video data is subjected to a multi-dimensional analysis using a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data. The fine-tuning model is a video classification model integrated with a visual backbone model. Based on the temporal relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain an initial action interval set containing the complete experimental action, confidence information, and action category, wherein the initial action interval set includes at least one initial action interval related to the experimental action; By using the fine-tuning model, based on the confidence information and the action category, the initial action interval set is subjected to action merging by eliminating breakpoints, so as to obtain a target action interval with coherence corresponding to the experimental action; According to a preset flexible scoring strategy, an experimental action score is performed based on each target action interval corresponding to the experimental video data to obtain an experimental action scoring result of the experimental video data; Among them, the action classification prediction is performed through sliding window prediction based on the temporal relationship and the experimental video data to obtain an initial action interval set containing complete experimental actions, confidence information and action categories, including: using sliding window technology to scan the video content of the experimental video data according to a preset scanning method to obtain a sliding window set; with respect to the temporal relationship, using the sliding window prediction method, the experimental actions in the sliding window set are predicted and processed by the fine-tuning model to obtain at least one initial action interval related to the experimental action and the action category corresponding to the initial action interval; with respect to the action category, the confidence information of the initial action interval is predicted by the fine-tuning model; the confidence information is used to characterize the correlation between the initial action interval and the action category; The method of scoring the experimental action according to a preset flexible scoring strategy and based on each target action interval corresponding to the experimental video data to obtain the experimental action scoring result of the experimental video data includes: when the flexible scoring strategy is a nonlinear scoring strategy, identifying the nonlinear operation in the target action interval; identifying scoring points for the nonlinear operation to obtain duplicate scoring points and / or jump scoring points, wherein the duplicate scoring points are scoring points corresponding to repetitive nonlinear operations, and the jump scoring points are scoring points corresponding to jumpy nonlinear operations; and performing scoring judgment based on the duplicate scoring points and / or jump scoring points to obtain the experimental action scoring result.

2. The method according to claim 1, characterized in that The multi-dimensional analysis of the experimental video data using a preset fine-tuning model to obtain the temporal relationship of the experimental actions in the experimental video data includes: Inputting the experimental video data into the fine-tuning model; The experimental video data is subjected to spatial dimension analysis and temporal dimension analysis by the fine-tuning model to obtain the temporal relationship between the contexts of the experimental actions in the experimental video data.

3. The method according to claim 1, characterized in that Predicting the experimental actions in the sliding window set using the fine-tuning model to obtain at least one initial action interval related to the experimental action includes: Performing overlapping prediction on the sliding window set by the fine-tuning model to obtain window dispersion information of the experimental action within the sliding window set, wherein the window dispersion information includes non-overlapping window information and overlapping window information; When the window dispersion information is non-overlapping window information, directly extracting a sliding window containing the experimental action from the sliding window set to obtain at least one initial action interval related to the experimental action; When the window dispersion information is overlapping window information, creating an overlapping sliding window set according to the sliding window set; Action prediction is performed according to the overlapping sliding window set to obtain at least three initial action intervals related to the experimental action.

4. The method according to claim 1, wherein The fine-tuning model is used to perform action merging of the initial action interval set by eliminating breakpoints based on the confidence information and the action category to obtain a target action interval with coherence corresponding to the experimental action, including: Get the preset confidence threshold; For each confidence information corresponding to the initial action interval, when the confidence information is not lower than the confidence threshold, performing breakpoint elimination on the corresponding initial action interval to merge the actions, so as to obtain a coherent target action interval corresponding to the experimental action; When the confidence information is lower than the confidence threshold, the initial action interval corresponding to the confidence information below the confidence threshold is filtered through a preset post-processing module, and the initial action interval corresponding to the confidence information not lower than the confidence threshold is merged with breakpoints eliminated to obtain a target action interval with continuity corresponding to the experimental action.

5. The method according to claim 4, characterized in that Performing interval filtering on the initial action intervals corresponding to the confidence information below the confidence threshold by a preset post-processing module, and performing breakpoint elimination and action merging on the initial action intervals corresponding to the confidence information not below the confidence threshold, including: According to the formula , performing interval filtering on the initial action intervals corresponding to the confidence information lower than the confidence threshold, and obtaining the initial action intervals whose confidence information is not lower than the confidence threshold; According to the formula , for the initial action interval after filtering Merge and use time interval threshold Perform a merge operation; in, is the preset confidence threshold, is the initial action interval after filtering, For experimental actions, is the starting frame corresponding to the experimental action, is the end frame corresponding to the experimental action, For experimental actions The initial action interval, satisfy , and the index and , initial action interval and Close in time.

6. The method according to any one of claims 1 to 5, characterized in that The step of performing an experimental action scoring based on each target action interval corresponding to the experimental video data according to a preset flexible scoring strategy to obtain an experimental action scoring result of the experimental video data includes: When the flexible scoring strategy is a parallel scoring strategy, scoring and identifying the target experimental action corresponding to each target action interval is performed according to the parallel scoring strategy to obtain a score action point corresponding to each experimental action; At the same time, all scoring action points are judged in parallel to obtain the action score corresponding to each scoring action point; An experimental action scoring result of the experimental video data is determined according to the action scoring.

7. The method according to any one of claims 1 to 5, characterized in that The step of performing an experimental action scoring based on each target action interval corresponding to the experimental video data according to a preset flexible scoring strategy to obtain an experimental action scoring result of the experimental video data includes: When the flexible scoring strategy is a combined scoring strategy, performing a correlation analysis on the target experimental actions corresponding to the target action intervals to obtain a target action interval combination with correlation; An overall evaluation is performed on the target action interval combination to obtain the experimental action scoring result.

8. An intelligent scoring device for experimental video actions of nonlinear operation, characterized in that: include: An experimental video data acquisition module is used to acquire the experimental video data to be predicted; A multi-dimensional analysis module, configured to perform a multi-dimensional analysis of the experimental video data using a preset fine-tuning model to obtain a temporal relationship between the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data. The fine-tuning model is a video classification model integrated with a visual backbone model. An action classification prediction module is configured to perform action classification prediction through sliding window prediction based on the temporal relationship and the experimental video data, and obtain an initial action interval set containing a complete experimental action, confidence information, and an action category, wherein the initial action interval set includes at least one initial action interval related to the experimental action; an action merging module, configured to perform action merging by eliminating breakpoints on the initial action interval set based on the confidence information and the action category using the fine-tuning model, to obtain a target action interval with coherence corresponding to the experimental action; A scoring module is used to score the experimental action based on each target action interval corresponding to the experimental video data according to a preset flexible scoring strategy, so as to obtain an experimental action scoring result of the experimental video data; Among them, the action classification prediction module includes: a video content scanning submodule, which is used to use the sliding window technology to scan the video content of the experimental video data according to a preset scanning method to obtain a sliding window set; a prediction processing submodule, which is used to use the sliding window prediction method to predict the experimental action in the sliding window set through the fine-tuning model based on the time sequence relationship, and obtain at least one initial action interval related to the experimental action and the action category corresponding to the initial action interval; a confidence prediction submodule, which is used to perform confidence prediction on the initial action interval through the fine-tuning model for the action category to obtain confidence information of the initial action interval; the confidence information is used to characterize the correlation between the initial action interval and the action category; The scoring module includes: a nonlinear scoring submodule, which is used to identify nonlinear operations in the target action interval when the flexible scoring strategy is a nonlinear scoring strategy; identify scoring points for the nonlinear operations to obtain duplicate scoring points and / or jump scoring points, wherein the duplicate scoring points are scoring points corresponding to repetitive nonlinear operations, and the jump scoring points are scoring points corresponding to jumpy nonlinear operations; and perform scoring evaluation based on the duplicate scoring points and / or jump scoring points to obtain an experimental action scoring result.

Images