Intelligent scoring method and device for experiment video actions of nonlinear operation

Abstract

The invention relates to an action scoring method and device for an experiment video of nonlinear operation, and relates to the technical field of video action recognition, and the method comprises the steps: obtaining experiment video data, carrying out the multi-dimensional analysis of the experiment video data through a fine tuning model, obtaining the time sequence relation of experiment actions in the experiment video data, and obtaining the time sequence relation of the experiment actions in the experiment video data; according to the time sequence relation and the experiment video data, motion classification prediction is carried out through sliding window prediction, an initial motion interval set, confidence coefficient information and motion types with continuity are obtained, motion merging of breakpoint elimination is carried out on initial motion intervals based on the confidence coefficient information and the motion types through a fine tuning model, and motion merging is carried out. According to the method, target action intervals with coherence are obtained, each target action interval is scored according to a flexible scoring strategy, and an experiment action scoring result is obtained, so that accurate action interval positioning is realized, and video clips where scoring points are located can be quickly found and focused; and the accuracy and reliability of scoring are improved through a flexible scoring mode.

Description

Technical Field

[0001] The present application relates to the technical field of video action recognition, and in particular to a method and device for intelligently scoring experimental video actions of non-linear operations. Background Art

[0002] The rapid development of artificial intelligence (AI) technology has profoundly affected the field of education and has been applied to various scoring processes such as grading of physics, chemistry and biology laboratory exams. At present, AI technology is widely used in the education community at home and abroad to automatically evaluate experimental exam videos. This method accurately scores each scoring point in the exam by analyzing the experimental operation videos recorded in the examination room. The AI ​​scoring system not only improves the efficiency of scoring, but also ensures the fairness and accuracy of scoring through the teacher's review process. However, how to further improve the intelligence level of the scoring system and how to optimize the scoring process to make it more efficient and accurate are technical problems that need to be solved urgently.

[0003] Existing AI scoring algorithms mainly use a preset fixed-order scoring mechanism. Under standard circumstances, the scoring points are determined for the experimental actions in the video according to a preset specific order, and then the scoring points are scored. However, this rigid sequential scoring method cannot effectively cope with complex situations in actual exams. For example, students may repeat the operation after completing a certain scoring point, or complete the experimental steps in a non-standard order (ie, non-linear experimental steps). The existing technology can only terminate the scoring process after identifying a preset number of scoring points, ignoring various auxiliary situations that may occur, such as repeated operations or changes in order, and has great limitations, which causes the system to miss additional valid operations, affecting the comprehensiveness and fairness of the scoring. Summary of the invention

[0004] The present application provides a method and device for intelligent scoring of experimental video actions of non-linear operations. Through innovative initial action interval positioning and flexible scoring methods, accurate action analysis and initial action interval positioning are achieved. The diversity and non-linear characteristics of experimental operations are fully considered, and the video clips where the scoring points are located can be quickly found and focused on. A more flexible and intelligent scoring method is used to adapt to the uncertainty of experimental operations, accurately capture scoring points, and avoid scoring errors caused by algorithm limitations. That is, the accuracy, reliability, comprehensiveness and fairness of scoring are improved through flexible scoring methods. Various situations such as repeated operations and sequence adjustments that may occur in students during the experiment can be handled, and problems such as missed judgments and wrong judgments caused by the inability of existing technologies to be applicable to complex and changeable experimental operations are solved.

[0005] In a first aspect, the present application provides a method for intelligently scoring experimental video actions of nonlinear operations, comprising:

[0006] Obtain experimental video data to be predicted;

[0007] Performing a multi-dimensional analysis on the experimental video data through a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data;

[0008] According to the time series relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain an initial action interval set including a complete experimental action, confidence information, and an action category, wherein the initial action interval set includes at least one initial action interval related to the experimental action;

[0009] By using the fine-tuning model, based on the confidence information and the action category, the initial action interval set is subjected to action merging by eliminating breakpoints, so as to obtain a target action interval with coherence corresponding to the experimental action;

[0010] According to a preset flexible scoring strategy, the target experimental action corresponding to each target action interval is scored to obtain an experimental action scoring result.

[0011] Optionally, performing a multi-dimensional analysis on the experimental video data by using a preset fine-tuning model to obtain a temporal relationship of experimental actions in the experimental video data includes:

[0012] Inputting the experimental video data into the fine-tuning model;

[0013] The spatial dimension analysis and the temporal dimension analysis are performed on the experimental video data through the fine-tuning model to obtain the contextual temporal relationship between the experimental actions in the experimental video data.

[0014] Optionally, the step of performing action classification prediction by sliding window prediction based on the temporal relationship and the experimental video data to obtain an initial action interval set containing a complete experimental action, confidence information, and action category includes:

[0015] Using sliding window technology, scanning the video content of the experimental video data according to a preset scanning method to obtain a sliding window set;

[0016] According to the time series relationship, a sliding window prediction method is used to predict the experimental actions in the sliding window set through the fine-tuning model to obtain at least one initial action interval related to the experimental action and an action category corresponding to the initial action interval;

[0017] For the action category, confidence prediction is performed on the initial action interval by using the fine-tuning model to obtain confidence information of the initial action interval;

[0018] The confidence information is used to characterize the correlation between the initial action interval and the action category.

[0019] Optionally, the performing prediction processing on the experimental actions in the sliding window set by the fine-tuning model to obtain at least one initial action interval related to the experimental action includes:

[0020] Performing sliding window prediction on the experimental actions in the sliding window set by using the fine-tuning model to obtain at least one target sliding window containing the experimental actions;

[0021] The interval is acquired based on the target sliding window to obtain at least one initial action interval corresponding to the experimental action, which includes a start frame and an end frame.

[0022] Optionally, the fine-tuning model is used to perform action merging with breakpoint elimination on the initial action interval set based on the confidence information and the action category to obtain a target action interval with coherence corresponding to the experimental action, including:

[0023] Get the preset confidence threshold;

[0024] For the confidence information corresponding to each of the initial action intervals, when the confidence information is not lower than the confidence threshold, the corresponding initial action intervals are merged by eliminating breakpoints to obtain a target action interval with continuity corresponding to the experimental action;

[0025] When the confidence information is lower than the confidence threshold, the initial action interval corresponding to the confidence information below the confidence threshold is filtered through a preset post-processing module, and the initial action interval corresponding to the confidence information not lower than the confidence threshold is merged by eliminating breakpoints to obtain a target action interval with continuity corresponding to the experimental action.

[0026] Optionally, the initial action intervals corresponding to the confidence information lower than the confidence threshold are interval filtered by a preset post-processing module, including:

[0027] According to formula C filtered =Filter(C,θ)=(l,[s,e])∈C|Conf([l,s,e])≥θ, the initial action interval corresponding to the confidence information lower than the confidence threshold is filtered, and the initial action interval whose confidence information is not lower than the confidence threshold is obtained according to the formula

[0028] Merge(C filtered , δ) = (l, [s′, e′]), l∈L, s′=min i∈I (si ), e′=max i∈I (e i ), for the filtered initial action interval C filtered Merge and use the time interval threshold δ to perform the merge operation;

[0029] Among them, θ is the preset confidence threshold, C filtered is the initial action interval after filtering, l is the experimental action, s is the starting frame corresponding to the experimental action, e is the ending frame corresponding to the experimental action, (l, [s, e]) is the initial action interval of the experimental action l, and I satisfies l i = l, and for index i and j, the initial action interval [s i , e i ] and [s j , e j ]Close in time.

[0030] Optionally, scoring the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result includes:

[0031] When the flexible scoring strategy is a parallel scoring strategy, score identification is performed on all experimental actions corresponding to the target action interval according to the parallel scoring strategy to obtain a score action point corresponding to each experimental action;

[0032] At the same time, all scoring action points are judged in parallel to obtain the action score corresponding to each scoring action point;

[0033] An experimental action scoring result is determined according to the action scoring.

[0034] Optionally, scoring the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result includes:

[0035] When the flexible scoring strategy is a nonlinear scoring strategy, identifying nonlinear operations in the target action interval;

[0036] Performing score point identification for the nonlinear operation to obtain duplicate score points and / or jump score points, wherein the duplicate score points are score points corresponding to repetitive nonlinear operations, and the jump score points are score points corresponding to jumpy nonlinear operations;

[0037] Scoring is performed according to the deduplication scoring points and / or jump scoring points to obtain the experimental action scoring result.

[0038] Optionally, scoring the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result includes:

[0039] When the flexible scoring strategy is a combined scoring strategy, a correlation analysis is performed on the target experimental actions corresponding to the target action intervals to obtain a target action interval combination with correlation.

[0040] In a second aspect, the present application provides an intelligent scoring device for experimental video actions of non-linear operation, comprising:

[0041] An experimental video data acquisition module is used to acquire experimental video data to be predicted;

[0042] A multi-dimensional analysis module, used to perform multi-dimensional analysis on the experimental video data through a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data;

[0043] An action classification prediction module is used to perform action classification prediction through sliding window prediction based on the temporal relationship and the experimental video data, and obtain an initial action interval set containing complete experimental actions, confidence information, and action categories;

[0044] An action merging module, configured to perform action merging by eliminating breakpoints on the initial action interval set based on the confidence information and the action category through the fine-tuning model, so as to obtain a target action interval with coherence corresponding to the experimental action;

[0045] The scoring module is used to score the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result.

[0046] In summary, the embodiment of the present application obtains experimental video data to be predicted, performs multi-dimensional analysis on the experimental video data through a fine-tuning model, obtains the temporal relationship of the experimental actions in the experimental video data, and uses the temporal relationship to characterize the contextual timing between the experimental actions of the nonlinear operations in the experimental video data. Subsequently, based on the temporal relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain a set of initial action intervals with continuity, confidence information and action categories. Further, through the fine-tuning model, the initial action intervals are merged by eliminating breakpoints based on the confidence information and action categories to obtain a target action interval with continuity corresponding to the experimental action. According to the flexible scoring strategy, the target experimental action corresponding to the target action interval is scored to obtain the experimental action scoring result, thereby achieving accurate initial action interval positioning, being able to quickly find and focus on the video clips where the scoring points are located, and improving the accuracy and reliability of the scoring through a flexible scoring method. It can be seen that the present application realizes accurate motion analysis and initial motion interval positioning through innovative initial motion interval positioning and flexible scoring methods, fully considers the diversity and nonlinear characteristics of experimental operations, can quickly find and focus on the video clips where the scoring points are located, and uses more flexible and intelligent scoring methods to adapt to the uncertainty of experimental operations, accurately capture scoring points, and avoid scoring errors caused by algorithm limitations. That is, through a flexible scoring method, the accuracy, reliability, comprehensiveness and fairness of scoring are improved, and the problems of missed judgments and wrong judgments caused by the inability of existing technologies to be applicable to complex and changeable experimental operations are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 A schematic diagram of a flow chart of an intelligent scoring method for experimental video actions of non-linear operations provided in an embodiment of the present application;

[0050] Figure 2 It is a schematic diagram of the steps of a method for intelligent scoring of experimental video actions of non-linear operation provided by an optional embodiment of the present application;

[0051] Figure 3 It is a predictive reasoning schematic diagram provided by an optional example of this application;

[0052] Figure 4 A structural block diagram of an intelligent scoring device for experimental video actions of non-linear operation provided in an embodiment of the present application;

[0053] Figure 5 It is a structural diagram of an electronic device provided as an optional example of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0055] In the related art, there are two main technologies for locating the scoring points of experimental videos: ① The target detection method based on a single frame determines the action by analyzing the interaction of the detection frame. This method has obvious shortcomings in practical applications. Due to the lack of overall analysis of the continuous frame sequence, it is difficult for single-frame detection to accurately capture the start and end moments of the action, especially for operations involving subtle or rapid changes. As a result, the system may incorrectly divide the initial action interval and fail to accurately locate the position of certain key scoring point actions, thereby affecting the accuracy of the score; ② Using a preset fixed order scoring mechanism, for example, under standard conditions, an experiment usually contains about 10 scoring points completed in a specific order. The rigid fixed order scoring mechanism can only score the experimental operations that meet the characteristic order as scoring points. This rigid sequential scoring method cannot effectively cope with the complex situations in actual exams. It can only terminate the scoring process after identifying a preset number of scoring points, ignoring the possibility that students may repeat the operation after completing a certain scoring point, or complete the experimental steps in a non-standard order, and fail to pay attention to repeated operations or changes in order. This limitation easily leads to missed judgments and misjudgments of the system. It can be seen that the existing experimental video action scoring methods have great limitations and cannot accurately, comprehensively and fairly score actions.

[0056] To solve the above problems, the present application provides an intelligent scoring method for experimental video actions of nonlinear operations. The method first uses a fine-tuning model to perform multi-dimensional analysis on the experimental video to obtain the temporal relationship between the contexts of the experimental actions of the nonlinear operations, and fully considers the problem of changes in the order of experimental actions. Subsequently, based on the temporal relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain the initial action interval, confidence information and action category of the experimental action with coherence. The fine-tuning model is used to merge the actions of the initial action interval with breakpoints eliminated based on the confidence information and action category to obtain a target action interval with coherence corresponding to the experimental action. Then, according to a flexible scoring strategy, the target experimental action corresponding to the target action interval is scored to obtain the experimental action scoring result. Therefore, the present application fully considers the diversity and nonlinearity of experimental operations and other complex characteristics, realizes accurate motion analysis and initial motion interval positioning, and fully considers the diversity and nonlinearity of experimental operations. It can quickly find and focus on the video clips where the scoring points are located, and use more flexible and intelligent scoring methods to adapt to the uncertainty of experimental operations, accurately capture scoring points, and avoid scoring errors caused by algorithm limitations. That is, through flexible scoring methods, the accuracy, reliability, comprehensiveness and fairness of scoring are improved, and it can handle various situations such as repeated operations and sequence adjustments that may occur in students during the experiment, and solve the problems of missed judgments and wrong judgments caused by the inability of existing technologies to be applicable to complex and changeable experimental operations.

[0057] To facilitate the understanding of the embodiments of the present application, further explanation will be given below in conjunction with the drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present application.

[0058] The nonlinear operation experimental video action intelligent scoring method provided in this application has broad application prospects in the fields of physics, chemistry and biology experimental teaching and examination, and can effectively promote the innovation and development of physics, chemistry and biology experimental operation evaluation. Figure 1 The following is a flow chart of a method for intelligently scoring experimental video actions for nonlinear operations provided in an embodiment of the present application. Figure 1 As shown, the method for intelligent scoring of experimental video actions of non-linear operations provided in the embodiment of the present application may specifically include the following steps:

[0059] Step 110: Acquire experimental video data to be predicted.

[0060] In this embodiment, the experimental video data refers to an experimental video, such as a video of an experimental operation of physics, chemistry and biology, which is not limited in this embodiment. The experimental video data may include multiple experimental actions to be identified. Taking the physics, chemistry and biology experimental examination as an example, the experimental video data is obtained by filming a series of experimental operations performed by students.

[0061] Step 120, performing a multi-dimensional analysis on the experimental video data using a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data.

[0062] The timing relationship is used to characterize the timing of the context between each experimental action in the experimental video data.

[0063] In the specific implementation, considering the complex characteristics of the experimental operation such as diversity and nonlinearity, the experimental steps in the experimental video data are not necessarily completed in strict accordance with the established order, that is, the experimental actions in the experimental video data may be in accordance with the established order, and some experimental actions may not conform to the established order (i.e., non-standard order). To solve the problem of nonlinear operation, in this embodiment, the fine-tuning model can be integrated with a visual backbone model, and the experimental video data is input into the pre-trained fine-tuning model, so that the visual backbone model in the fine-tuning model is used to perform a multi-dimensional analysis of the experimental video data including the time dimension and the space dimension, and the context relationship between each experimental action is obtained as the temporal relationship of the context. The temporal relationship of the context can be used to clarify the context relationship between the linear operation and the nonlinear operation in each experimental action, and the temporal relationship of the context of the video action can also be deeply understood. Compared with the prior art, the model adopted in this embodiment does not need to rely on additional optical flow extraction or other pre-processing steps, so as to simplify the processing flow and improve efficiency.

[0064] Therefore, this embodiment performs a multi-dimensional analysis of the experimental actions in the experimental video data. For the experimental actions that conform to a predetermined sequence, the experimental actions of non-linear operations, and possible repetitive experimental actions in the experimental video data, the sequential relationship between each experimental action can be accurately analyzed, thereby facilitating subsequent classification prediction and scoring.

[0065] Step 130 , based on the temporal relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain an initial action interval set including complete experimental actions, confidence information, and action categories.

[0066] The initial action interval set includes at least one initial action interval related to the experimental action.

[0067] In this embodiment, after the temporal relationship of the experimental actions is obtained, the contextual relationship of each experimental action in the experimental video data can be fully understood. This embodiment uses a sliding window prediction method to first divide the experimental video data into video frames according to a certain number of video frames or according to the input requirements of the model, so as to divide the experimental video data into multiple sliding windows (sliding windows for short). Subsequently, the multiple sliding windows are subjected to action classification recognition / prediction to obtain the action category of each experimental action, and for each action category, the sliding window is subjected to action prediction to obtain the initial action interval set of each experimental action.

[0068] In actual implementation, the initial action interval set of the experimental action with consistency may include video frames including the starting video frame (referred to as the starting frame) to the ending video frame (referred to as the ending frame) of the experimental action. Therefore, the initial action interval corresponding to the experimental action in this embodiment is the sliding window containing the action category. For the sliding window predicted by the model to contain the same action category, it can be used as the initial action interval. The initial action interval can contain video frames of the same action category. By aggregating all the initial action intervals of the same action category, the initial action interval set of the experimental action with consistency is obtained, which is all video frames of the complete target action interval corresponding to the action category.

[0069] In a specific implementation, for the same experimental action, when the experimental action belongs to only one sliding window, the sliding window can be used as the initial action interval; when the experimental action belongs to multiple sliding windows, the initial action interval set can include multiple initial action intervals, so that this embodiment realizes the identification of coherent experimental actions from multiple sliding windows. Subsequently, a confidence prediction is performed on each initial action interval to obtain the confidence that the initial action interval is the action category, which is used as confidence information for subsequent breakpoint elimination processing.

[0070] Step 140 , using the fine-tuning model, based on the confidence information and the action category, the initial action interval set is merged to eliminate breakpoints, so as to obtain a target action interval with coherence corresponding to the experimental action.

[0071] In this embodiment, the target action interval with continuity can be understood as a video containing a complete experimental action, or as a set of multiple video frames including a start frame to an end frame of the experimental action.

[0072] In a specific implementation, the embodiment of the present application performs breakpoint elimination processing on the initial action interval set corresponding to each experimental action of the same action category in turn through a fine-tuning model, and determines whether to retain the initial action interval and merge it by judging whether the confidence corresponding to each initial action interval in the initial action interval set is greater than a preset confidence threshold. Specifically, the present embodiment can process each initial action interval in the initial action interval set in turn or in parallel, and when the confidence corresponding to the initial action interval is less than a preset confidence threshold, the initial action interval is filtered; when the confidence corresponding to the initial action interval is not less than the preset confidence threshold, the initial action interval is retained, until each initial action interval in the initial action interval set is judged, and all retained initial action intervals are merged, thereby achieving breakpoint elimination and obtaining a coherent target action interval corresponding to the experimental action. Therefore, this embodiment first ensures that the confidence levels of all initial action intervals are higher than or equal to a threshold, and then merges these high-confidence initial action intervals, thereby avoiding merging low-confidence prediction results into the final initial action intervals. This helps to improve the accuracy and reliability of action recognition, and can accurately identify and divide the specific time interval of each scoring point in the video, so that the scoring points can be quickly located in subsequent processing, and then accurate action analysis can be performed.

[0073] In actual processing, for each experimental action in the experimental video data, this embodiment can perform sliding window prediction, breakpoint elimination and other processing on the experimental actions sequentially / in parallel, so as to obtain the target action interval corresponding to each experimental action in the experimental video data, and then perform action scoring on each target action interval to obtain the final action scoring result of the experimental video data, that is, execute step 150.

[0074] Step 150 , according to a preset flexible scoring strategy, perform an experimental action scoring based on each of the target action intervals corresponding to the experimental video data to obtain an experimental action scoring result of the experimental video data.

[0075] In this embodiment, the flexible scoring strategy may include but is not limited to: parallel scoring, nonlinear scoring and combined scoring. Among them, parallel scoring refers to judging all identified scoring point actions at the same time, so that it can be free from the constraints of the preset order, and the target experimental action corresponding to the target action interval can be quickly scored; nonlinear scoring can be applicable to nonlinear operations that students may redo or jump to complete the scoring points. The experimental actions of nonlinear operations such as repeated operations or jump operations can be scored by using the nonlinear scoring method. It can be used to respond to the complex situations in the actual experimental operation examination, solve the problems of system omission and misjudgment of additional valid operations caused by the use of a preset fixed order scoring mechanism in the existing AI scoring algorithm, and improve the comprehensiveness and fairness of the scoring; combined scoring can be applied to the situation where multiple target action intervals are combined for scoring. As needed, multiple related target action intervals can be combined for overall judgment. Therefore, the embodiment of the present application optimizes the scoring process and improves the scoring efficiency through a flexible scoring strategy, making the scoring of the experimental test video more efficient and accurate. When the teacher needs to review, he can also quickly find and focus on the video clip where the scoring point is located.

[0076] In summary, the embodiment of the present application obtains experimental video data to be predicted, performs multi-dimensional analysis on the experimental video data through a fine-tuning model, obtains the temporal relationship of the experimental actions in the experimental video data, and uses the temporal relationship to characterize the contextual timing between the experimental actions of the nonlinear operations in the experimental video data. Subsequently, based on the temporal relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain a set of initial action intervals with continuity, confidence information and action categories. Further, through the fine-tuning model, the initial action intervals are merged by eliminating breakpoints based on the confidence information and action categories to obtain a target action interval with continuity corresponding to the experimental action. According to the flexible scoring strategy, the target experimental action corresponding to the target action interval is scored to obtain the experimental action scoring result, thereby achieving accurate initial action interval positioning, being able to quickly find and focus on the video clips where the scoring points are located, and improving the accuracy and reliability of the scoring through a flexible scoring method. It can be seen that the present application realizes accurate motion analysis and initial motion interval positioning through innovative initial motion interval positioning and flexible scoring methods, fully considers the diversity and nonlinear characteristics of experimental operations, can quickly find and focus on the video clips where the scoring points are located, and uses more flexible and intelligent scoring methods to adapt to the uncertainty of experimental operations, accurately capture scoring points, and avoid scoring errors caused by algorithm limitations. That is, through a flexible scoring method, the accuracy, reliability, comprehensiveness and fairness of scoring are improved, and the problems of missed judgments and wrong judgments caused by the inability of existing technologies to be applicable to complex and changeable experimental operations are solved.

[0077] Reference Figure 2 , shows a schematic flow chart of the steps of a method for intelligent scoring of experimental video actions of non-linear operation provided by an optional embodiment of the present application. The method may specifically include the following steps:

[0078] Step 210: Acquire experimental video data to be predicted.

[0079] For the description of step 210, reference may be made to the above step 110, which will not be described in detail in this embodiment.

[0080] Step 220, performing a multi-dimensional analysis on the experimental video data using a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data.

[0081] The timing relationship is used to characterize the context timing between each experimental action in the experimental video data.

[0082] In actual implementation, this embodiment adopts an advanced pre-trained visual backbone model, such as a visual backbone model that may include but is not limited to: VideoMamba model. Before using the visual backbone model to perform multi-dimensional analysis on the experimental video data, the visual backbone model can be trained first. For example, the visual backbone model can be trained using large-scale video data sets such as Kinetics-400, so that the visual backbone model can have the ability to perform in-depth analysis of the video content. After completing the model training, the visual backbone model can be integrated into the fine-tuned video classification model M, referred to as the fine-tuning model.

[0083] Exemplarily, model training for a fine-tuning model may mainly include but is not limited to: sample data preprocessing, model fine-tuning and other processes.

[0084] For sample data: Take the middle school chemistry experiment "Preparation and Testing of Carbon Dioxide" as an example. After years of examination, this experiment has accumulated a large amount of experimental video data. These data are recorded from a unified examination room perspective and cover different equipment, lighting, desktop background and other scenes, with sufficient environmental diversity. First of all, according to the examination requirements, the experiment can generally be divided into 8 specific scoring points: ① Connect the catheter; ② Check air tightness; ③ Add marble to the test tube; ④ Pour dilute hydrochloric acid; ⑤ Fix the test tube on the iron stand; ⑥ Collect gas; ⑦ Carbon dioxide test; ⑧ Arrange equipment and clean the desktop. The specific number of scoring points and divisions can be adjusted as needed. The video samples are marked with scoring point intervals. Specifically, the start and end frames of the scoring point actions are recorded through data annotation tools, and the categories are marked, and then a large-scale data set is formed, thereby completing the sample data preprocessing.

[0085] For model fine-tuning: This example uses the VideoMamba model for fine-tuning (it should be noted that VideoMamba is only one of the available models, and other video backbone models can be selected for replacement). The model uses the cross entropy loss function and optimization methods such as gradient descent to fit the model prediction results with the actual category results, and finally obtains the fine-tuned model M.

[0086] In actual processing, this embodiment inputs the experimental video data into the fine-tuning model. The model can directly receive video clips such as experimental video data as input data, and conduct a comprehensive analysis of its spatial and temporal dimensions, so as to deeply understand the temporal relationship between the context of the video action. Compared with the prior art, the model adopted in this application does not need to rely on additional optical flow extraction or other preprocessing steps, which simplifies the processing flow and improves efficiency. By fine-tuning the model using pre-labeled physical, chemical and biological experimental action data, model M is obtained as a fine-tuning model, which ensures that the model can accurately capture the key actions and scoring points in the experimental video to meet the needs of specific application scenarios.

[0087] In an optional embodiment, the embodiment of the present application performs a multi-dimensional analysis on the experimental video data through a preset fine-tuning model to obtain the temporal relationship of the experimental actions in the experimental video data, which may specifically include: inputting the experimental video data into the fine-tuning model; performing spatial dimension analysis and temporal dimension analysis on the experimental video data through the fine-tuning model to obtain the temporal relationship between the context of the experimental actions in the experimental video data.

[0088] Step 230, using sliding window technology, scanning the video content of the experimental video data according to a preset scanning method to obtain a sliding window set.

[0089] In this embodiment, the sliding window set may include one or more sliding windows. By adopting the sliding window technology, the fine-tuning model can scan the experimental video data frame by frame or according to the set time window to scan the video content, thereby realizing continuous action recognition.

[0090] In actual implementation, this embodiment can use sliding window technology to divide the experimental video data into windows according to the input requirements of the fine-tuning model to obtain multiple sliding windows; of course, the window division can also be performed according to usage requirements. For example, for shorter experimental video data, only one sliding window can be set to improve the speed of action classification and recognition; for longer experimental video data, the division rules can be preset as needed, such as setting the width and step size of the sliding window, and dividing the video frames in the experimental video data according to the width and step size to obtain multiple sliding windows as a sliding window set.

[0091] For example, a set of all frame sequences contained in a certain experimental video is F, then F = f1, f2, ..., f N , where N is the total number of video frames of the experimental video, f1 represents the first video frame of the experimental video, and f N Represents the last video frame of the experimental video. Set the width of the sliding window W to w and the step size to s. Then the j-th sliding window can be expressed as:

[0092] W j =f (j-1)s+1 , f (j-1)s+2 , ..., f js

[0093] Among them, j ranges from 1 to

[0094] Step 240, based on the time series relationship, using the sliding window prediction method, the experimental actions in the sliding window set are predicted by the fine-tuning model to obtain at least one initial action interval related to the experimental action and the action category corresponding to the initial action interval.

[0095] In this embodiment, the action category is usually related to the video data. For example, for an experimental video, its action category is an action related to the experiment. For example, the action category related to the experiment can refer to the above-mentioned example of "production and testing of carbon dioxide". It should be noted that the above-mentioned "production and testing of carbon dioxide" is only an example of one of the experiments and does not constitute a specific limitation on the action category.

[0096] In a specific implementation, the fine-tuned video classification model M (i.e., the fine-tuning model) has the ability to perform in-depth analysis of the video content. By acquiring the input experimental video data, the experimental actions in the experimental video data are classified and identified, and the action category corresponding to each experimental action in the experimental video data is obtained. Then, the initial action interval of the experimental action can be predicted for the action category through a sliding window. Specifically, the fine-tuning model can respectively identify the sliding window where the starting frame of the experimental action and the sliding window where the ending frame are located by predicting the experimental action in the sliding window set. For the same category of experimental actions, if the experimental action involves multiple sliding windows, such as the starting frame and the ending frame of a certain experimental action belong to different sliding windows, then it can be determined first which sliding windows are involved from the starting frame to the ending frame of the experimental action, and then the sliding windows involved are processed until the sliding window corresponding to the ending frame is processed, thereby obtaining multiple initial action intervals containing complete experimental actions; if the experimental action only involves a single sliding window, the complete video frame of the experimental action can be directly identified to obtain a single initial action interval.

[0097] Optionally, the above-mentioned prediction processing of the experimental action in the sliding window set by the fine-tuning model to obtain at least one initial action interval related to the experimental action may specifically include: performing overlapping prediction on the sliding window set by the fine-tuning model to obtain window dispersion information of the experimental action in the sliding window set, the window dispersion information including non-overlapping window information and overlapping window information; when the window dispersion information is non-overlapping window information, directly extracting the sliding window including the experimental action from the sliding window set to obtain at least one initial action interval related to the experimental action; when the window dispersion information is overlapping window information, creating an overlapping sliding window set according to the sliding window set; performing action prediction according to the overlapping sliding window set to obtain at least three initial action intervals related to the experimental action.

[0098] In actual implementation, there are three situations in which the fine-tuning model predicts the same experimental action: ① The experimental action only involves a single sliding window; ② The experimental action spans two adjacent sliding windows, that is, the start frame and the end frame of the experimental action are respectively located in two adjacent sliding windows; ③ The experimental action spans three or more adjacent sliding windows, such as a certain experimental action is scattered in three adjacent sliding windows (such as sliding window A, sliding window B and sliding window C), the start frame and part of the frames after the start frame are located in sliding window A, part of the middle frame is located in sliding window B, and the end frame and part of the frames before the end frame are located in sliding window C. For the case involving a single sliding window or two adjacent sliding windows, when performing overlapping prediction, it is non-overlapping window information, and there is no need to re-divide the sliding window. The sliding window involved in the experimental action can be directly used as the initial action interval; for the case involving three or more adjacent sliding windows, when performing overlapping prediction, it is overlapping window information. This embodiment can re-divide the sliding window set to create an overlapping sliding window set, and obtain at least three initial action intervals related to the experimental action by performing action prediction on the overlapping sliding window set.

[0099] Step 250 : For the action category, confidence prediction is performed on the initial action interval by using the fine-tuning model to obtain confidence information of the initial action interval.

[0100] The confidence information is used to characterize the correlation between the initial action interval and the action category.

[0101] In this embodiment, when the fine-tuning model predicts the action category for the initial action interval, it can perform confidence prediction for the predicted action category. Through the confidence prediction, all initial action intervals of the same action category can be merged / filtered later.

[0102] As an example, for each sliding window Wj , model M predicts an action category l j and confidence Conf(W j ):

[0103] M:W j →(l j , Conf(W j )

[0104] Where C = (l1, [s1, e1], (l2, [s2, e2], ..., (l k ,[s k , e k ]) are all action intervals and their categories predicted by the model, where s i and e i are the start and end frames of the ith action, respectively, and k represents the total number of predicted intervals.

[0105] Exemplarily, for the experimental video to be predicted, taking a 100-frame video as an example (it should be noted that the number of video frames actually used can be more, and this example does not limit the specific total number of video frames), it is assumed that the fine-tuning model accepts a 16-frame sequence input.

[0106] For the case of spanning two adjacent sliding windows (non-overlapping windows): when performing sliding window prediction, a sliding window with a width of 16 frames is used, the step size s=16, and the sliding window is divided into: the first sliding window: 1-16 frames; the second sliding window: 17-32 frames; the third sliding window: 33-48 frames, and so on. Assuming that a certain experimental action, the action category is connecting the catheter, the experimental action starts from the 11th frame and ends at the 25th frame, that is, the starting frame is 11 and the ending frame is 25. According to the above sliding window division, the experimental action of connecting the catheter will be dispersed in two sliding windows, among which the first sliding window (1-16 frames) contains the 11th to 16th frames of the experimental action of connecting the catheter; the second sliding window (17-32 frames) contains the 17th to 25th frames of the experimental action of connecting the catheter. After the model performs sliding window prediction, it can be obtained that the two sliding windows are used as two initial action intervals, so that the first initial action interval contains the 11th to 16th frames of the experimental action, and the second initial action interval contains the 11th to 16th frames of the experimental action. Assume that the model predicts each initial action interval to obtain the action category and confidence. Initial action interval 1 (1-16 frames): predicted as "connect the catheter", confidence 0.75; initial action interval 2 (17-32 frames): predicted as "connect the catheter", confidence 0.90, and then merged and filtered according to the preset confidence threshold and the confidence corresponding to the initial action interval.

[0107] For the case of spanning three or more adjacent sliding windows (overlapping windows): use a sliding window with a width of 16 frames, assuming a step size of s = 8, create overlapping sliding windows, and obtain the sliding window division: the first sliding window: 1-16 frames; the second sliding window: 9-24 frames; the third sliding window: 17-32 frames; the fourth sliding window: 25-40 frames, and so on. A certain experimental action starts from the 11th frame and ends at the 25th frame. This action will be included in three overlapping sliding windows, that is, the first sliding window (1-16 frames) contains the 11th to 16th frames of the action, the second sliding window (9-24 frames) is completely included in the action interval, and the third sliding window (17-32 frames) contains the 17th to 25th frames of the action. These three sliding windows can all be used as the initial action interval of the experimental action. The model predicts each sliding window to obtain the action category and confidence. For example: initial action interval 1 (1-16 frames): predicted as "connect the catheter", confidence 0.69; initial action interval 2 (9-24 frames): predicted as "connect the catheter", confidence 0.95; initial action interval 3 (17-32 frames): predicted as "connect the catheter", confidence 0.85. Subsequently, the preset confidence threshold can be used in combination with the confidence corresponding to the initial action interval for merge filtering.

[0108] Step 260: Obtain a preset confidence threshold.

[0109] Step 270, for the confidence information corresponding to each of the initial action intervals, when the confidence information is not lower than the confidence threshold, the corresponding initial action intervals are merged by eliminating breakpoints to obtain a target action interval with continuity corresponding to the experimental action.

[0110] Step 280, when the confidence information is lower than the confidence threshold, the initial action interval corresponding to the confidence information lower than the confidence threshold is interval filtered through a preset post-processing module, and the initial action interval corresponding to the confidence information not lower than the confidence threshold is action merged by eliminating breakpoints to obtain a target action interval with continuity corresponding to the experimental action.

[0111] A unified description of steps 250 to 280 is given as follows:

[0112] In the specific implementation, in order to overcome the limitations of the model in processing long sequences, this application introduces an efficient post-processing module, which is used to merge the continuous action categories predicted by the sliding window, that is, to merge the initial action intervals of the experimental action, so as to eliminate the potential breakpoints caused by window division and ensure the continuity of the action. By accurately locating the experimental action in the experimental video and accurately dividing the action intervals, the accuracy of subsequent scoring is improved.

[0113] In this embodiment, the post-processing module can identify and exclude those prediction results with low confidence, thereby further improving the accuracy and reliability of classification. Specifically, this embodiment can set a confidence threshold. For each initial action interval predicted by the sliding window as the same action, the fine-tuning model can predict the confidence corresponding to the initial action interval, compare the confidence threshold with the confidence corresponding to the initial action interval, and use the post-processing module to filter the initial action intervals with confidence lower than the confidence threshold, and merge the initial action intervals not lower than the confidence threshold, so as to obtain a complete and coherent target action interval of the same action category.

[0114] Exemplarily, it is assumed that the confidence threshold is set to 0.7. Combined with the above example, for the case of spanning two adjacent sliding windows, it can be seen from the above example that the confidence of the initial action interval 1 and the initial action interval 2 of the "connect the catheter" action are both greater than the confidence threshold 0.7, so the prediction results of the initial action interval 1 and the initial action interval 2 will be retained. The post-processing module will merge the overlapping initial action intervals predicted to be the same action. In this example, the initial action interval 1 and the initial action interval 2 both predict the "connect the catheter" action, and they are adjacent, so the merged target action interval is: (1, 32). After merging, the system will identify a continuous "connect the catheter" action, starting from the 1st frame and ending at the 32nd frame. Although the range is larger than the actual action (11-25 frames), the entire action process is successfully captured.

[0115] For the case of spanning three or more adjacent sliding windows, according to the above example, the confidence of the initial action interval 1 of the "connect the catheter" action is less than the confidence threshold of 0.7, and the initial action interval 1 will be filtered, while the confidence of the initial action interval 2 and the initial action interval 3 are both greater than the confidence threshold of 0.7, so the prediction results of the initial action interval 2 and the initial action interval 3 will be retained. In this example, the initial action interval 2 and the initial action interval 3 both predict "connect the catheter", and they overlap. When merging the action intervals, the post-processing module will merge the overlapping sliding windows predicted as the same action, and the target action interval obtained after the merger is: (9, 32). Finally, the system will identify a continuous "connect the catheter" action, starting at the 9th frame and ending at the 32nd frame. Through this merging method, even if the action spans multiple sliding windows, the post-processing module can still identify and merge it into a continuous action, solving the problem of the same action being dispersed in different sliding windows.

[0116] It should be noted that the size, step size, and whether the window is divided overlappingly in this example can be adjusted as needed.

[0117] As an example, see Figure 3 As shown in the figure, an input test video, through the sliding window method, inputs the frame sequence w (the window length is the sequence length that the model can accept) into the fine-tuned model M to obtain a classification result and confidence. After completing the prediction of all frames, a set of classification results and corresponding confidence levels will be obtained. After passing through a post-processing model, the intervals with low confidence levels are merged and filtered, and then this filtered set is scored through a flexible scoring strategy module to obtain the final score result.

[0118] Optionally, in the embodiment of the present application, the initial action interval corresponding to the confidence information lower than the confidence threshold is filtered by a preset post-processing module, which may specifically include: according to formula C filtered =Filter(C, θ)=(l, [s, e])∈C|Conf([l, s, e])≥θ, the initial action intervals corresponding to the confidence information lower than the confidence threshold are filtered to obtain the initial action intervals whose confidence information is not lower than the confidence threshold; according to the formula Merge(C filtered , δ) = (l, [s′, e′]), l∈L, s′=min i∈I (s i ), e′=max i∈I (e i ), for the filtered initial action interval C filtered Merge and use the time interval threshold δ to perform the merging operation; where θ is the preset confidence threshold, C filtered is the initial action interval after filtering, l is the experimental action, s is the starting frame corresponding to the experimental action, e is the ending frame corresponding to the experimental action, (l, [S, e]) is the initial action interval of the experimental action l, and I satisfies l i = l, and for index i and j, the initial action interval [s i , e i ] and [s j , e j ]Close in time.

[0119] Step 290 , scoring the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy, to obtain an experimental action scoring result.

[0120] In actual implementation, this embodiment can construct a scoring point analysis module, which can be composed of target detection and hand information interaction logic. The model can call the scoring point analysis module to perform accurate right and wrong analysis on the specific scoring interval, so as to assign points to the corresponding scoring actions of the examinee, so that the judgment of the score of the experimental video will no longer be limited to the preset judgment order, and the re-completed actions can also be taken into account. Therefore, the present application optimizes the scoring process, making it more efficient and accurate, and improves the efficiency of the scoring process. When the user (such as a teacher in an experimental test) needs to review, he can also quickly find and focus on the video clip where the scoring point is located, and the present embodiment can also significantly identify the repeated operations performed by the student on a certain scoring point during the experiment, which can adapt to the uncertainty of the student's operation, accurately capture the scoring point, and avoid the scoring error caused by the algorithm limitation, thereby further improving the accuracy and reliability of the scoring.

[0121] In an optional embodiment, the embodiment of the present application scores the target experimental action corresponding to each of the target action intervals according to a preset flexible scoring strategy to obtain an experimental action scoring result, which may specifically include: when the flexible scoring strategy is a parallel scoring strategy, scoring and identifying all experimental actions corresponding to the target action interval according to the parallel scoring strategy to obtain a scored action point corresponding to each experimental action; simultaneously, all scored action points are judged in parallel to obtain an action score corresponding to each scored action point; and the experimental action scoring result is determined according to the action score.

[0122] In this embodiment, based on all pre-identified action scoring intervals, the system can judge multiple scoring points at the same time. The scoring process is not constrained by the preset order, and the judging logic of all scoring points can be executed in parallel, thereby achieving parallel scoring. For example: a. The system simultaneously evaluates all identified action intervals such as "connecting the catheter", "adding marbles to the test tube", and "pouring dilute hydrochloric acid"; b. The scoring logic of each action runs independently, and there is no need to wait for the scoring of other actions to be completed. This method improves the scoring efficiency, especially when processing large amounts of video data.

[0123] In an optional embodiment, the embodiment of the present application scores the target experimental action corresponding to each target action interval according to a preset flexible scoring strategy to obtain an experimental action scoring result, which may specifically include: when the flexible scoring strategy is a nonlinear scoring strategy, identifying the nonlinear operations in the target action interval; identifying scoring points for the nonlinear operations to obtain deduplicated scoring points and / or jump scoring points, the deduplicated scoring points are scoring points corresponding to repetitive nonlinear operations, and the jump scoring points are scoring points corresponding to jumpy nonlinear operations; scoring and judging are performed according to the deduplicated scoring points and / or jump scoring points to obtain the experimental action scoring result.

[0124] In this embodiment, nonlinear scoring enables the system to adapt to the redo or non-standard sequence of operations that may occur when students complete the experiment, but still abide by the basic logical order of the experiment. Correctly handle situations where students may try the same action multiple times. For example: a. Students may perform "airtightness check" immediately after "connecting the catheter" and then "add marble to the test tube"; b. Students may try "connecting the catheter" multiple times, and the system can identify the attempt that is finally successful; c. There are other actions between the two points of "adding dilute hydrochloric acid to the test tube" and "carbon dioxide test", but "adding clear lime water to the test tube" is a sub-step in the "carbon dioxide test". When students operate, this action may be performed after "adding dilute hydrochloric acid to the test tube". The system can still correctly evaluate this non-standard but reasonable operation sequence.

[0125] In an optional embodiment, the embodiment of the present application scores the target experimental actions corresponding to each of the target action intervals according to a preset flexible scoring strategy to obtain an experimental action scoring result, which may specifically include: when the flexible scoring strategy is a combination scoring strategy, performing a correlation analysis on the target experimental actions corresponding to the target action intervals to obtain a target action interval combination with correlation; performing an overall evaluation on the target action interval combination to obtain an experimental action scoring result.

[0126] In this embodiment, combined scoring refers to combining multiple action intervals that are close in time and logically related and making an overall assessment. For example: a. "Connect the catheter" and "Airtightness check" may be considered as a combined action, and the system will check whether the catheter is connected correctly and whether the airtightness check is performed afterwards. b. "Collect gas" and "Carbon dioxide test" can be evaluated in combination, and the system will check the correct order and execution quality of these two actions.

[0127] In summary, the embodiment of the present application obtains the experimental video data to be predicted, and uses the fine-tuning model to perform multi-dimensional analysis on the experimental video data to obtain the temporal relationship between the contexts of each experimental action in the experimental video data, and uses the sliding window technology to scan the experimental video data according to a preset scanning method to obtain a sliding window set. Then, based on the temporal relationship, the sliding window prediction method is used to predict the experimental actions in the sliding window set through the fine-tuning model to obtain at least one initial action interval related to the experimental action and the action category corresponding to the initial action interval. Then, based on the action category, the initial action interval is predicted by the fine-tuning model to obtain confidence information of the initial action interval. For the confidence information corresponding to the initial action interval, the initial action intervals whose confidence information is lower than a preset confidence threshold are filtered through the post-processing module, and the initial action intervals whose confidence information is not lower than the confidence threshold are merged by eliminating breakpoints, so as to obtain a target action interval with coherence. According to the preset flexible scoring strategy, the target experimental action corresponding to each target action interval is scored to obtain the experimental action scoring result. It can be seen that this application realizes accurate frame-level positioning of key actions in experimental videos through advanced sliding window prediction and post-processing optimization technology, which not only solves the limitations of existing technologies, but also brings a variety of breakthrough improvements, including but not limited to: ① Accurate action interval recognition, through sliding window prediction and post-processing optimization, the start and end frame numbers of each experimental action can be accurately determined, and accurate positioning provides a clear and accurate time interval basis for subsequent scoring; ② Flexible scoring strategy, breaking through the limitations of traditional sequential scoring, can support a variety of flexible scoring methods for subsequent scoring points; ③ Adaptable scoring mechanism, can handle various situations of students during the experiment, such as repeated operations, sequence adjustment, etc. It can support dynamic adjustment of scoring standards and easily cope with changes in different experimental requirements or scoring rules; ④ Efficient review support, when the scoring results need to be reviewed, teachers can directly access specific action intervals, thereby significantly improving the efficiency and accuracy of the review work, especially when processing a large number of test videos; ⑤ Comprehensive scoring coverage, can ensure that all key operations are accurately captured and evaluated, avoiding possible omissions in traditional methods.

[0128] The embodiments of this application bring unprecedented accuracy and flexibility to experimental test scoring, open up new possibilities in the field of educational assessment, and provide more comprehensive and fairer scoring results. Through this innovative action interval positioning and flexible scoring method, this application not only improves the accuracy and comprehensiveness of scoring, but also greatly enhances the adaptability and efficiency of the system in dealing with complex experimental scenarios, effectively solving various problems existing in the prior art.

[0129] It should be noted that, for the purpose of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited to the described order of actions, because according to the embodiments of the present application, certain steps may be performed in other orders or simultaneously.

[0130] like Figure 4 As shown, the embodiment of the present application also provides a non-linear operation experimental video action intelligent scoring device 400, comprising:

[0131] The experimental video data acquisition module 410 is used to acquire the experimental video data to be predicted;

[0132] A multi-dimensional analysis module 420, used to perform a multi-dimensional analysis on the experimental video data through a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data;

[0133] An action classification prediction module 430 is used to perform action classification prediction through sliding window prediction based on the temporal relationship and the experimental video data, and obtain an initial action interval set containing a complete experimental action, confidence information, and an action category, wherein the initial action interval set contains at least one initial action interval related to the experimental action;

[0134] An action merging module 440 is used to perform action merging by eliminating breakpoints on the initial action interval set based on the confidence information and the action category through the fine-tuning model to obtain a target action interval with coherence corresponding to the experimental action;

[0135] The scoring module 450 is used to score the experimental action based on each target action interval corresponding to the experimental video data according to a preset flexible scoring strategy, so as to obtain the experimental action scoring result of the experimental video data.

[0136] Optionally, the multi-dimensional analysis module 420 is specifically used to input the experimental video data into the fine-tuning model; perform spatial dimension analysis and temporal dimension analysis on the experimental video data through the fine-tuning model to obtain the contextual temporal relationship between the experimental actions in the experimental video data.

[0137] Optionally, the action classification prediction module 430 includes:

[0138] The video content scanning submodule is used to use the sliding window technology to scan the video content of the experimental video data according to a preset scanning method to obtain a sliding window set;

[0139] A prediction processing submodule is used to predict the experimental actions in the sliding window set by using the fine-tuning model according to the time series relationship and using a sliding window prediction method to obtain at least one initial action interval related to the experimental action and an action category corresponding to the initial action interval;

[0140] The confidence prediction submodule is used to perform confidence prediction on the initial action interval according to the action category through the fine-tuning model to obtain confidence information of the initial action interval; wherein the confidence information is used to characterize the correlation between the initial action interval and the action category.

[0141] Optionally, the prediction processing submodule includes:

[0142] An overlapping prediction unit, configured to perform overlapping prediction on the sliding window set by using the fine-tuning model to obtain window dispersion information of the experimental action within the sliding window set, wherein the window dispersion information includes non-overlapping window information and overlapping window information;

[0143] A first initial action interval determination unit is used to directly extract a sliding window containing the experimental action from the sliding window set when the window dispersion information is non-overlapping window information, to obtain at least one initial action interval related to the experimental action;

[0144] The second initial action interval determination unit is used to create an overlapping sliding window set according to the sliding window set when the window dispersion information is overlapping window information; perform action prediction according to the overlapping sliding window set to obtain at least three initial action intervals related to the experimental action.

[0145] Optionally, the action merging module 440 includes:

[0146] A confidence threshold acquisition submodule is used to obtain a preset confidence threshold;

[0147] The action merging submodule is used to perform action merging by eliminating breakpoints on the corresponding initial action intervals for the confidence information corresponding to each initial action interval when the confidence information is not lower than the confidence threshold, so as to obtain a target action interval with continuity corresponding to the experimental action; when the confidence information is lower than the confidence threshold, perform interval filtering on the initial action intervals corresponding to the confidence information lower than the confidence threshold through a preset post-processing module, and perform action merging by eliminating breakpoints on the initial action intervals corresponding to the confidence information not lower than the confidence threshold, so as to obtain a target action interval with continuity corresponding to the experimental action.

[0148] Optionally, the action merging submodule is specifically used to:filtered =Filter(C,θ)=(l,[s,e])∈C|Conf([l,s,e])≥θ, the initial action intervals corresponding to the confidence information lower than the confidence threshold are filtered to obtain the initial action intervals whose confidence information is not lower than the confidence threshold; according to the formula Merge(C filtered , δ) = (l, [s′, e′]), l∈L, s′=min i∈I (s i ), e′=max i∈I (e i ), for the filtered initial action interval C filtered Merge and use the time interval threshold δ to perform the merging operation; where θ is the preset confidence threshold, C filtered is the initial action interval after filtering, l is the experimental action, s is the starting frame corresponding to the experimental action, e is the ending frame corresponding to the experimental action, (l, [S, e]) is the initial action interval of the experimental action l, and I satisfies l i = l, and for index i and j, the initial action interval [s i , e i ] and [s j , e j ]Close in time.

[0149] Optionally, the scoring module 450 includes:

[0150] A parallel scoring submodule is used for, when the flexible scoring strategy is a parallel scoring strategy, to score and identify the target experimental action corresponding to each target action interval according to the parallel scoring strategy, and obtain the scored action point corresponding to each experimental action; to judge all the scored action points in parallel, and obtain the action score corresponding to each scored action point; and to determine the experimental action scoring result of the experimental video data according to the action score;

[0151] The nonlinear scoring submodule is used to identify the nonlinear operation in the target action interval when the flexible scoring strategy is a nonlinear scoring strategy; perform scoring point identification on the nonlinear operation to obtain duplicate scoring points and / or jump scoring points, wherein the duplicate scoring points are scoring points corresponding to repetitive nonlinear operations, and the jump scoring points are scoring points corresponding to jumpy nonlinear operations; perform scoring judgment based on the duplicate scoring points and / or jump scoring points to obtain the experimental action scoring result.

[0152] The combined scoring submodule is used to perform a correlation analysis on the target experimental actions corresponding to the target action intervals when the flexible scoring strategy is a combined scoring strategy, to obtain a target action interval combination with correlation; and to perform an overall evaluation on the target action interval combination to obtain an experimental action scoring result.

[0153] It should be noted that the device for intelligently scoring experimental video actions of non-linear operations provided in the embodiments of the present application can execute the method for intelligently scoring experimental video actions of non-linear operations provided in any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.

[0154] In a specific implementation, the above-mentioned nonlinear operation experimental video action intelligent scoring device can be integrated in a device, so that the device can accurately classify and predict the experimental actions in the experimental video data in combination with the post-processing module by means of sliding window prediction, and obtain the target action interval with coherence corresponding to the experimental action, and then score the target action interval according to the flexible scoring strategy, so as to achieve accurate initial action interval positioning, quickly find and focus on the video clip where the scoring point is located, and improve the accuracy and reliability of the scoring as an electronic device. The electronic device can be composed of two or more physical entities, or it can be composed of one physical entity, such as the electronic device can be a personal computer (PC), a computer, a server, etc., and the embodiment of the present application does not make specific restrictions on this.

[0155] like Figure 5As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114; the memory 113 is used to store computer programs; the processor 111 is used to implement the steps of the experimental video action intelligent scoring method for non-linear operations provided by any of the aforementioned method embodiments when executing the program stored in the memory 113. Exemplarily, the steps of the intelligent scoring method for experimental video actions of nonlinear operations may include the following steps: obtaining experimental video data to be predicted; performing multi-dimensional analysis on the experimental video data through a preset fine-tuning model to obtain the temporal relationship of the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data; performing action classification prediction through sliding window prediction based on the temporal relationship and the experimental video data to obtain an initial action interval set containing complete experimental actions, confidence information and action categories, wherein the initial action interval set contains at least one initial action interval related to the experimental action; performing action merging with breakpoint elimination on the initial action interval set based on the confidence information and the action category through the fine-tuning model to obtain a target action interval with continuity corresponding to the experimental action; performing experimental action scoring based on each of the target action intervals corresponding to the experimental video data according to a preset flexible scoring strategy to obtain an experimental action scoring result of the experimental video data.

[0156] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for intelligent scoring of experimental video actions for non-linear operations provided in any of the aforementioned method embodiments are implemented.

[0157] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0158] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A nonlinear operation experimental video action intelligent scoring method, characterized in that: include: Obtain experimental video data to be predicted; Performing a multi-dimensional analysis on the experimental video data through a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data; According to the time series relationship and the experimental video data, action classification prediction is performed through sliding window prediction to obtain an initial action interval set including a complete experimental action, confidence information, and an action category, wherein the initial action interval set includes at least one initial action interval related to the experimental action; By using the fine-tuning model, based on the confidence information and the action category, the initial action interval set is subjected to action merging by eliminating breakpoints, so as to obtain a target action interval with coherence corresponding to the experimental action; According to a preset flexible scoring strategy, an experimental action scoring is performed based on each target action interval corresponding to the experimental video data to obtain an experimental action scoring result of the experimental video data.

2. The method according to claim 1, characterized in that The multi-dimensional analysis of the experimental video data by using a preset fine-tuning model to obtain the temporal relationship of the experimental actions in the experimental video data includes: Inputting the experimental video data into the fine-tuning model; The spatial dimension analysis and the temporal dimension analysis are performed on the experimental video data through the fine-tuning model to obtain the contextual temporal relationship between the experimental actions in the experimental video data.

3. The method according to claim 1, characterized in that According to the temporal relationship and the experimental video data, the action classification prediction is performed through sliding window prediction to obtain an initial action interval set containing a complete experimental action, confidence information and action category, including: Using sliding window technology, scanning the video content of the experimental video data according to a preset scanning method to obtain a sliding window set; According to the time series relationship, a sliding window prediction method is used to predict the experimental actions in the sliding window set through the fine-tuning model to obtain at least one initial action interval related to the experimental action and an action category corresponding to the initial action interval; For the action category, confidence prediction is performed on the initial action interval by using the fine-tuning model to obtain confidence information of the initial action interval; The confidence information is used to characterize the correlation between the initial action interval and the action category.

4. The method according to claim 3, characterized in that The experimental actions in the sliding window set are predicted by the fine-tuning model to obtain at least one initial action interval related to the experimental action, including: Performing overlapping prediction on the sliding window set by using the fine-tuning model to obtain window dispersion information of the experimental action in the sliding window set, wherein the window dispersion information includes non-overlapping window information and overlapping window information; When the window dispersion information is non-overlapping window information, directly extracting a sliding window including the experimental action from the sliding window set to obtain at least one initial action interval related to the experimental action; When the window dispersion information is overlapping window information, creating an overlapping sliding window set according to the sliding window set; Action prediction is performed according to the overlapping sliding window set to obtain at least three initial action intervals related to the experimental action.

5. The method according to claim 1, characterized in that The fine-tuning model is used to perform action merging of the initial action interval set by eliminating breakpoints based on the confidence information and the action category to obtain a target action interval with coherence corresponding to the experimental action, including: Get the preset confidence threshold; For the confidence information corresponding to each of the initial action intervals, when the confidence information is not lower than the confidence threshold, the corresponding initial action intervals are merged by eliminating breakpoints to obtain a target action interval with continuity corresponding to the experimental action; When the confidence information is lower than the confidence threshold, the initial action interval corresponding to the confidence information below the confidence threshold is filtered through a preset post-processing module, and the initial action interval corresponding to the confidence information not lower than the confidence threshold is merged by eliminating breakpoints to obtain a target action interval with continuity corresponding to the experimental action.

6. The method according to claim 5, characterized in that The preset post-processing module is used to filter the initial action intervals corresponding to the confidence information below the confidence threshold, and the initial action intervals corresponding to the confidence information not below the confidence threshold are merged by eliminating breakpoints, including: According to formula C filtered =Filter(C,θ)=(l,[s,e])∈C|Conf([l,s,e])≥λ, filtering the initial action intervals corresponding to the confidence information lower than the confidence threshold to obtain the initial action intervals whose confidence information is not lower than the confidence threshold; According to the formula Merge(C filtered ,δ)=(l,[s',e']),l∈L,s'=min i∈I (s i ),e'=max i∈I (e i ), for the filtered initial action interval C filtered Merge and use the time interval threshold δ to perform the merge operation; Among them, θ is the preset confidence threshold, C filtered is the initial action interval after filtering, l is the experimental action, s is the starting frame corresponding to the experimental action, e is the ending frame corresponding to the experimental action, (l, [s, e]) is the initial action interval of the experimental action l, I satisfies l i = l, and for index i and j, the initial action interval [s i ,e i ] and [s j ,e j ]Close in time.

7. The method according to any one of claims 1 to 6, characterized in that: The step of performing an experimental action scoring based on each target action interval corresponding to the experimental video data according to a preset flexible scoring strategy to obtain an experimental action scoring result of the experimental video data includes: When the flexible scoring strategy is a parallel scoring strategy, score identification is performed on the target experimental action corresponding to each target action interval according to the parallel scoring strategy to obtain a score action point corresponding to each experimental action; At the same time, all scoring action points are judged in parallel to obtain the action score corresponding to each scoring action point; An experimental action scoring result of the experimental video data is determined according to the action scoring.

8. The method according to any one of claims 1 to 6, characterized in that: The step of performing an experimental action scoring based on each target action interval corresponding to the experimental video data according to a preset flexible scoring strategy to obtain an experimental action scoring result of the experimental video data includes: When the flexible scoring strategy is a nonlinear scoring strategy, identifying nonlinear operations in the target action interval; Performing score point identification for the nonlinear operation to obtain duplicate score points and / or jump score points, wherein the duplicate score points are score points corresponding to repetitive nonlinear operations, and the jump score points are score points corresponding to jumpy nonlinear operations; Scoring is performed according to the deduplication scoring points and / or jump scoring points to obtain the experimental action scoring result.

9. The method according to any one of claims 1 to 6, characterized in that: The step of performing an experimental action scoring based on each target action interval corresponding to the experimental video data according to a preset flexible scoring strategy to obtain an experimental action scoring result of the experimental video data includes: When the flexible scoring strategy is a combined scoring strategy, a correlation analysis is performed on the target experimental actions corresponding to the target action intervals to obtain a target action interval combination with correlation; An overall evaluation is performed on the target action interval combination to obtain an experimental action scoring result.

10. An intelligent scoring device for experimental video actions of nonlinear operation, characterized in that: include: An experimental video data acquisition module is used to acquire experimental video data to be predicted; A multi-dimensional analysis module, used to perform multi-dimensional analysis on the experimental video data through a preset fine-tuning model to obtain a temporal relationship of the experimental actions in the experimental video data, wherein the temporal relationship is used to characterize the temporal sequence of the context between each experimental action in the experimental video data; An action classification prediction module is used to perform action classification prediction through sliding window prediction based on the time series relationship and the experimental video data to obtain an initial action interval set containing a complete experimental action, confidence information, and action category, wherein the initial action interval set contains at least one initial action interval related to the experimental action; An action merging module, configured to perform action merging by eliminating breakpoints on the initial action interval set based on the confidence information and the action category through the fine-tuning model, so as to obtain a target action interval with coherence corresponding to the experimental action; The scoring module is used to score the experimental action based on each target action interval corresponding to the experimental video data according to a preset flexible scoring strategy to obtain the experimental action scoring result of the experimental video data.

Images