Experiment video processing method and device

Abstract

The invention discloses an experiment video processing method, and the method comprises the steps: obtaining an experiment video which comprises a plurality of video frame images; extracting image features of the video frame image through an image recognition algorithm so as to recognize key steps in the experiment video; recording timestamps corresponding to the key steps; and displaying the video frame corresponding to the timestamp. According to the scheme, the experiment video can be processed, and when a teacher scores the experiment video, the time stamp corresponding to the key step can be quickly skipped to, so that the video watching time of the teacher is shortened, and the scoring efficiency is greatly improved; in addition, visual fatigue caused by long-time video watching of the teacher can be reduced, and the scoring accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to an experimental video processing method and device. Background Art

[0002] In the traditional experimental examination grading process, teachers usually rely on manually watching the video recording of the entire experimental process and grading students' experimental operations accordingly. This method mainly uses manual observation and subjective judgment as the main means of capturing key steps, and requires teachers to be professional and experienced. On the one hand, this method is time-consuming and labor-intensive, and teachers need to invest a lot of time to watch the complete video. Especially when there are a large number of students, this process becomes particularly cumbersome and inefficient. On the other hand, watching videos for a long time can easily cause visual fatigue in teachers, affecting the accuracy of judgment. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides an experimental video processing method and device, which can improve the efficiency of experimental video scoring.

[0004] In a first aspect, a method for processing an experimental video is provided, comprising: acquiring an experimental video, the experimental video comprising a plurality of video frame images; extracting image features of the video frame images through an image recognition algorithm to identify key steps in the experimental video; recording timestamps corresponding to the key steps; and displaying the video frames corresponding to the timestamps.

[0005] In combination with the first aspect, in some embodiments of the first aspect, there may be multiple key steps, and the processing module may record a corresponding timestamp for each identified key step.

[0006] In combination with the first aspect, in some embodiments of the first aspect, the image features may include one or more of color features, edge features, and key point features.

[0007] In combination with the first aspect, in some embodiments of the first aspect, the extracting image features of the video frame image through an image recognition algorithm to identify the key steps in the experimental video includes: extracting image features of the video frame image through an image recognition algorithm and a trained deep learning model to identify the key steps in the experimental video.

[0008] In combination with the first aspect, in some embodiments of the first aspect, the method further includes: performing intelligent scoring on the experimental operation according to the key steps.

[0009] In combination with the first aspect, in some embodiments of the first aspect, the intelligent scoring of the experimental operation based on the key steps includes: determining the order of the key steps based on LSTM, and intelligently scoring the experimental operation based on the order of the key steps; and / or, intelligently scoring the experimental operation based on the compliance of equipment use in the key steps; and / or, using frame difference technology to determine the integrity of the experimental operation process, and intelligently scoring the experimental operation based on the integrity of the experimental operation process.

[0010] In combination with the first aspect, in some embodiments of the first aspect, before extracting the image features of the video frame image through an image recognition algorithm to identify the key steps in the experimental video, the method also includes denoising the experimental video.

[0011] In combination with the first aspect, in some embodiments of the first aspect, the method further includes storing the experimental video, timestamp information, and associated student personal information and answer data.

[0012] In combination with the first aspect, in some embodiments of the first aspect, a user rating interface and the intelligent rating are presented through a web page.

[0013] In combination with the first aspect, in some embodiments of the first aspect, the method further includes: jumping the experimental video to the timestamp through a synchronous control system.

[0014] In a second aspect, an experimental video processing device is provided, including: an acquisition module for acquiring an experimental video, wherein the experimental video includes multiple video frame images; a processing module for extracting image features of the video frame images through an image recognition algorithm to identify key steps in the experimental video; the processing module is also used to record timestamps corresponding to the key steps; and a display module for displaying the video frame corresponding to the timestamp. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of an experimental video processing method provided in this application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] In order to solve the tedious and inefficient problems caused by manually watching videos to capture key steps and scoring them, the present application provides an experimental video processing method, which can automatically detect the key steps in the experimental process, record the corresponding timestamps or video frames, and provide assistance during scoring to improve scoring efficiency and accuracy.

[0018] It should be understood that the applicable scenarios of the embodiments of the present application are not limited to experimental video processing, but can also be applied to other video scenarios. For the sake of brevity, some embodiments of the present application are described below using the experimental video scenario as an example, but those skilled in the art will be aware that this description does not constitute a limitation on the scope of the present application.

[0019] See also Figure 1 , Figure 1 : is a schematic diagram of an experimental video processing method provided in an embodiment of the present application, the method comprising:

[0020] S110, acquiring an experimental video, wherein the experimental video includes a plurality of video frame images;

[0021] S120, extracting image features of the video frame image by an image recognition algorithm to identify key steps in the experimental video;

[0022] S130, recording the timestamp corresponding to the key step;

[0023] S140: Display the video frame corresponding to the timestamp.

[0024] Based on the above implementation scheme, by identifying the key steps in the experiment and recording the timestamps corresponding to the key steps, the experimental video can be processed, so that when the teacher grades the experimental video, he can quickly jump to the timestamps corresponding to the key steps, reducing the time the teacher spends watching the video, thereby greatly improving the efficiency of the grading work, and reducing the visual fatigue of the teacher caused by watching the video for a long time, thereby improving the accuracy of the grading. In addition, by being able to quickly jump to a specific time point in the video, the teacher can easily review the score, that is, the score review.

[0025] The experiment video may be a recording of the entire experiment process of the student's experimental operation. For example, the experiment may be a chemistry experiment, a physics experiment, etc.

[0026] Specifically, the experimental video in step S100 may be a video directly obtained by the computing device without being processed, for example, uploaded by a user through a user device, downloaded from other devices through a network, shot by a camera, etc., which is not specifically limited in this application; the experimental video may also be a video generated by the computing device through preprocessing. For example, when the perspective of the video is not facing the front of the experimental operation, the computing device may preprocess the video to generate a video with perspective correction.

[0027] In some implementations, there may be multiple key steps, and in step S130, a corresponding timestamp may be recorded for each identified key step.

[0028] In some embodiments, the image features in step S120 may include color features, edge features, key point features, etc., and the key steps are determined based on the image features. For example, the color of the reagent is identified by color features; the outline of the equipment is identified by edge features and key point features, and the placement of the equipment is identified by spatial relationships; the answering action, such as pouring action, is identified by edge features and key point features.

[0029] For example, for color features, different reagent bottles can be identified by color histogram analysis. For example, by calculating the pixel distribution in each color channel, the system can identify the difference between the blue color of copper sulfate solution and other reagents.

[0030] The color histogram can be used to calculate the pixel distribution in each color channel. The calculation process can be expressed as the following formula (1).

[0031]

[0032] Where P(c) is the probability density of color c, N c is the number of pixels of color c, and N is the total number of pixels in the image.

[0033] For example, the Canny algorithm can be used to identify edge features. Specifically, the Canny edge detection algorithm can be used to identify the contour of the equipment, and the calculation process can be expressed as the following formula (2).

[0034]

[0035] Where G(x,y) is the gradient magnitude at point (x,y), I x and I y are the gradients of the image in the x and y directions.

[0036] For example, the key point features can be identified by the OpenPose system, for example, the key points of the examinee's hands can be tracked by the OpenPose system. Specifically, the video or image data can be input into the OpenPose system, the pre-trained OpenPose model can be loaded, the model can be run, the heat map and PAFs of the key points can be output, the key points and connections can be analyzed, and the complete posture of the human body can be obtained.

[0037] In some embodiments, step S120 may specifically include extracting image features of the video frame image through an image recognition algorithm and a trained deep learning model to identify key steps in the experimental video.

[0038] Exemplarily, the deep learning model may include a convolutional layer, a pooling layer, and a fully connected layer.

[0039] In the convolution layer, each neuron only responds to the input within its receptive field, and extracts features from the input feature map by using a series of convolution kernels (or filters). The convolution operation is shown in formula (3).

[0040]

[0041] Among them, O i is the i-th element of the output feature map, W ij is the weight of the convolution kernel, I j is the element of the input feature map, b i is the bias term.

[0042] The pooling layer usually follows the convolutional layer to reduce the spatial size of the feature map, thereby reducing the number of parameters and computational complexity. Max pooling is a common pooling operation that selects the maximum value in a given area as the output, as shown in formula (4).

[0043] P i =max(O i1 ,O i2 ,...,O ik ) (4)

[0044] Among them, P i It is the i-th element of the feature map after pooling, and is the maximum value selected from a region in the input feature map.

[0045] A fully connected layer is a dense layer in a deep learning model, where each neuron is connected to all activation values ​​of the previous layer. The output calculation of the fully connected layer is shown in formula (5).

[0046]

[0047] Among them, Z i is the i-th output of the fully connected layer, O j is the output of the previous layer, W ij is the weight, b i is bias.

[0048] Specifically, the convolutional layers in the deep learning model can be used to identify image features in the video, or key visual features, such as the shape, color, and position of experimental equipment; the pooling layer can be used to reduce the spatial dimension of the features while retaining the most important information, reducing the computational burden of subsequent layers; and the fully connected layer can integrate the features extracted by the previous layers.

[0049] In some embodiments, the method may further include, according to the key steps, intelligently scoring the experimental operation. For example, the color feature may be used to identify whether the examinee has selected the correct reagent; the equipment outline may be used to identify whether the equipment is correct; the edge and key point features may be used to assess whether the pouring action is accurate, or whether the focus is adjusted gently; the spatial relationship feature may be used to check whether the equipment is placed in accordance with the specification, that is, to determine whether the shape of the equipment placement is consistent with the experimental requirements; and the time series analysis may be used to determine whether the order of the experimental steps is correct.

[0050] In some embodiments, the operation sequence recognition can be performed by LSTM (long short term) to analyze the experimental operation sequence in the experimental video. In some embodiments, the operation sequence recognition LSTM can be used to analyze the sequence of key steps in the experimental video.

[0051] In some experiments that require a specific operation sequence, the LSTM network can be used to analyze the time series of the candidates' operations to determine whether the experimental operations are performed in the correct order. Specifically, the following steps can be included:

[0052] S210, acquiring data.

[0053] It can be understood that in the embodiment of the present application, the data may be an experimental video, that is, the video of this step may be the experimental video acquired in step S100, including multiple video frame images.

[0054] S220, extract features.

[0055] This can be completed in the above step S120.

[0056] S230, establishing a sequence.

[0057] Arrange the feature vectors of each step in the chronological order of the experimental process to form a sequence. Each feature vector can be regarded as a time step in the sequence.

[0058] S240, training model.

[0059] The LSTM model is trained using a sequence of experimental operations with annotated order. The model learns the sequence of feature vectors for each step and the temporal dependencies between them.

[0060] S250, identifying the experimental operation.

[0061] During the experiment, the system extracts the feature vector of the current step in real time and inputs it into the trained LSTM model.

[0062] S260, determining the order.

[0063] The LSTM model uses its internal state and memory mechanism to analyze the correlation between the feature vector of the current step and the previous step, and determines whether the current step conforms to the preset operation sequence.

[0064] Specifically, it can be achieved by the following formula.

[0065] Forget Gate, used to determine the state h from the previous state t-1 Which information is discarded in , as shown in formula (6).

[0066] f t =σ(W f .[h t-1 ,x t ]+b f ) (6)

[0067] Input Gate, which determines what new information will be stored in the cell state As shown in formulas (7) and (8).

[0068] i t =σ(W i .[h t-1 ,x t ]+b i ) (7)

[0069]

[0070] Update the cell state (Cell State), where the cell state C t It is obtained by the weighted sum of the outputs of the forget gate and the input gate, as shown in formula (9).

[0071]

[0072] Output Gate, used to set a hidden state h t The value of C is based on the current cell state t The nonlinear transformation is shown in formulas (10) and (11).

[0073] O t =σ(W O.[h t-1 ,x t ]+b O ) (10)

[0074] h t =o t .tanh(C t ) (11)

[0075] In some embodiments, the compliance of equipment use can be determined using SIFT feature matching. In some embodiments, the experimental operation can be intelligently scored according to the compliance of equipment use in key steps.

[0076] In some cases, before the experiment begins, the system can detect the initial placement of the equipment on the experimental table through the spatial relationship feature to ensure that the equipment is placed according to the specified layout. Specifically, the following steps can be included.

[0077] S310, extracting equipment features.

[0078] The SIFT algorithm can be used to extract key points and descriptors from images of standardized equipment in advance as a reference feature set.

[0079] S320, acquiring a real-time image.

[0080] The video acquisition module can be used to capture images of students using equipment in real time.

[0081] S330, detecting key points.

[0082] A SIFT algorithm can be applied to the captured image to detect key points.

[0083] S340, describe the features.

[0084] A SIFT descriptor is generated for each key point, which can represent the characteristics of the local surface in the image, as shown in formula (12).

[0085] D p =SIFT (I,p) (12)

[0086] Where D p is the descriptor of keypoint p.

[0087] S350, matching features.

[0088] Features in the live image can be matched to descriptors in the reference feature set using a nearest neighbor search or other matching strategy.

[0089] S360, Compliance Assessment:

[0090] Based on the matching results, it is possible to assess whether the equipment is being used in compliance with regulations. For example, if the matching characteristics of a balance indicate that it is not placed horizontally, then the use is not in compliance with regulations.

[0091] Specifically, the key point positioning is shown in formula (13):

[0092] L(x,y,σ)=(G*I)(x,y,kσ) (13)

[0093] Where L is the image in scale space, G is the Gaussian kernel, I is the original image, σ is the scale parameter, and k is the scale ratio.

[0094] Exemplarily, regarding descriptor construction, the SIFT descriptor can be used to sample the local area around the key point and construct it by counting the histogram of the gradient direction.

[0095] It can be understood that step S320 can be completed in step S110; S330, S340 and S350 can be completed in S120.

[0096] In some embodiments, the experimental operation can be intelligently scored according to the integrity of the experimental operation process. The integrity of the experimental operation process can be judged by frame difference. Specifically, frame difference technology is a technology that can combine timestamp recording and behavior recognition, and the system can automatically detect whether the student performs the experimental operation according to the prescribed steps, including the following steps:

[0097] S410, obtaining the experimental video;

[0098] S420, extracting a video frame image, or referred to as a frame, from the experimental video;

[0099] Extract consecutive frame pairs from the video, e.g., the current frame.

[0100] S430, frame difference calculation.

[0101] Compute the difference between two frames using the absolute difference method.

[0102] It is understandable that steps S410, S420, and S430 can be completed in step S120.

[0103] In some embodiments, before step 120, the method further includes: preprocessing the experimental video. Exemplarily, the preprocessing may include denoising, which can be used to reduce image noise, improve image quality, and make subsequent feature extraction more accurate.

[0104] The denoising process can be shown as formula (14).

[0105] I′=f(I) (14)

[0106] Where I is the original video frame, I′ is the denoised video frame, and f is the denoising function, which can be a spatial filter or a temporal filter.

[0107] In some embodiments, all video data and timestamp information may be stored in a database management system. The video data includes the above-mentioned experimental video. Further, the video data and timestamp information may be associated with the student's personal information and answer data. The answer data may be the student's score in the experimental video or in a written test or interview in other exams, or other written test videos or interview videos, which are not limited here.

[0108] In some implementations, the timestamp recording in step S130 may be as shown in formula (15).

[0109] Event trigger:

[0110] T = { t | C t >θ} (15)

[0111] Where T is the timestamp set, C t is the confidence at time t, and θ is the preset threshold.

[0112] In some embodiments, the experimental video can be presented to the user through a user rating interface, and the user can be a teacher. Specifically, the user rating interface can be presented on a fixed terminal (or PC terminal), a web terminal (or web terminal), and a mobile terminal. Furthermore, the user rating interface allows the teacher to view the video, the rating criteria, and the student answer data, and to perform ratings.

[0113] In some implementations, the method further includes, by means of a synchronous control system, jumping the experimental video to the above timestamp. Based on the above embodiment, it can be ensured that when the teacher is grading, the video player can automatically jump to a specific timestamp related to the grading.

[0114] In some implementations, the method further includes presenting intelligent grading. Based on this embodiment, it can serve as a reference and auxiliary to the teacher's grading.

[0115] The present invention also provides an experimental video processing device, which comprises:

[0116] An acquisition module, used to acquire an experimental video, wherein the experimental video includes a plurality of video frame images;

[0117] A processing module, used for extracting image features of the video frame images through an image recognition algorithm to identify key steps in the experimental video;

[0118] The processing module is also used to record the timestamp corresponding to the key step;

[0119] The display module is used to display the video frame corresponding to the timestamp.

[0120] In some implementations, there may be multiple key steps, and the processing module may record a corresponding timestamp for each identified key step.

[0121] In some embodiments, the image features may include one or more of color features, edge features, and key point features.

[0122] In some embodiments, specifically, the processing module is used to extract image features of the video frame image through an image recognition algorithm and a trained deep learning model to identify key steps in the experimental video.

[0123] In some embodiments, the processing module is further used to perform intelligent scoring on the experimental operation based on the key steps.

[0124] In some embodiments, the processing module is specifically used to determine the order of the key steps according to LSTM, and intelligently score the experimental operations according to the order of the key steps.

[0125] In some embodiments, the processing module is specifically used to intelligently score the experimental operation according to the compliance of equipment use in key steps.

[0126] In some embodiments, the processing module is specifically used to use frame difference technology to determine the integrity of the experimental operation process, and to intelligently score the experimental operation according to the integrity of the experimental operation process.

[0127] In some embodiments, the processing module is specifically used to perform denoising on the experimental video.

[0128] In some embodiments, the device further includes a storage module, which is used to store the experimental video, timestamp information, and associated student personal information and answer data.

[0129] In some implementations, the display module is further used to present a user rating interface.

[0130] In some implementations, the display module is further configured to present the smart score.

[0131] In some implementations, the display module is further used, and the processing module is further used, to jump the experimental video to the timestamp through a synchronous control system.

[0132] It should be noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0133] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An experimental video processing method, characterized in that: include: Acquire an experimental video, wherein the experimental video includes a plurality of video frame images; Extracting image features of the video frame images by an image recognition algorithm to identify key steps in the experimental video; Record the timestamps corresponding to the key steps; The video frame corresponding to the timestamp is displayed.

2. The method according to claim 1, characterized in that There are multiple key steps, and the processing module records a corresponding timestamp for each identified key step.

3. The method according to claim 1, characterized in that The image features include one or more of color features, edge features, and key point features.

4. The method according to claim 1, characterized in that: The extracting the image features of the video frame images by an image recognition algorithm to identify the key steps in the experimental video includes: The image features of the video frame images are extracted through an image recognition algorithm and a trained deep learning model to identify the key steps in the experimental video implementation.

5. The method according to claim 1, characterized in that The method further comprises: Based on the key steps, the experimental operations are intelligently scored.

6. The method according to claim 5, characterized in that The step of intelligently scoring the experimental operation according to the key steps includes: Determine the sequence of the key steps according to LSTM, and intelligently score the experimental operation according to the sequence of the key steps; And / or, intelligently score the experimental operation according to the compliance of equipment use in key steps; And / or, using frame difference technology to judge the integrity of the experimental operation process, and intelligently scoring the experimental operation according to the integrity of the experimental operation process.

7. The method according to claim 1, characterized in that Before extracting the image features of the video frame images by an image recognition algorithm to identify the key steps in the experimental video, the method also includes performing denoising processing on the experimental video.

8. The method according to claim 1, characterized in that The method further comprises, The experimental video, timestamp information, and associated student personal information and answer data are stored.

9. The method according to claim 1, characterized in that: The method further comprises: jumping the experimental video to the timestamp through a synchronous control system.

10. An experimental video processing device, characterized in that: include: An acquisition module, used to acquire an experimental video, wherein the experimental video includes a plurality of video frame images; A processing module, used for extracting image features of the video frame images through an image recognition algorithm to identify key steps in the experimental video; The processing module is also used to record the timestamp corresponding to the key step; The display module is used to display the video frame corresponding to the timestamp.

Images