Scene response time determination method and device based on screen recording video and storage medium

Abstract

According to the scene response time determination method and device based on the screen recording video and the storage medium provided by the invention, the screen recording video is acquired, and the screen recording video comprises a starting video frame when entering a target scene, an ending video frame when exiting the target scene, a previous video frame of the starting video frame and a next video frame of the ending video frame; determining a foreground and a background of each video frame in the screen recording video by adopting a Gaussian mixture model, marking the foreground of each video frame by adopting a first color, and marking the background of each video frame by adopting a second color; performing frame splitting on the marked screen recording video, and calculating the similarity corresponding to each connected video frame; and determining a starting video frame and an ending video frame according to the similarities, and determining a time difference between the starting video frame and the ending video frame to obtain response time of the target scene. In this way, the analysis accuracy and stability of the screen recording video can be effectively improved, and then the evaluation effect of the scene response time is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and storage medium for determining a scene response time based on screen recording video. Background Art

[0002] With the development of digitalization, screen recording video is not only a tool for analyzing application response time, but also an important resource for machine vision. Specifically, screen recording video can record the real-time operation process of the computer interface. By analyzing the screen recording video, the response time of the application to different operation scenarios can be accurately evaluated. Screen recording video can also play an important role in the field of machine vision. By analyzing the screen recording video, it can help the algorithm accurately identify the response time of moving objects in different motion scenarios.

[0003] Currently, although there are methods that can perform video content analysis by calculating the similarity between connected frames to determine the scene response time, when calculating the similarity, some small changes in UI pixels may affect the calculation results. For example, when the UI pixels are slightly improved, the similarity calculation results will also be different. In addition, for UIs containing dynamic elements, such as GIFs or UIs that require animation effects, calculating similarity will be more difficult because the changes in dynamic elements may not be simple pixel-level changes, but require more complex processing and analysis.

[0004] Based on this, in the prior art, due to errors in the similarity calculation results between connected frames of the screen recording video, there is a problem of low accuracy in determining the scene response time. Summary of the invention

[0005] The purpose of the present application is to solve at least one of the above-mentioned technical defects, especially the technical defect that the determination of the scene response time in the prior art has low accuracy.

[0006] In a first aspect, the present application provides a method for determining a scene response time based on a screen recording video, the method comprising:

[0007] Obtaining a screen recording video, the screen recording video includes a start video frame when entering a target scene, an end video frame when exiting a target scene, a previous video frame of the start video frame, and a next video frame of the end video frame;

[0008] A Gaussian mixture model is used to determine the foreground and background of each video frame in the screen recording video, and the foreground of each video frame is marked with a first color, and the background of each video frame is marked with a second color;

[0009] Deframe the annotated screen recording video and calculate the similarity of each connected video frame;

[0010] According to each similarity, a start video frame and an end video frame are determined, and a time difference between the start video frame and the end video frame is determined to obtain a response time of the target scene.

[0011] In one embodiment, the step of obtaining the screen recording video includes:

[0012] When in the initial scene, start screen recording, and jump from the initial scene to the target scene after a first preset time;

[0013] When the target scene ends, the screen recording ends after a second preset time, and a screen recording video is obtained.

[0014] In one embodiment, the step of using a Gaussian mixture model to determine the foreground and background of each video frame in the screen recording video includes:

[0015] For each pixel position in the screen recording video, determine the value of the initial parameter type of the Gaussian mixture model corresponding to the pixel position. The initial parameter type of the Gaussian mixture model corresponding to each pixel position is the same, and the initial parameter type of the Gaussian mixture model includes an initial mean value.

[0016] For each pixel point of each video frame, the pixel mean of the pixel point is calculated. If the pixel point is determined to belong to Gaussian distribution according to the absolute value of the difference between the pixel mean value and the initial mean value corresponding to the pixel point position where the pixel point is located, then the pixel point is the background pixel point corresponding to the video frame; otherwise, the pixel point is the foreground pixel point corresponding to the video frame;

[0017] The foreground of each video frame is determined according to each foreground pixel point corresponding to each video frame, and the background of each video frame is determined according to each background pixel point corresponding to each video frame.

[0018] In one embodiment, the step of defragging the annotated screen recording video and calculating the similarity corresponding to each connected video frame includes:

[0019] Split the annotated screen recording video into multiple annotated video frames;

[0020] The features corresponding to each annotated video frame are extracted, and based on each feature, the similarity between each pair of adjacent annotated video frames is calculated.

[0021] In one embodiment, the step of determining the start video frame and the end video frame according to the similarities includes:

[0022] Starting from the first annotated video frame corresponding to the annotated screen recording video, search backward frame by frame. When a pair of adjacent annotated video frames is found to have a similarity lower than a preset threshold, the unannotated video frame corresponding to the later annotated video frame in the pair of adjacent annotated video frames is the starting video frame.

[0023] Starting from the next labeled video frame of the start video frame, search backward frame by frame. When a pair of adjacent labeled video frames are found whose similarity is lower than a preset threshold, the unlabeled video frame corresponding to the earlier labeled video frame in the pair of adjacent labeled video frames is the end video frame.

[0024] In a second aspect, the present application provides a device for determining a scene response time based on a screen recording video, the device comprising:

[0025] A screen recording video acquisition module is used to acquire the screen recording video, which includes a start video frame when entering a target scene, an end video frame when exiting a target scene, a previous video frame of the start video frame, and a next video frame of the end video frame;

[0026] A foreground and background determination module is used to determine the foreground and background of each video frame in the screen recording video using a Gaussian mixture model, and mark the foreground of each video frame with a first color and the background of each video frame with a second color;

[0027] A similarity calculation module is used to deframe the annotated screen recording video and calculate the similarity corresponding to each connected video frame;

[0028] The response time determination module is used to determine the start video frame and the end video frame according to each similarity, and determine the time difference between the start video frame and the end video frame to obtain the response time of the target scene.

[0029] In one embodiment, the screen recording video acquisition module includes:

[0030] A screen recording start unit, used to start screen recording when in an initial scene, and jump from the initial scene to a target scene after a first preset time;

[0031] The screen recording end unit is used to end the screen recording after a second preset time when the target scene ends, and obtain the screen recording video.

[0032] In one of the embodiments, the foreground background determination module includes:

[0033] A parameter initialization unit, used to determine, for each pixel position in the screen recording video, a value of an initial parameter type of a Gaussian mixture model corresponding to the pixel position, wherein the initial parameter types of the Gaussian mixture model corresponding to each pixel position are the same, and the initial parameter types of the Gaussian mixture model include an initial mean value;

[0034] A foreground and background pixel determination unit is used to calculate the pixel mean of each pixel of each video frame, if the pixel is determined to belong to Gaussian distribution according to the absolute value of the difference between the pixel mean and the initial mean value corresponding to the pixel position where the pixel is located, then the pixel is the background pixel corresponding to the video frame, otherwise the pixel is the foreground pixel corresponding to the video frame;

[0035] The foreground-background determination unit is used to determine the foreground of each video frame according to each foreground pixel point corresponding to each video frame, and to determine the background of each video frame according to each background pixel point corresponding to each video frame.

[0036] In a third aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of a method for determining a scene response time based on a screen recording video as described in any of the above embodiments.

[0037] In a fourth aspect, the present application provides a computer device, comprising: one or more processors, and a memory;

[0038] Computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by one or more processors, the steps of the method for determining the scene response time based on the screen recording video in any of the above embodiments are performed.

[0039] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0040] In the scene response time determination method, device, storage medium and computer equipment based on screen recording video provided in the present application, the screen recording video is first obtained, and the screen recording video includes a start video frame when entering the target scene, an end video frame when exiting the target scene, a previous video frame of the start video frame and a next video frame of the end video frame; then, a Gaussian mixture model is used to determine the foreground and background of each video frame in the screen recording video, so as to better handle the changes of dynamic elements, even if the changes of dynamic elements are not simple pixel-level changes, the similarity can be calculated more accurately; then, the foreground of each video frame is marked with a first color, and the background of each video frame is marked with a second color, which can reduce the impact of slight changes in UI pixels on the similarity calculation results, so that when the marked screen recording video is deframed and the similarities corresponding to each connected video frame are calculated, the actual changes in the UI can be more accurately identified, thereby reducing errors and improving the accuracy and stability of the similarity calculation; further, according to each similarity, the start video frame and the end video frame can be more reliably determined, and the time difference between the start video frame and the end video frame can be determined to obtain the response time of the target scene. In this way, the accuracy and stability of the analysis of screen recording videos can be effectively improved, thereby improving the evaluation effect of scene response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A schematic diagram of a flow chart of a method for determining a scene response time based on screen recording video provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of the structure of a device for determining scene response time based on screen recording video provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] In one embodiment, the present application provides a method for determining scene response time based on screen recording video. The following embodiments are described by taking the method applied to a computer device as an example. It can be understood that the computer device can be any device with data processing functions, including but not limited to a single server, a server cluster, a personal laptop, a desktop computer, etc. Figure 1 As shown, the method includes:

[0047] S101: Obtain a screen recording video, where the screen recording video includes a start video frame when entering a target scene, an end video frame when exiting the target scene, a previous video frame of the start video frame, and a next video frame of the end video frame.

[0048] Among them, screen recording video refers to a video file generated by recording the operation process on the computer screen, which can be the user's interaction in the application or web page, or the automatic execution process of the system or software. The target scene refers to the specific scene or operation that is of interest in the screen recording video, which may be a specific interface or a specific operation task. The starting video frame refers to the video frame corresponding to the start of the target scene in the screen recording video of the target scene, that is, the frame in which the target scene first appears in the screen recording video. The ending video frame refers to the video frame corresponding to the end of the target scene in the screen recording video of the target scene, that is, the frame in which the target scene last appears in the screen recording video. The previous video frame refers to the video frame before the starting video frame, that is, the frame before the start of the target scene. The next video frame refers to the video frame after the ending video frame, that is, the frame after the end of the target scene.

[0049] In this step, after determining the target scene to be analyzed, you can use a special screen recording software or the system's built-in screen recording function to perform the target operation on the computer, record the entire operation process to generate a video file. When recording the target scene, you need to start recording before entering the target scene and end recording after exiting the target scene to ensure that the operation process of the target scene can be fully recorded.

[0050] S102: Use a Gaussian mixture model to determine the foreground and background of each video frame in the screen recording video, and use a first color to mark the foreground of each video frame and a second color to mark the background of each video frame.

[0051] Among them, Gaussian Mixture Model (GMM) is a statistical model used to model and cluster data. In image processing, GMM is used for background modeling, that is, decomposing the image into two parts: background and foreground. It assumes that each pixel in the image is composed of multiple Gaussian distributions, one part of which corresponds to the background and the other part corresponds to the foreground. In image processing, the foreground refers to the object of interest or the main target in the image, while the background refers to the other parts corresponding to the foreground, usually the relatively static and unchanged part of the background in the image. Color annotation refers to marking or coloring different parts of the image with different colors to analyze, understand or visualize the image.

[0052] In this step, for each frame in the screen recording video, the GMM model is used to segment the foreground and background to determine whether each pixel belongs to the foreground or the background. According to the segmentation results of the foreground and background, each video frame is color-labeled, using the first color to label the foreground and the second color to label the background. It can be understood that each frame can also be binarized according to the segmentation results of the foreground and background to distinguish the foreground from the background.

[0053] In one example, the GMM model needs to be initialized first, including determining the number of Gaussian distributions in the model and initializing the mean, variance and weight of each Gaussian distribution. Next, the EM algorithm (Expectation-Maximization) is used to train the GMM model. The EM algorithm is an iterative optimization algorithm that maximizes the likelihood function by alternating between the E step (Expectation step) and the M step (Maximization step). The probability that each pixel belongs to each Gaussian distribution is calculated, and the probability that each pixel belongs to each Gaussian distribution is calculated using the Bayesian theorem based on the current model parameters and pixel values. According to the posterior probability calculated in the E step, the model parameters are updated, including the mean, variance and weight of each Gaussian distribution, to maximize the likelihood function. The E step and the M step are repeated until the model parameters converge or the preset number of iterations is reached. After the training is completed, the trained GMM model can be used to segment the image foreground and background. For each pixel, according to its posterior probability of belonging to each Gaussian distribution, the background or foreground corresponding to the Gaussian distribution with the maximum posterior probability is selected. In order to further improve the accuracy of segmentation, threshold processing can be performed according to the posterior probability value, and pixels with posterior probabilities lower than a certain threshold are marked as background, and pixels with posterior probabilities higher than the threshold are marked as foreground.

[0054] S103: De-frame the annotated screen recording video and calculate the similarity corresponding to each connected video frame.

[0055] The video is composed of a series of still image frames, and each video frame is a static image in the video. Frame splitting refers to the process of breaking a video into a series of separate image frames. Similarity refers to the similarity of the content between adjacent video frames.

[0056] In this step, you can use a video processing library or tool to split the annotated screen recording video frame by frame into a series of image frames and save them as separate image files. For each pair of adjacent video frames, compare the similarity between adjacent frames. You can use various similarity measurement methods, such as Euclidean distance, cosine similarity, etc., to calculate the similarity between them. The higher the similarity, the smaller the content difference, and the lower the similarity, the greater the content difference.

[0057] S104: Determine a start video frame and an end video frame according to each similarity, and determine a time difference between the start video frame and the end video frame to obtain a response time of the target scene.

[0058] The time difference refers to the time interval between the start video frame and the end video frame.

[0059] In this step, the similarities are sorted according to the calculated similarities between adjacent video frames, and the adjacent frame pairs with the highest similarity are found. Based on the sorted adjacent frame pairs, the start video frame and the end video frame are determined. In one example, a threshold can be set. When the similarity is higher than the threshold, it is considered to be a continuous frame in the same scene, thereby determining the start and end frames. After determining the start and end video frames, the time difference between them can be calculated by the timestamps of the two frames or the position information (number of frames) in the video. Furthermore, if the screen recording video contains timestamp information, the difference in timestamps is calculated directly; if there is no timestamp, the difference in the number of frames is calculated according to the frame rate of the video and converted into a time difference. After obtaining the time difference between the start video frame and the end video frame, it can be used as the response time of the target scene. The response time represents the time interval between the system or event receiving an instruction or trigger and generating feedback or results.

[0060] In the above embodiment, a screen recording video is first obtained, and the screen recording video includes a start video frame when entering the target scene, an end video frame when exiting the target scene, a previous video frame of the start video frame, and a next video frame of the end video frame; then, a Gaussian mixture model is used to determine the foreground and background of each video frame in the screen recording video, so as to better handle the changes of dynamic elements, even if the changes of dynamic elements are not simple pixel-level changes, the similarity can be calculated more accurately; then, the foreground of each video frame is marked with a first color, and the background of each video frame is marked with a second color, which can reduce the influence of small changes in UI pixels on the similarity calculation results, so that when the marked screen recording video is deframed and the similarities corresponding to each connected video frame are calculated, the actual changes in the UI can be more accurately identified, thereby reducing errors and improving the accuracy and stability of similarity calculation; further, the start video frame and the end video frame can be more reliably determined according to each similarity, and the time difference between the start video frame and the end video frame can be determined to obtain the response time of the target scene. In this way, the analysis accuracy and stability of the screen recording video can be effectively improved, thereby improving the evaluation effect of the scene response time.

[0061] In one embodiment, the step of obtaining the screen recording video includes:

[0062] When in the initial scene, start screen recording, and jump from the initial scene to the target scene after a first preset time;

[0063] When the target scene ends, the screen recording ends after a second preset time, and a screen recording video is obtained.

[0064] The initial scene refers to a scene that is different from the target scene and has a jump relationship with the target scene, and the target scene can be directly jumped from the initial scene.

[0065] Specifically, you can use appropriate screen recording software or programming interfaces, such as Python's pyautogui library, to start the screen recording function in the initial scene. Start recording the video and record the start timestamp. The program enters a waiting state and continues to monitor the current scene. When the preset first duration is reached, it switches to the target scene. Among them, the switch from the initial scene to the target scene can be achieved through a programming interface or simulated keystrokes. Record the switching timestamp as the start time of the target scene. In the target scene, the program continues to monitor. After the target scene ends, record the end timestamp. After the target scene ends, the program waits for the second preset duration to ensure that the entire content of the target scene is captured. After the second preset duration ends, stop recording the video and save the recorded video file.

[0066] In this embodiment, the start and end time of the recorded video is controlled by presetting the duration, ensuring that the recorded video includes the complete target scene and that the recording is kept long enough when necessary.

[0067] In one embodiment, the step of using a Gaussian mixture model to determine the foreground and background of each video frame in the screen recording video includes:

[0068] For each pixel position in the screen recording video, determine the value of the initial parameter type of the Gaussian mixture model corresponding to the pixel position. The initial parameter type of the Gaussian mixture model corresponding to each pixel position is the same, and the initial parameter type of the Gaussian mixture model includes an initial mean value.

[0069] For each pixel point of each video frame, the pixel mean of the pixel point is calculated. If the pixel point is determined to belong to Gaussian distribution according to the absolute value of the difference between the pixel mean value and the initial mean value corresponding to the pixel point position where the pixel point is located, then the pixel point is the background pixel point corresponding to the video frame; otherwise, the pixel point is the foreground pixel point corresponding to the video frame;

[0070] The foreground of each video frame is determined according to each foreground pixel point corresponding to each video frame, and the background of each video frame is determined according to each background pixel point corresponding to each video frame.

[0071] Specifically, a corresponding Gaussian mixture model can be constructed for each pixel position in the screen recording video, and the parameters of each Gaussian mixture model can be initialized so that the Gaussian mixture model parameters corresponding to each pixel are the same, wherein the parameters of the Gaussian mixture model include the mean. Next, the pixel mean corresponding to each pixel of each video frame is calculated, and the difference is made with the mean of the Gaussian mixture model. If the pixel satisfies the Gaussian distribution according to the absolute value of the difference, then the pixel belongs to the background, otherwise it belongs to the foreground. Afterwards, for each video frame, the pixels belonging to the foreground are marked as foreground, and the pixels belonging to the background are marked as background.

[0072] In one example, for each pixel position in the video, from the beginning to the end of the video, the parameters of a Gaussian mixture model are initialized, including K Gaussian models, each of which has three parameters: Gaussian mixture coefficient t, mean m and variance d. It is calculated whether the absolute value of the pixel mean of each pixel in each video frame minus the mean of the corresponding Gaussian model is within 2d. If so, it means that the point belongs to the Gaussian distribution, and the pixel belongs to the background. If not, it does not belong to the Gaussian distribution model, and the pixel belongs to the foreground.

[0073] In this embodiment, the foreground and background are identified by using the pixel mean and the Gaussian mixture model, so that the accuracy of distinguishing the foreground from the background can be improved.

[0074] In one embodiment, the steps of defragging the annotated screen recording video and calculating the similarity of each connected video frame include:

[0075] Split the annotated screen recording video into multiple annotated video frames;

[0076] The features corresponding to each annotated video frame are extracted, and based on each feature, the similarity between each pair of adjacent annotated video frames is calculated.

[0077] Specifically, the annotated screen recording video is first split into multiple annotated video frame images. For each pair of adjacent video frames, image processing techniques, such as feature extraction algorithms, can be used to extract their feature representations, such as using a convolutional neural network (CNN) to extract the feature vector of the image. To compare the feature vectors between adjacent frames, various similarity metrics, such as Euclidean distance, cosine similarity, etc., can be used to calculate the similarity between them.

[0078] In this embodiment, by calculating the similarity based on the features corresponding to each labeled video frame, the similarity between adjacent video frames can be calculated more accurately.

[0079] In one embodiment, the step of determining the start video frame and the end video frame according to the similarities includes:

[0080] Starting from the first annotated video frame corresponding to the annotated screen recording video, search backward frame by frame. When a pair of adjacent annotated video frames is found to have a similarity lower than a preset threshold, the unannotated video frame corresponding to the later annotated video frame in the pair of adjacent annotated video frames is the starting video frame.

[0081] Starting from the next labeled video frame of the start video frame, search backward frame by frame. When a pair of adjacent labeled video frames are found whose similarity is lower than a preset threshold, the unlabeled video frame corresponding to the earlier labeled video frame in the pair of adjacent labeled video frames is the end video frame.

[0082] Specifically, in the annotated screen recording video, starting from the first frame, the size relationship between the similarity of adjacent video frames and the preset threshold is compared frame by frame. Since the screen recording video enters the target scene from one scene and then exits the target scene to enter another scene, the similarity between the video frames corresponding to the moment of scene change is low. Therefore, when the similarity between the first pair of adjacent video frames is less than the preset threshold, the later video frame is used as the starting video frame. After that, starting from the starting video frame, the size relationship between the similarity of adjacent video frames and the preset threshold is compared frame by frame. When the similarity between the first pair of adjacent video frames is less than the preset threshold, the earlier video frame is used as the ending video frame.

[0083] In this embodiment, each video frame can be checked without omission through frame-by-frame analysis, and the ending video frame can be determined after the starting video frame is determined, which can reduce the amount of calculation for video frame analysis. Moreover, since the starting video frame and the ending video frame exist in a time sequence, the ending video frame can be found more accurately after the starting video frame.

[0084] The following is a description of a scene response time determination device based on screen recording video provided in an embodiment of the present application. The scene response time determination device based on screen recording video described below and the scene response time determination method based on screen recording video described above can be referred to each other. Figure 2 As shown, the device comprises:

[0085] The screen recording video acquisition module 201 is used to acquire the screen recording video, which includes the start video frame when entering the target scene, the end video frame when exiting the target scene, the previous video frame of the start video frame, and the next video frame of the end video frame;

[0086] A foreground and background determination module 202 is used to determine the foreground and background of each video frame in the screen recording video using a Gaussian mixture model, and mark the foreground of each video frame with a first color and mark the background of each video frame with a second color;

[0087] A similarity calculation module 203 is used to deframe the annotated screen recording video and calculate the similarity corresponding to each connected video frame;

[0088] The response time determination module 204 is used to determine the start video frame and the end video frame according to each similarity, and determine the time difference between the start video frame and the end video frame to obtain the response time of the target scene.

[0089] In one embodiment, the screen recording video acquisition module 201 includes:

[0090] A screen recording start unit, used to start screen recording when in an initial scene, and jump from the initial scene to a target scene after a first preset time;

[0091] The screen recording end unit is used to end the screen recording after a second preset time when the target scene ends, and obtain the screen recording video.

[0092] In one embodiment, the foreground background determination module 202 includes:

[0093] A parameter initialization unit, for each pixel position in the screen recording video, determines the value of the initial parameter type of the Gaussian mixture model corresponding to the pixel position, the initial parameter type of the Gaussian mixture model corresponding to each pixel position is the same, and the initial parameter type of the Gaussian mixture model includes an initial mean;

[0094] A foreground and background pixel determination unit is used to calculate the pixel mean of each pixel of each video frame, if the pixel is determined to belong to Gaussian distribution according to the absolute value of the difference between the pixel mean and the initial mean value corresponding to the pixel position where the pixel is located, then the pixel is the background pixel corresponding to the video frame, otherwise the pixel is the foreground pixel corresponding to the video frame;

[0095] The foreground-background determination unit is used to determine the foreground of each video frame according to each foreground pixel point corresponding to each video frame, and to determine the background of each video frame according to each background pixel point corresponding to each video frame.

[0096] In one embodiment, the similarity calculation module 203 includes:

[0097] A screen recording video frame splitting unit is used to split the annotated screen recording video into multiple annotated video frames;

[0098] The similarity calculation unit is used to extract the features corresponding to each labeled video frame, and calculate the similarity between each pair of adjacent labeled video frames based on each feature.

[0099] In one embodiment, the response time determination module 204 includes:

[0100] A start video frame determination unit is used to start from the first marked video frame corresponding to the marked screen recording video, and search backward frame by frame. When a pair of adjacent marked video frames are found to have a similarity lower than a preset threshold, the unmarked video frame corresponding to the later marked video frame in the pair of adjacent marked video frames is the start video frame;

[0101] The end video frame determination unit is used to start from the next labeled video frame of the start video frame and search backward frame by frame. When a pair of adjacent labeled video frames are found to have a similarity lower than a preset threshold, the unlabeled video frame corresponding to the earlier labeled video frame in the pair of adjacent labeled video frames is the end video frame.

[0102] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for determining the scene response time based on the screen recording video as described in any of the above embodiments.

[0103] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for determining the scene response time based on the screen recording video as described in any of the above embodiments.

[0104] Indicatively, Figure 3 As shown, Figure 3 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 3 The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301, for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the scene response time determination method based on screen recording video of any of the above embodiments.

[0105] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0106] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, "one", "one", "said", "the" and "it" may also include plural forms, unless the context clearly indicates another way. A plurality refers to at least two cases, such as 2, 3, 5 or 8, etc. "And / or" includes any and all combinations of the relevant listed items.

[0108] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0109] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining scene response time based on screen recording video, characterized in that: The method comprises: Acquire the screen recording video, wherein the screen recording video includes a start video frame when entering a target scene, an end video frame when exiting the target scene, a previous video frame of the start video frame, and a next video frame of the end video frame; A Gaussian mixture model is used to determine the foreground and background of each video frame in the screen recording video, and the foreground of each video frame is marked with a first color, and the background of each video frame is marked with a second color; Deframe the annotated screen recording video and calculate the similarity of each connected video frame; According to each of the similarities, the start video frame and the end video frame are determined, and a time difference between the start video frame and the end video frame is determined to obtain a response time of the target scene.

2. The method for determining scene response time based on screen recording video according to claim 1, characterized in that: The step of obtaining the screen recording video includes: When in the initial scene, start screen recording, and jump from the initial scene to the target scene after a first preset time; When the target scene ends, the screen recording ends after a second preset time period, and the screen recording video is obtained.

3. The method for determining scene response time based on screen recording video according to claim 1, characterized in that: The step of using a Gaussian mixture model to determine the foreground and background of each video frame in the screen recording video includes: For each pixel position in the screen recording video, determine the value of the initial parameter type of the Gaussian mixture model corresponding to the pixel position, the initial parameter type of the Gaussian mixture model corresponding to each pixel position is the same, and the initial parameter type of the Gaussian mixture model includes an initial mean; For each pixel point of each of the video frames, a pixel mean value of the pixel point is calculated; if the pixel point is determined to belong to a Gaussian distribution based on an absolute value of a difference between the pixel mean value and an initial mean value corresponding to the pixel point position where the pixel point is located, then the pixel point is a background pixel point corresponding to the video frame; otherwise, the pixel point is a foreground pixel point corresponding to the video frame; The foreground of each video frame is determined according to each foreground pixel point corresponding to each video frame, and the background of each video frame is determined according to each background pixel point corresponding to each video frame.

4. The method for determining scene response time based on screen recording video according to claim 1, characterized in that: The step of defragging the annotated screen recording video and calculating the similarity corresponding to each connected video frame includes: Split the annotated screen recording video into multiple annotated video frames; The features corresponding to each of the labeled video frames are extracted, and the similarity between each pair of adjacent labeled video frames is calculated based on the features.

5. The method for determining scene response time based on screen recording video according to claim 2, characterized in that: The step of determining the start video frame and the end video frame according to each of the similarities comprises: Starting from the first annotated video frame corresponding to the annotated screen recording video, searching backward frame by frame, when a pair of adjacent annotated video frames is found to have a similarity lower than a preset threshold, the unannotated video frame corresponding to the later annotated video frame in the pair of adjacent annotated video frames is the starting video frame; Starting from the next labeled video frame of the start video frame, searching backward frame by frame, when a pair of adjacent labeled video frames are found whose similarity is lower than the preset threshold, in the pair of adjacent labeled video frames, the unlabeled video frame corresponding to the earlier labeled video frame is the end video frame.

6. A device for determining scene response time based on screen recording video, characterized in that: The device comprises: A screen recording video acquisition module, used to acquire the screen recording video, wherein the screen recording video includes a start video frame when entering a target scene, an end video frame when exiting the target scene, a previous video frame of the start video frame, and a next video frame of the end video frame; A foreground and background determination module, used to determine the foreground and background of each video frame in the screen recording video using a Gaussian mixture model, and mark the foreground of each video frame with a first color and mark the background of each video frame with a second color; A similarity calculation module is used to deframe the annotated screen recording video and calculate the similarity corresponding to each connected video frame; The response time determination module is used to determine the start video frame and the end video frame according to each of the similarities, and determine the time difference between the start video frame and the end video frame to obtain the response time of the target scene.

7. The device for determining scene response time based on screen recording video according to claim 6, characterized in that: The screen recording video acquisition module includes: A screen recording start unit, used to start screen recording when in an initial scene, and jump from the initial scene to the target scene after a first preset time; The screen recording end unit is used to end the screen recording after a second preset time period when the target scene ends, and obtain the screen recording video.

8. The device for determining scene response time based on screen recording video according to claim 6, characterized in that: The foreground background determination module comprises: A parameter initialization unit, used to determine, for each pixel position in the screen recording video, a value of an initial parameter type of a Gaussian mixture model corresponding to the pixel position, wherein the initial parameter types of the Gaussian mixture model corresponding to each pixel position are the same, and the initial parameter types of the Gaussian mixture model include an initial mean value; a foreground and background pixel determination unit, configured to calculate a pixel mean value of each pixel of each of the video frames, and if the pixel is determined to belong to a Gaussian distribution based on an absolute value of a difference between the pixel mean value and an initial mean value corresponding to the pixel position where the pixel is located, then the pixel is a background pixel corresponding to the video frame; otherwise, the pixel is a foreground pixel corresponding to the video frame; The foreground-background determination unit is used to determine the foreground of each video frame according to each foreground pixel point corresponding to each video frame, and to determine the background of each video frame according to each background pixel point corresponding to each video frame.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for determining the scene response time based on the screen recording video as described in any one of claims 1 to 5.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the method for determining scene response time based on screen recording video as described in any one of claims 1 to 5 are performed.

Images