Key frame searching method and device, storage medium and computer device

Abstract

The application provides a key frame searching method and device, a storage medium and a computer device. A first frame or a last frame of a video is compared with a control frame, and a similarity fluctuation value is obtained according to the comparison. A key frame is determined according to a maximum similarity fluctuation value, so that the interference of GIF dynamic effects can be eliminated as much as possible, and the accuracy of key frame searching can be improved. On the other hand, for each video frame, the absolute values of pixel difference between the video frame and the control frame at each pixel coordinate point are calculated respectively, and the absolute values of pixel difference that meet the condition are selectively amplified, so as to calculate an image difference value corresponding to the video frame, so that the image difference value can reflect the slight difference between the video frame and the control frame. According to the difference fluctuation value of each image difference value, the key frame is determined, the key frame can be searched in combination with the slight difference between the video frame and the control frame, and the accuracy of key frame searching can be improved.

Description

Technical Field

[0001] This application relates to the field of application testing technology, and in particular to a keyframe search method, apparatus, storage medium, and computer device. Background Technology

[0002] During the testing of the application under test (AUT), it is necessary to obtain the complete response time of the AUT in response to the triggering of a control. This allows technical personnel to determine whether optimization of the AUT is needed based on the complete response time. Specifically, after a user triggers a control in the AUT, the AUT responds to the event by displaying a result interface corresponding to that event. The testing algorithm determines the triggering time of the control and the earliest display time of the result interface, thus using the time interval between the triggering time and the earliest display time as the aforementioned complete response time.

[0003] Currently, to automatically determine the trigger moment of a control and the earliest display moment of the result interface, the complete response process of the application under test to a single control response can be recorded in advance to obtain a video of the control response. For example, with the application under test running on the terminal, after screen recording of the terminal, the controls of the application under test are triggered through actual user operation or simulated user operation. After the application under test smoothly displays the corresponding result interface, the screen recording ends. This way, a video of the control response can be obtained.

[0004] After obtaining the control response video, existing technologies use existing image similarity algorithms to calculate the similarity between two adjacent frames in the control response video. When the similarity is less than a similarity threshold, the frame corresponding to the preceding frame is taken as the end frame of the response process. Then, existing technologies automatically determine the response time based on the frame difference between the first and end frames of the control response video. However, the inventors have discovered that in some cases, the end frame determined by existing technologies deviates significantly from the actual end frame; in other words, existing technologies suffer from low accuracy. Summary of the Invention

[0005] The purpose of this application is to address at least one of the aforementioned technical deficiencies, particularly the low accuracy of existing technologies.

[0006] In a first aspect, embodiments of this application provide a keyframe lookup method, the method comprising:

[0007] Obtain each video frame of the control response video, wherein the control response video is a video obtained by recording the complete control response process;

[0008] Use the video frame with the earliest timestamp or the video frame with the latest timestamp as the comparison frame.

[0009] For each pair of video frames with adjacent timestamps, a first similarity value is calculated between the video frame with the earlier timestamp and the comparison frame, and a second similarity value is calculated between the video frame with the later timestamp and the comparison frame. Based on the difference between the second similarity value and the first similarity value, the similarity fluctuation value corresponding to the two adjacent timestamps is obtained.

[0010] The similarity fluctuation value with the largest absolute value among all the aforementioned similarity fluctuation values ​​is taken as the maximum similarity fluctuation value;

[0011] For each video frame, the absolute value of the pixel difference between the video frame and the comparison frame at each pixel coordinate point is calculated. The absolute values ​​of the pixel difference that are greater than or equal to the first preset pixel difference threshold are amplified. Based on the amplified absolute values ​​of the pixel difference and the absolute values ​​of the pixel difference that are less than the first preset pixel difference threshold, the image difference value between the video frame and the comparison frame is calculated.

[0012] For each pair of video frames with adjacent timestamps, the difference fluctuation value corresponding to the two adjacent timestamps is obtained based on the difference between the image difference value corresponding to the video frame with the later timestamp and the image difference value corresponding to the video frame with the earlier timestamp.

[0013] Based on the maximum similarity fluctuation value and each of the difference fluctuation values, key frames are determined from each of the video frames; wherein, when the comparison frame is the video frame with the earliest timestamp, the key frame is the starting frame of the control response process; when the comparison frame is the video frame with the latest timestamp, the key frame is the ending frame of the control response process.

[0014] In one embodiment, the step of determining keyframes from each of the video frames based on the maximum similarity fluctuation value and each of the difference fluctuation values ​​includes:

[0015] The video frame corresponding to the maximum similarity fluctuation value is taken as the maximum fluctuation frame, and the timestamp of the maximum fluctuation frame is taken as the target timestamp.

[0016] If the comparison frame is the video frame with the earliest timestamp among all the video frames, then based on the frame with the largest fluctuation, N target frames with sequentially adjacent timestamps are determined from all the video frames, and the variance of the difference values ​​of the image difference values ​​corresponding to the N target frames is calculated; wherein, the target timestamp is the latest timestamp among the N target frames, and N is a preset positive integer;

[0017] If the comparison frame is the video frame with the earliest timestamp among all the video frames, then when the variance of the difference value is greater than the preset variance threshold, the video frames with timestamps earlier than the target timestamp among all the video frames are taken as the search frames; when the variance of the difference value is less than or equal to the preset variance threshold, the video frames with timestamps no later than the target timestamp among all the video frames are taken as the search frames.

[0018] The key frame is determined from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched.

[0019] In one embodiment, the step of determining keyframes from each of the video frames based on the maximum similarity fluctuation value and each of the difference fluctuation values ​​includes:

[0020] The video frame corresponding to the maximum similarity fluctuation value is taken as the maximum fluctuation frame, and the timestamp of the maximum fluctuation frame is taken as the target timestamp.

[0021] If the comparison frame is the video frame with the latest timestamp among all the video frames, then based on the frame with the largest fluctuation, N target frames with sequentially adjacent timestamps are determined from all the video frames, and the variance of the difference values ​​of the image difference values ​​corresponding to the N target frames is calculated; wherein, the target timestamp is the earliest timestamp among the N target frames, and N is a preset positive integer;

[0022] If the comparison frame is the video frame with the latest timestamp among all the video frames, then when the variance of the difference value is greater than the preset variance threshold, the video frames with timestamps later than the target timestamp among all the video frames are taken as the search frames; when the variance of the difference value is less than or equal to the preset variance threshold, the video frames with timestamps no earlier than the target timestamp among all the video frames are taken as the search frames.

[0023] The key frame is determined from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched.

[0024] In one embodiment, the step of determining the key frame from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched includes:

[0025] The target extreme value frame is determined from each of the frames to be searched, and the target extreme value frame whose timestamp is closest to the target timestamp is taken as the key frame; wherein, the target extreme value frame is the video frame corresponding to the target difference fluctuation value extreme value, and the target difference fluctuation value extreme value is the difference fluctuation value extreme value whose absolute value is greater than a preset fluctuation threshold.

[0026] In one embodiment, the step of amplifying the absolute values ​​of pixel differences that are greater than or equal to a first preset pixel difference threshold includes:

[0027] For each pixel difference absolute value corresponding to the video frame, if the pixel difference absolute value is greater than or equal to the first preset pixel difference threshold, then the magnification factor is determined based on the pixel difference absolute value, and the pixel difference absolute value is magnified based on the magnification factor; wherein, the magnification factor is positively correlated with the pixel difference absolute value.

[0028] In one embodiment, the step of determining the magnification factor based on the absolute value of the pixel difference includes:

[0029] If the absolute value of the pixel difference is less than the second preset pixel difference threshold, then the absolute value of the pixel difference is used as the magnification factor; otherwise, the square of the absolute value of the pixel difference is used as the magnification factor; wherein, the second preset pixel difference threshold is greater than the first preset pixel difference threshold.

[0030] In one embodiment, the step of determining the image difference value between the video frame and the comparison frame based on the absolute values ​​of pixel differences that are less than the first preset pixel difference threshold and the absolute values ​​of pixel differences after magnification includes:

[0031] Calculate the sum of squares of the absolute values ​​of pixel differences that are less than the first preset pixel difference threshold to obtain the first sum of squares;

[0032] Calculate the sum of squares of the absolute values ​​of the differences between each magnified pixel to obtain the second sum of squares;

[0033] The first sum of squares is added to the second sum of squares to obtain the image difference value between the video frame and the comparison frame.

[0034] Secondly, embodiments of this application provide a keyframe lookup device, the device comprising:

[0035] The video frame acquisition module is used to acquire each video frame of the control response video, wherein the control response video is a video obtained by recording the complete control response process;

[0036] The comparison frame acquisition module is used to select the video frame with the earliest timestamp or the video frame with the latest timestamp from each video frame as the comparison frame.

[0037] The similarity fluctuation value determination module is used to calculate, for each pair of video frames with adjacent timestamps, a first similarity value between the video frame with the earlier timestamp and the comparison frame, and a second similarity value between the video frame with the later timestamp and the comparison frame, and obtain the similarity fluctuation value corresponding to the two adjacent timestamps based on the difference between the second similarity value and the first similarity value.

[0038] The maximum fluctuation value determination module is used to determine the maximum similarity fluctuation value as the largest similarity fluctuation value among all the similarity fluctuation values.

[0039] The difference value acquisition module is used to calculate the absolute value of the pixel difference between the video frame and the comparison frame at each pixel coordinate point for each video frame, amplify the absolute values ​​of the pixel difference that are greater than or equal to a first preset pixel difference threshold, and calculate the image difference value between the video frame and the comparison frame based on the amplified absolute values ​​of the pixel difference and the absolute values ​​of the pixel difference that are less than the first preset pixel difference threshold.

[0040] The difference fluctuation value determination module is used to determine the difference fluctuation value of each pair of adjacent video frames based on the difference between the image difference value corresponding to the video frame with the later timestamp and the image difference value corresponding to the video frame with the earlier timestamp.

[0041] The keyframe determination module is used to determine keyframes from each of the video frames based on the maximum similarity fluctuation value and each of the difference fluctuation values; wherein, when the comparison frame is the video frame with the earliest timestamp, the keyframe is the starting frame of the control response process; when the comparison frame is the video frame with the latest timestamp, the keyframe is the ending frame of the control response process.

[0042] Thirdly, embodiments of this application provide a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the keyframe search method described in any of the above embodiments.

[0043] Fourthly, embodiments of this application provide a computer device, including: one or more processors, and a memory;

[0044] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the keyframe lookup method described in any of the above embodiments.

[0045] In the keyframe search method, apparatus, storage medium and computer device of this application, the first frame or the last frame of the control response video is used as the comparison frame, and the similarity value between each video frame in the control response video and the comparison frame is calculated respectively. The similarity fluctuation value between each two video frames with adjacent timestamps is determined, and the similarity fluctuation value with the largest absolute value is taken as the maximum similarity fluctuation value.

[0046] For each video frame, the computer device can calculate the absolute value of the pixel difference between the video frame and the comparison frame at each pixel coordinate point. Based on the values ​​of the absolute values ​​of the pixel differences corresponding to the video frame, it selectively amplifies the absolute values ​​of the pixel differences that meet the conditions, and calculates the image difference value between the video frame and the comparison frame accordingly. After obtaining the image difference values ​​corresponding to each video frame, the computer device can determine the difference fluctuation value between every two adjacent timestamps of video frames, and determine the keyframes from each video frame based on the maximum similarity fluctuation value and each difference fluctuation value.

[0047] On one hand, this application compares the first or last frame of the video with the control response frame and calculates the similarity fluctuation value accordingly. Since the image undergoes significant abrupt changes during the control response process, leading to large fluctuations in the similarity value, this application determines keyframes based on the maximum similarity fluctuation value. This minimizes interference from GIF animations and improves the accuracy of keyframe finding. On the other hand, for each video frame, this application calculates the absolute value of the pixel difference between the video frame and the comparison frame at each pixel coordinate point, and selectively amplifies the absolute values ​​of pixel differences that meet certain conditions to calculate the image difference value corresponding to that video frame. This image difference value reflects the subtle differences between the video frame and the comparison frame. By determining keyframes based on the fluctuation values ​​of each image difference value, this application can find keyframes by combining the subtle differences between the video frame and the comparison frame, thereby improving the accuracy of keyframe finding. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating a keyframe lookup method in one embodiment;

[0050] Figure 2 This refers to the similarity value corresponding to each video frame in one embodiment;

[0051] Figure 3 for Figure 2 The similarity fluctuation values ​​corresponding to each video frame are shown in the example.

[0052] Figure 4 for Figure 2 The example shown shows the image difference values ​​corresponding to each video frame;

[0053] Figure 5 for Figure 4 The example shown illustrates the variation values ​​corresponding to each video frame;

[0054] Figure 6 This is a schematic diagram of the keyframe lookup device in one embodiment;

[0055] Figure 7 This is a schematic diagram of the structure of a computer device in one embodiment. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] As mentioned in the background section, existing technologies suffer from low accuracy in keyframe determination. The inventors' research revealed that this problem stems from two main causes: First, recorded control response videos may include GIF animations. These GIFs may exist both during and after the control response. Under the influence of GIFs, even if two adjacent frames are neither the start nor end frames, their similarity may be less than or equal to a threshold, leading to misclassification of keyframes. Second, in recorded control response videos, some frames may only have subtle differences within a small area. Similarity calculations need to consider these subtle differences to accurately assess the similarity between two frames and thus accurately determine keyframes. However, existing technologies are insensitive to these minute differences, easily leading to misclassification of keyframes.

[0058] To address the aforementioned problems, this application provides a keyframe finding method, apparatus, storage medium, and computer device. On one hand, this application determines keyframes based on the maximum similarity fluctuation value, thereby minimizing interference from GIF animations and improving the accuracy of keyframe finding. On the other hand, this application determines keyframes based on the difference fluctuation value of various image differences, allowing for the search of keyframes by considering subtle differences between video frames and comparison frames, thus further improving the accuracy of keyframe finding.

[0059] In one embodiment, this application provides a keyframe lookup method. The following embodiments illustrate this method applied to a computer device. It is understood that the computer device can be a terminal or a server; when the computer device is a server, the server can be a single server or a server cluster. Figure 1 As shown, the method specifically includes the following steps:

[0060] S102: Obtain each video frame of the control response video, wherein the control response video is a video obtained by recording the complete control response process.

[0061] The control response video is a video recording the complete control response process; in other words, it includes the entire process from triggering the control to the application under test smoothly displaying the result interface corresponding to the control trigger. In one embodiment, the control response video can begin recording before the control is triggered and end recording after the application under test smoothly displays the result interface corresponding to the control trigger.

[0062] Specifically, the computer device can acquire individual video frames of the control response video. In one embodiment, the computer device can acquire the control response video and process it using a video frame splitting tool (such as ffmpeg) to obtain the individual video frames of the control response video.

[0063] S104: Select the video frame with the earliest timestamp or the video frame with the latest timestamp from among all video frames as the comparison frame.

[0064] In the control response video, each video frame corresponds to a timestamp, which indicates the playback time and / or playback order of that video frame within the control response video. The video frame with the earliest timestamp is the first video frame in the control response video, i.e., the earliest video frame played. The video frame with the latest timestamp is the last video frame in the control response video, i.e., the latest video frame played.

[0065] The computer device can use either the first or last video frame of the control response video as the comparison frame, depending on the key frame to be determined. Specifically, if the key frame to be determined is the starting frame of the control response process, the computer device can use the first video frame among all video frames as the comparison frame. If the key frame to be determined is the ending frame of the control response process, the computer device can use the last video frame among all video frames as the comparison frame.

[0066] S106: For every two video frames with adjacent timestamps, calculate the first similarity value between the video frame with the earlier timestamp and the comparison frame, and the second similarity value between the video frame with the later timestamp and the comparison frame. Based on the difference between the second similarity value and the first similarity value, obtain the similarity fluctuation value corresponding to the two video frames with adjacent timestamps.

[0067] After obtaining each video frame of the control response video and determining the comparison frame, the computer device can calculate the similarity value between each video frame and the comparison frame, and determine the similarity fluctuation value corresponding to each video frame accordingly. This similarity fluctuation value reflects the change in the similarity value between two video frames with adjacent timestamps. It is understood that this application can use any image similarity calculation algorithm involved in the prior art to determine the similarity value between each video frame and the comparison frame. In one embodiment, this application can use a pixel point statistical method to determine the similarity value corresponding to each video frame separately.

[0068] Specifically, for every two video frames with adjacent timestamps, the computer device can calculate a similarity value between the video frame with the earlier timestamp and the comparison frame; this similarity value is the first similarity value. The computer device can also calculate a similarity value between the video frame with the later timestamp and the comparison frame; this similarity value is the second similarity value. After obtaining the first and second similarity values, the computer device can determine the similarity fluctuation value between the two video frames with adjacent timestamps based on the difference between the second and first similarity values. In one embodiment, the computer device can use the difference between the second and first similarity values ​​as the similarity fluctuation value between the two video frames with adjacent timestamps.

[0069] For example, Figure 2 The illustration shows the similarity values ​​corresponding to frames 1 to 160 in one embodiment. When calculating the similarity fluctuation value, the computer device can use the difference between the similarity value corresponding to frame 2 and the similarity value corresponding to frame 1 as the similarity fluctuation value corresponding to frame 2; the difference between the similarity value corresponding to frame 3 and the similarity value corresponding to frame 2 as the similarity fluctuation value corresponding to frame 3; the difference between the similarity value corresponding to frame 4 and the similarity value corresponding to frame 3 as the similarity fluctuation value corresponding to frame 4. This process continues until the calculated similarity value is obtained. Figure 3 The similarity fluctuation values ​​for each video frame are shown.

[0070] S108: Among all the similarity fluctuation values, the one with the largest absolute value is taken as the maximum similarity fluctuation value.

[0071] After obtaining the various similarity fluctuation values, the computer device can take the value with the largest absolute value as the maximum similarity fluctuation value. The video frames with two adjacent timestamps corresponding to the maximum similarity fluctuation value are the video frames with the greatest image change during the control response process.

[0072] S110: For each video frame, calculate the absolute value of the pixel difference between the video frame and the comparison frame at each pixel coordinate point. Amplify the absolute values ​​of the pixel differences that are greater than or equal to a first preset pixel difference threshold. Calculate the image difference value between the video frame and the comparison frame based on the amplified absolute values ​​of the pixel differences and the absolute values ​​of the pixel differences that are less than the first preset pixel difference threshold.

[0073] It is understood that the specific value of the first preset pixel difference threshold in this application can be determined according to the actual situation, and this application does not impose specific restrictions on it. In one example, the first preset pixel difference threshold is 5.

[0074] Specifically, a video frame may include multiple pixel coordinate points. For each video frame, the computer device may perform the following steps to calculate the image difference value between that video frame and a comparison frame:

[0075] For each pixel coordinate point of the video frame, the computer can obtain the first pixel value corresponding to the pixel coordinate point (x, y) of the video frame and the second pixel value corresponding to the pixel coordinate point (x, y) of the comparison frame, and calculate the absolute value of the difference between the first pixel value and the second pixel value. This absolute value of the difference is the absolute value of the pixel difference corresponding to the pixel coordinate point (x, y) of the video frame.

[0076] After obtaining the absolute value of the pixel difference between the video frame and the comparison frame at each pixel coordinate point, the computer device can take the absolute value of the pixel difference that is greater than or equal to the first preset pixel difference threshold as the target absolute value, and take the absolute value of the pixel difference that is less than the first preset pixel difference threshold as the non-target absolute value.

[0077] For each non-target absolute value, since its value is less than the first preset pixel difference threshold, it indicates that the pixel difference between the video frame and the comparison frame at the corresponding pixel coordinate point is not significant, so it does not need to be magnified; for each target absolute value, since its value is greater than or equal to the first preset pixel difference threshold, it indicates that the pixel difference between the video frame and the comparison frame at the corresponding pixel coordinate point is significant, so the computer device can magnify each target absolute value separately to obtain each magnified target absolute value;

[0078] The computer device can calculate the image difference value between the video frame and the comparison frame based on the absolute values ​​of each non-target and each magnified target. This image difference value can reflect the subtle differences between the video frame and the comparison frame.

[0079] S112: For every two video frames with adjacent timestamps, the difference fluctuation value corresponding to the two adjacent timestamps is obtained based on the difference between the image difference value corresponding to the video frame with the later timestamp and the image difference value corresponding to the video frame with the earlier timestamp.

[0080] After obtaining the image difference value corresponding to each video frame, the computer device can determine the difference fluctuation value corresponding to each video frame. This difference fluctuation value reflects the change in the image difference value between two video frames with adjacent timestamps.

[0081] Specifically, for any two video frames with adjacent timestamps, if the image difference value corresponding to the video frame with the later timestamp is used as the first difference value, and the image difference value corresponding to the video frame with the earlier timestamp is used as the second difference value, then the computer device can obtain the difference fluctuation value corresponding to the two video frames with adjacent timestamps based on the difference between the first difference value and the second difference value. In one embodiment, the computer device can use the difference between the first difference value and the second difference value as the difference fluctuation value corresponding to the two video frames with adjacent timestamps.

[0082] For example, Figure 4 The illustration shows image difference values ​​corresponding to frames 1 to 160 in one embodiment. When calculating the difference fluctuation value, the computer device can use the difference between the image difference value corresponding to frame 2 and the image difference value corresponding to frame 1 as the difference fluctuation value for frame 2; the difference between the image difference value corresponding to frame 3 and the image difference value corresponding to frame 2 as the difference fluctuation value for frame 3; the difference between the image difference value corresponding to frame 4 and the image difference value corresponding to frame 3 as the difference fluctuation value for frame 4. This process continues until the difference is calculated. Figure 5 The difference fluctuation values ​​corresponding to each video frame are shown.

[0083] S114: Determine key frames from each of the video frames based on the maximum similarity fluctuation value and each of the difference fluctuation values; wherein, when the comparison frame is the video frame with the earliest timestamp, the key frame is the starting frame of the control response process; when the comparison frame is the video frame with the latest timestamp, the key frame is the ending frame of the control response process.

[0084] Computer devices can determine keyframes by combining the maximum similarity fluctuation value and various difference fluctuation values. When the comparison frame is the first frame of the control response video, the determined keyframe is the starting frame of the control response process. When the comparison frame is the last frame of the control response video, the determined keyframe is the ending frame of the control response process.

[0085] In this embodiment, on one hand, the computer device compares the first or last frame of the video with a control response and calculates the similarity fluctuation value accordingly. Since the image undergoes significant abrupt changes during the control response, resulting in large fluctuations in the similarity value, the computer device determines keyframes based on the maximum similarity fluctuation value. This minimizes interference from GIF animations and improves the accuracy of keyframe finding. On the other hand, for each video frame, the computer device calculates the absolute value of the pixel difference between the video frame and the comparison frame at each pixel coordinate point. It selectively amplifies the absolute values ​​of pixel differences that meet certain conditions and calculates the corresponding image difference value for that video frame. This image difference value reflects the subtle differences between the video frame and the comparison frame. This application determines keyframes based on the fluctuation values ​​of each image difference value, combining the subtle differences between the video frame and the comparison frame to find keyframes, thereby improving the accuracy of keyframe finding.

[0086] In one embodiment, the step of determining keyframes from each of the video frames based on the maximum similarity fluctuation value and each of the difference fluctuation values ​​includes:

[0087] The video frame corresponding to the maximum similarity fluctuation value is taken as the maximum fluctuation frame, and the timestamp of the maximum fluctuation frame is taken as the target timestamp.

[0088] If the comparison frame is the video frame with the earliest timestamp among all the video frames, then based on the frame with the largest fluctuation, N target frames with sequentially adjacent timestamps are determined from all the video frames, and the variance of the difference values ​​of the image difference values ​​corresponding to the N target frames is calculated; wherein, the target timestamp is the latest timestamp among the N target frames, and N is a preset positive integer;

[0089] If the comparison frame is the video frame with the earliest timestamp among all the video frames, then when the variance of the difference value is greater than the preset variance threshold, the video frames with timestamps earlier than the target timestamp among all the video frames are taken as the search frames; when the variance of the difference value is less than or equal to the preset variance threshold, the video frames with timestamps no later than the target timestamp among all the video frames are taken as the search frames.

[0090] The key frame is determined from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched.

[0091] Specifically, when determining keyframes, the computer device can use the video frame corresponding to the maximum similarity fluctuation value as the maximum fluctuation frame and the timestamp of the maximum fluctuation frame as the target timestamp. In one embodiment, since the maximum similarity fluctuation value is the change in similarity between two video frames with adjacent timestamps, the computer device can use the video frame with the later timestamp among the two video frames corresponding to the maximum similarity fluctuation value as the maximum fluctuation frame.

[0092] If the first frame of the control response video is used as the comparison frame, the computer device needs to determine the starting frame of the control response process from each video frame. In this case, the computer device can sort the video frames according to their timestamps, with earlier timestamps having smaller sequence numbers. The computer device can then use the (T-N+1)th to the Tth video frames from the sorted video frames as the target frames. Here, T is the sequence number corresponding to the frame with the largest fluctuation, and N is a pre-set positive integer; in one example, N can be 60.

[0093] After determining N target frames, the computer device can calculate the variance of the image difference values ​​corresponding to these N target frames. This variance reflects the degree of dispersion of the image difference values ​​corresponding to the N target frames. A larger variance indicates that the N image difference values ​​are more dispersed, and the difference fluctuations among the N target frames are greater. Conversely, a smaller variance indicates that the N image difference values ​​are more aggregated, and the difference fluctuations among the N target frames are smaller.

[0094] After obtaining the variance of the difference value, the computer device can compare the variance of the difference value with a preset variance threshold. If the variance of the difference value is less than or equal to the preset variance threshold, it indicates that the difference fluctuation of the N target frames is small, and the frame with the largest fluctuation may be the starting frame of the control response. Therefore, the frame with the largest fluctuation in each video frame and each video frame with a timestamp earlier than the target timestamp can be used as the frame to be searched.

[0095] If the variance of this difference value is greater than a preset variance threshold, it indicates that the difference among the N target frames fluctuates significantly. This maximum similarity fluctuation value is not caused by the control's triggering, but by the interface changes during the control's response process. The starting frame of the control's response has a high probability of preceding this maximum fluctuation frame. Therefore, the computer device can use video frames whose timestamps are earlier than the target timestamp as the frames to be searched.

[0096] After determining the frame to be searched, the computer device can determine a video frame as the starting frame for the control response based on the target timestamp and the difference fluctuation value corresponding to each frame to be searched.

[0097] In this embodiment, N target frames are determined based on the timestamp of the frame with the largest fluctuation, and the variance of the difference values ​​of the image differences corresponding to the N target frames is calculated. This variance allows for the determination of the frame to be searched, facilitating the identification of keyframes from each of the search frames. Thus, keyframes of the control response process can be accurately and quickly determined from each video frame.

[0098] In one embodiment, the step of determining keyframes from each of the video frames based on the maximum similarity fluctuation value and each of the difference fluctuation values ​​includes:

[0099] The video frame corresponding to the maximum similarity fluctuation value is taken as the maximum fluctuation frame, and the timestamp of the maximum fluctuation frame is taken as the target timestamp.

[0100] If the comparison frame is the video frame with the latest timestamp among all the video frames, then based on the frame with the largest fluctuation, N target frames with sequentially adjacent timestamps are determined from all the video frames, and the variance of the difference values ​​of the image difference values ​​corresponding to the N target frames is calculated; wherein, the target timestamp is the earliest timestamp among the N target frames, and N is a preset positive integer;

[0101] If the comparison frame is the video frame with the latest timestamp among all the video frames, then when the variance of the difference value is greater than the preset variance threshold, the video frames with timestamps later than the target timestamp among all the video frames are taken as the search frames; when the variance of the difference value is less than or equal to the preset variance threshold, the video frames with timestamps no earlier than the target timestamp among all the video frames are taken as the search frames.

[0102] The key frame is determined from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched.

[0103] Specifically, if the last frame of the control response video is used as the comparison frame, the computer device needs to determine the end frame of the control response process from each video frame. In this case, the computer device can sort the video frames according to their timestamps, with earlier timestamps having smaller sequence numbers. The computer device can then select the T-th to (T+N-1)-th video frames from the sorted video frames as the target frames. Here, T is the sequence number corresponding to the frame with the largest fluctuation, and N is a pre-set positive integer.

[0104] After determining N target frames, the computer device can calculate the variance of the difference values ​​between the corresponding image difference values ​​of the N target frames. The specific explanation and related principles of the maximum fluctuation frame and the variance of the difference values ​​involved in this embodiment can be found in the previous embodiment, and will not be repeated here.

[0105] If the variance of the difference value is less than or equal to a preset variance threshold, it indicates that the difference fluctuation among the N target frames is small, and the end frame of the control response is likely to be the frame with the largest fluctuation. Therefore, the frame with the largest fluctuation and all video frames with timestamps later than the target timestamp can be used as the search frames. If the variance of the difference value is greater than the preset variance threshold, it indicates that the difference fluctuation among the N target frames is large. This maximum similarity fluctuation value is not caused by the end of the control response, but by the interface changes during the control response process. The end frame of the control response is likely to be later than the frame with the largest fluctuation. Therefore, the computer device can use all video frames with timestamps later than the target timestamp as the search frames.

[0106] After determining the frame to be searched, the computer device can determine a video frame as the end frame of the control response based on the target timestamp and the difference fluctuation value corresponding to each frame to be searched.

[0107] In this embodiment, N target frames are determined based on the timestamp of the frame with the largest fluctuation, and the variance of the difference values ​​of the image differences corresponding to the N target frames is calculated. This variance allows for the determination of the frame to be searched, facilitating the identification of keyframes from each of the search frames. Thus, keyframes of the control response process can be accurately and quickly determined from each video frame.

[0108] In one embodiment, the step of determining the key frame from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched includes:

[0109] The target extreme value frame is determined from each of the frames to be searched, and the target extreme value frame whose timestamp is closest to the target timestamp is taken as the key frame; wherein, the target extreme value frame is the video frame corresponding to the target difference fluctuation value extreme value, and the target difference fluctuation value extreme value is the difference fluctuation value extreme value whose absolute value is greater than a preset fluctuation threshold.

[0110] Specifically, computer equipment can use the frame to be searched that meets the following conditions and whose timestamp is closest to the target timestamp as the keyframe to accurately determine the keyframe:

[0111] (1) The frame to be searched corresponds to the maximum or minimum value of the difference fluctuation value;

[0112] (2) The absolute value of the difference fluctuation value corresponding to the frame to be searched is greater than the preset fluctuation threshold.

[0113] For example, in Figure 3 Among the various similarity fluctuation values ​​shown, the frame with the largest fluctuation is frame 91. When N is 60, since the variance of the difference values ​​corresponding to frames 32 to 91 is greater than the preset variance threshold, the frames to be searched are frames 1 to 90. Figure 5As shown, among frames 1 to 90, the video frame that meets the above conditions and is closest to frame 91 is frame 49, so the key frame is frame 49.

[0114] In one embodiment, the step of amplifying the absolute values ​​of pixel differences that are greater than or equal to a first preset pixel difference threshold includes:

[0115] For each pixel difference absolute value corresponding to the video frame, if the pixel difference absolute value is greater than or equal to the first preset pixel difference threshold, then the magnification factor is determined based on the pixel difference absolute value, and the pixel difference absolute value is magnified based on the magnification factor; wherein, the magnification factor is positively correlated with the pixel difference absolute value.

[0116] Specifically, the computer device performs the following steps for each video frame:

[0117] For each pixel difference absolute value corresponding to the video frame, if the pixel difference absolute value is greater than a first preset pixel difference threshold, a magnification factor can be determined based on the pixel difference absolute value. The larger the pixel difference absolute value, the larger the magnification factor, and the pixel difference absolute value is magnified according to the determined magnification factor. This further increases the sensitivity of the similarity results to changes in image pixels, thereby further improving the accuracy of keyframe finding. In one embodiment, the computer device can use the product of the pixel difference absolute value and the magnification factor as the magnified pixel difference absolute value.

[0118] In one embodiment, the step of determining the magnification factor based on the absolute value of the pixel difference includes:

[0119] If the absolute value of the pixel difference is less than the second preset pixel difference threshold, then the absolute value of the pixel difference is used as the magnification factor; otherwise, the square of the absolute value of the pixel difference is used as the magnification factor; wherein, the second preset pixel difference threshold is greater than the first preset pixel difference threshold.

[0120] It is understood that the specific value of the second preset pixel difference threshold in this application can be determined according to the actual situation. This application does not impose specific restrictions on it, as long as it is greater than the first preset pixel difference threshold. In one example, the second preset pixel difference threshold is 10.

[0121] Assume the first preset pixel difference threshold is thresh1 and the second preset pixel difference threshold is thresh2. If the absolute value of a pixel difference is within the interval [thresh1, thresh2), then the absolute value of the pixel difference is used as the magnification factor. In this case, if the magnification is performed by multiplying the magnification factor by the absolute value of the pixel difference, then the absolute value of the magnified pixel difference is the square of the absolute value of the original pixel difference.

[0122] If the absolute value of a pixel difference falls within the range [thresh2, ∞), then the square of that absolute value is used as the magnification factor. In this case, if magnification is performed by multiplying the magnification factor by the absolute value of the pixel difference, the magnified absolute value of the pixel difference will be the cube of the original absolute value of the pixel difference. This allows for quick determination of the magnification factor for the absolute value of the difference during magnification, further improving recognition efficiency.

[0123] In one embodiment, the step of determining the image difference value between the video frame and the comparison frame based on the absolute values ​​of pixel differences that are less than the first preset pixel difference threshold and the absolute values ​​of pixel differences after magnification includes:

[0124] Calculate the sum of squares of the absolute values ​​of pixel differences that are less than the first preset pixel difference threshold to obtain the first sum of squares;

[0125] Calculate the sum of squares of the absolute values ​​of the differences between each magnified pixel to obtain the second sum of squares;

[0126] The first sum of squares is added to the second sum of squares to obtain the image difference value between the video frame and the comparison frame.

[0127] Specifically, for each video frame, the computer device can selectively amplify the absolute value of the pixel difference as described in the above embodiment, and perform a sum-of-squares processing on the processed absolute values ​​of each pixel difference to obtain the image difference value corresponding to that video frame. In this way, the similarity result between the first target image and the second target image can be quickly calculated, further improving recognition efficiency.

[0128] For example, for a certain video frame, if the video frame corresponds to three absolute pixel difference values, namely the first absolute pixel difference value, the second absolute pixel difference value, and the third absolute pixel difference value, where the first absolute pixel difference value is greater than a first preset pixel difference threshold, and the second and third absolute pixel difference values ​​are both less than the first preset pixel difference threshold, then the computer device can amplify the first absolute pixel difference value and calculate the sum of the squares of the second absolute pixel difference value, the third absolute pixel difference value, and the amplified absolute pixel difference value. This sum of squares is the image difference value corresponding to the video frame.

[0129] In one embodiment, this application also provides a method for determining response time, the method specifically including:

[0130] The keyframe lookup method described in any of the above embodiments is used to determine the starting frame from the control response video;

[0131] The keyframe lookup method described in any of the above embodiments is used to determine the end frame from the control response video;

[0132] The response time of this control is determined based on the frame difference between the start frame and the end frame.

[0133] The keyframe finding apparatus provided in the embodiments of this application is described below. The keyframe finding apparatus described below can be referred to in correspondence with the keyframe finding method described above.

[0134] In one embodiment, this application provides a keyframe lookup device 200. For example... Figure 6 As shown, the device 200 specifically includes a video frame acquisition module 210, a comparison frame acquisition module 220, a similarity fluctuation value determination module 230, a maximum fluctuation value determination module 240, a difference value acquisition module 250, a difference fluctuation value determination module 260, and a key frame determination module 270. Wherein:

[0135] The video frame acquisition module 210 is used to acquire each video frame of the control response video, wherein the control response video is a video obtained by recording the complete control response process;

[0136] The comparison frame acquisition module 220 is used to take the video frame with the earliest timestamp or the video frame with the latest timestamp among each video frame as the comparison frame.

[0137] The similarity fluctuation value determination module 230 is used to calculate, for each pair of video frames with adjacent timestamps, a first similarity value between the video frame with the earlier timestamp and the comparison frame, and a second similarity value between the video frame with the later timestamp and the comparison frame, and obtain the similarity fluctuation value corresponding to the two adjacent timestamps based on the difference between the second similarity value and the first similarity value.

[0138] The maximum fluctuation value determination module 240 is used to take the similarity fluctuation value with the largest absolute value among all the similarity fluctuation values ​​as the maximum similarity fluctuation value;

[0139] The difference value acquisition module 250 is used to calculate the absolute value of the pixel difference between the video frame and the comparison frame at each pixel coordinate point for each video frame, amplify the absolute values ​​of the pixel difference that are greater than or equal to a first preset pixel difference threshold, and calculate the image difference value between the video frame and the comparison frame based on the amplified absolute values ​​of the pixel difference and the absolute values ​​of the pixel difference that are less than the first preset pixel difference threshold.

[0140] The difference fluctuation value determination module 260 is used to determine the difference fluctuation value corresponding to each pair of adjacent video frames based on the difference between the image difference value corresponding to the video frame with the later time stamp and the image difference value corresponding to the video frame with the earlier time stamp.

[0141] The keyframe determination module 270 is used to determine keyframes from each of the video frames based on the maximum similarity fluctuation value and each of the difference fluctuation values; wherein, when the comparison frame is the video frame with the earliest timestamp, the keyframe is the starting frame of the control response process; when the comparison frame is the video frame with the latest timestamp, the keyframe is the ending frame of the control response process.

[0142] In one embodiment, the keyframe determination module 270 includes a target timestamp determination unit, a first target frame determination unit, a first frame to be searched determination unit, and a keyframe determination unit. Wherein:

[0143] The target timestamp determination unit is used to take the video frame corresponding to the maximum similarity fluctuation value as the maximum fluctuation frame and the timestamp of the maximum fluctuation frame as the target timestamp.

[0144] The first target frame determination unit is used to determine N target frames with sequentially adjacent timestamps from each of the video frames based on the maximum fluctuation frame if the comparison frame is the video frame with the earliest timestamp among the video frames, and to calculate the variance of the difference values ​​of the image difference values ​​corresponding to the N target frames; wherein, the target timestamp is the latest timestamp among the N target frames, and N is a preset positive integer.

[0145] The first frame-to-find determination unit is configured to, if the comparison frame is the video frame with the earliest timestamp among all the video frames, then when the variance of the difference value is greater than a preset variance threshold, take the video frames with timestamps earlier than the target timestamp as the frames to be searched; and when the variance of the difference value is less than or equal to the preset variance threshold, take the video frames with timestamps no later than the target timestamp as the frames to be searched.

[0146] The keyframe determination unit is used to determine the keyframe from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched.

[0147] In one embodiment, the keyframe determination module 270 includes a target timestamp determination unit, a second target frame determination unit, a second frame to be searched determination unit, and a keyframe determination unit. Wherein:

[0148] The target timestamp determination unit is used to take the video frame corresponding to the maximum similarity fluctuation value as the maximum fluctuation frame and the timestamp of the maximum fluctuation frame as the target timestamp.

[0149] The second target frame determination unit is used to determine N target frames with sequentially adjacent timestamps from each of the video frames based on the maximum fluctuation frame if the comparison frame is the video frame with the latest timestamp among the video frames, and to calculate the variance of the difference values ​​of the image difference values ​​corresponding to the N target frames; wherein, the target timestamp is the earliest timestamp among the N target frames, and N is a preset positive integer.

[0150] The second frame-to-find determination unit is configured to, if the comparison frame is the video frame with the latest timestamp among all the video frames, then when the variance of the difference value is greater than a preset variance threshold, take the video frames with timestamps later than the target timestamp as the frames to be searched; and when the variance of the difference value is less than or equal to the preset variance threshold, take the video frames with timestamps not earlier than the target timestamp as the frames to be searched.

[0151] The keyframe determination unit is used to determine the keyframe from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched.

[0152] In one embodiment, the keyframe determination unit is configured to include a target extreme value frame determination unit. This target extreme value frame determination unit is configured to determine a target extreme value frame from each of the frames to be searched, and to use the target extreme value frame whose timestamp is closest to the target timestamp as the keyframe; wherein the target extreme value frame is the video frame corresponding to the target difference fluctuation value extreme value, and the target difference fluctuation value extreme value is the difference fluctuation value extreme value whose absolute value is greater than a preset fluctuation threshold.

[0153] In one embodiment, the difference value acquisition module 250 includes an amplification unit. This amplification unit is used to determine a magnification factor based on the absolute value of the pixel difference for each pixel difference corresponding to the video frame, provided that the absolute value of the pixel difference is greater than or equal to the first preset pixel difference threshold, and then amplifies the absolute value of the pixel difference based on the magnification factor; wherein the magnification factor is positively correlated with the absolute value of the pixel difference.

[0154] In one embodiment, the magnification unit includes a magnification factor determination unit. This magnification factor determination unit is configured to use the absolute value of the pixel difference as the magnification factor if the absolute value of the pixel difference is less than a second preset pixel difference threshold; otherwise, it uses the square of the absolute value of the pixel difference as the magnification factor; wherein the second preset pixel difference threshold is greater than the first preset pixel difference threshold.

[0155] In one embodiment, the difference value acquisition module 250 includes a first sum of squares calculation unit, a second sum of squares calculation unit, and a difference value calculation unit. The first sum of squares calculation unit calculates the sum of squares of the absolute values ​​of pixel differences whose values ​​are less than the first preset pixel difference threshold, to obtain a first sum of squares. The second sum of squares calculation unit calculates the sum of squares of the absolute values ​​of the magnified pixel differences, to obtain a second sum of squares. The difference value calculation unit adds the first sum of squares to the second sum of squares to obtain the image difference value between the video frame and the comparison frame.

[0156] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the keyframe lookup method as described in any of the above embodiments.

[0157] In one embodiment, this application also provides a computer device. The computer device stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the keyframe lookup method as described in any of the above embodiments.

[0158] Indicatively, Figure 7 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. In one example, the computer device can be a server. (Refer to...) Figure 7 The computer device 900 includes a processing component 902, which further includes one or more processors, and memory resources represented by memory 901 for storing instructions, such as application programs, that can be executed by the processing component 902. The application programs stored in memory 901 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 902 is configured to execute instructions to perform the steps of the keyframe lookup method described in any of the above embodiments.

[0159] The computer device 900 may also include a power supply component 903 configured to perform power management of the computer device 900, a wired or wireless network interface 904 configured to connect the computer device 900 to a network, and an input / output (I / O) interface 905. The computer device 900 may operate on an operating system stored in memory 901, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0160] Those skilled in the art will understand that the internal structure of the computer device shown in this application is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0161] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the related listed items.

[0162] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0163] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A keyframe lookup method, characterized in that, The method includes: Obtain each video frame of the control response video, wherein the control response video is a video obtained by recording the complete control response process; Use the video frame with the earliest timestamp or the video frame with the latest timestamp as the comparison frame. For each pair of video frames with adjacent timestamps, a first similarity value is calculated between the video frame with the earlier timestamp and the comparison frame, and a second similarity value is calculated between the video frame with the later timestamp and the comparison frame. Based on the difference between the second similarity value and the first similarity value, the similarity fluctuation value corresponding to the two adjacent timestamps is obtained. The similarity fluctuation value with the largest absolute value among all the aforementioned similarity fluctuation values ​​is taken as the maximum similarity fluctuation value; For each video frame, the absolute value of the pixel difference between the video frame and the comparison frame at each pixel coordinate point is calculated. The absolute values ​​of the pixel difference that are greater than or equal to the first preset pixel difference threshold are amplified. Based on the amplified absolute values ​​of the pixel difference and the absolute values ​​of the pixel difference that are less than the first preset pixel difference threshold, the image difference value between the video frame and the comparison frame is calculated. For each pair of video frames with adjacent timestamps, the difference fluctuation value corresponding to the two adjacent timestamps is obtained based on the difference between the image difference value corresponding to the video frame with the later timestamp and the image difference value corresponding to the video frame with the earlier timestamp. Based on the maximum similarity fluctuation value and each of the difference fluctuation values, key frames are determined from each of the video frames; wherein, when the comparison frame is the video frame with the earliest timestamp, the key frame is the starting frame of the control response process; when the comparison frame is the video frame with the latest timestamp, the key frame is the ending frame of the control response process.

2. The method according to claim 1, characterized in that, The step of determining keyframes from each of the video frames based on the maximum similarity fluctuation value and each of the difference fluctuation values ​​includes: The video frame corresponding to the maximum similarity fluctuation value is taken as the maximum fluctuation frame, and the timestamp of the maximum fluctuation frame is taken as the target timestamp. If the comparison frame is the video frame with the earliest timestamp among all the video frames, then based on the frame with the largest fluctuation, N target frames with sequentially adjacent timestamps are determined from all the video frames, and the variance of the difference values ​​of the image difference values ​​corresponding to the N target frames is calculated; wherein, the target timestamp is the latest timestamp among the N target frames, and N is a preset positive integer; If the comparison frame is the video frame with the earliest timestamp among all the video frames, then when the variance of the difference value is greater than the preset variance threshold, the video frames with timestamps earlier than the target timestamp among all the video frames are taken as the search frames; when the variance of the difference value is less than or equal to the preset variance threshold, the video frames with timestamps no later than the target timestamp among all the video frames are taken as the search frames. The key frame is determined from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched.

3. The method according to claim 1, characterized in that, The step of determining keyframes from each of the video frames based on the maximum similarity fluctuation value and each of the difference fluctuation values ​​includes: The video frame corresponding to the maximum similarity fluctuation value is taken as the maximum fluctuation frame, and the timestamp of the maximum fluctuation frame is taken as the target timestamp. If the comparison frame is the video frame with the latest timestamp among all the video frames, then based on the frame with the largest fluctuation, N target frames with sequentially adjacent timestamps are determined from all the video frames, and the variance of the difference values ​​of the image difference values ​​corresponding to the N target frames is calculated; wherein, the target timestamp is the earliest timestamp among the N target frames, and N is a preset positive integer; If the comparison frame is the video frame with the latest timestamp among all the video frames, then when the variance of the difference value is greater than the preset variance threshold, the video frames with timestamps later than the target timestamp among all the video frames are taken as the search frames; when the variance of the difference value is less than or equal to the preset variance threshold, the video frames with timestamps no earlier than the target timestamp among all the video frames are taken as the search frames. The key frame is determined from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched.

4. The method according to claim 2 or 3, characterized in that, The step of determining the key frame from each of the frames to be searched based on the target timestamp and the difference fluctuation value corresponding to each of the frames to be searched includes: The target extreme value frame is determined from each of the frames to be searched, and the target extreme value frame whose timestamp is closest to the target timestamp is taken as the key frame; wherein, the target extreme value frame is the video frame corresponding to the target difference fluctuation value extreme value, and the target difference fluctuation value extreme value is the difference fluctuation value extreme value whose absolute value is greater than a preset fluctuation threshold.

5. The method according to any one of claims 1 to 3, characterized in that, The step of amplifying the absolute values ​​of pixel differences that are greater than or equal to a first preset pixel difference threshold includes: For each pixel difference absolute value corresponding to the video frame, if the pixel difference absolute value is greater than or equal to the first preset pixel difference threshold, then the magnification factor is determined based on the pixel difference absolute value, and the pixel difference absolute value is magnified based on the magnification factor; wherein, the magnification factor is positively correlated with the pixel difference absolute value.

6. The method according to claim 5, characterized in that, The step of determining the magnification factor based on the absolute value of the pixel difference includes: If the absolute value of the pixel difference is less than the second preset pixel difference threshold, then the absolute value of the pixel difference is used as the magnification factor; otherwise, the square of the absolute value of the pixel difference is used as the magnification factor; wherein, the second preset pixel difference threshold is greater than the first preset pixel difference threshold.

7. The method according to any one of claims 1 to 3, characterized in that, The step of determining the image difference value between the video frame and the comparison frame based on the absolute values ​​of pixel differences that are less than the first preset pixel difference threshold and the absolute values ​​of pixel differences after magnification includes: Calculate the sum of squares of the absolute values ​​of pixel differences that are less than the first preset pixel difference threshold to obtain the first sum of squares; Calculate the sum of squares of the absolute values ​​of the differences between each magnified pixel to obtain the second sum of squares; The first sum of squares is added to the second sum of squares to obtain the image difference value between the video frame and the comparison frame.

8. A keyframe lookup device, characterized in that, The device includes: The video frame acquisition module is used to acquire each video frame of the control response video, wherein the control response video is a video obtained by recording the complete control response process; The comparison frame acquisition module is used to select the video frame with the earliest timestamp or the video frame with the latest timestamp from each video frame as the comparison frame. The similarity fluctuation value determination module is used to calculate, for each pair of video frames with adjacent timestamps, a first similarity value between the video frame with the earlier timestamp and the comparison frame, and a second similarity value between the video frame with the later timestamp and the comparison frame, and obtain the similarity fluctuation value corresponding to the two adjacent timestamps based on the difference between the second similarity value and the first similarity value. The maximum fluctuation value determination module is used to determine the maximum similarity fluctuation value as the largest similarity fluctuation value among all the similarity fluctuation values. The difference value acquisition module is used to calculate the absolute value of the pixel difference between the video frame and the comparison frame at each pixel coordinate point for each video frame, amplify the absolute values ​​of the pixel difference that are greater than or equal to a first preset pixel difference threshold, and calculate the image difference value between the video frame and the comparison frame based on the amplified absolute values ​​of the pixel difference and the absolute values ​​of the pixel difference that are less than the first preset pixel difference threshold. The difference fluctuation value determination module is used to determine the difference fluctuation value of each pair of adjacent video frames based on the difference between the image difference value corresponding to the video frame with the later timestamp and the image difference value corresponding to the video frame with the earlier timestamp. The keyframe determination module is used to determine keyframes from each of the video frames based on the maximum similarity fluctuation value and each of the difference fluctuation values; wherein, when the comparison frame is the video frame with the earliest timestamp, the keyframe is the starting frame of the control response process; when the comparison frame is the video frame with the latest timestamp, the keyframe is the ending frame of the control response process.

9. A storage medium, characterized in that, The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the keyframe lookup method as described in any one of claims 1 to 7.

10. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions that, when executed by the one or more processors, perform the steps of the keyframe lookup method as described in any one of claims 1 to 7.

Images