Video frame detection method and device and storage medium

Abstract

The invention discloses a video frame detection method and device and a storage medium, and belongs to the technical field of image processing. The method comprises the steps that a to-be-detected video obtained by shooting a screen when a target terminal executes a camera operation response process is acquired, an operation control will be used when the camera operation response process is started, and a corresponding operation control graph is acquired in advance; and performing region matching in the video frame according to the position of the operation control in the operation control graph to obtain a first preset region of the operation control in the video frame. And detecting whether pixel change occurs in the first preset area in the adjacent video frames, and taking the detected first frame where the operation control begins to change as a start frame. And then pixel change in the video frame after the starting frame or the adjacent video frame is detected, whether a picture appears or whether an image completely appears is judged, so that the video frame which responds to the end is detected, and the video frame is taken as the end frame, the automatic detection process is realized, and the detection efficiency and the detection accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video frame detection method, device and storage medium. Background Art

[0002] In recent years, with the rapid development of the terminal manufacturing industry, users' photo-taking experience on terminals has received more and more attention. In terms of user photo-taking experience, the user's perceived photo-taking process duration is an important factor affecting user experience.

[0003] To evaluate the duration of the photo process, the current common practice is to use a high-speed camera to shoot the screen of the test prototype, record the camera operation response process performed by the test prototype, and obtain a video frame sequence. Then, according to the given starting and ending standards of the key nodes, the video frame sequence is manually judged to filter out the starting frame and ending frame of the photo process from the video frame sequence. Then, the number of shooting frames between the starting frame and the ending frame can be determined, and then the performance parameters such as the duration of the photo process can be determined based on the number of shooting frames.

[0004] Currently, the detection process of the start frame and the end frame requires manual comparison of each video frame. Manual data comparison is often inefficient, and the detection results are subjective, which reduces the efficiency and accuracy of video frame detection. Summary of the invention

[0005] The present application provides a video frame detection method, device and storage medium, which can improve the efficiency and accuracy of video frame detection. The technical solution is as follows:

[0006] In a first aspect, a video frame detection method is provided, which may include: obtaining a video to be detected by shooting a screen of a target terminal when executing a camera operation response process, wherein the video to be detected includes a plurality of continuous video frames. The camera operation response process is a process in which the target terminal operates different shooting functions, and different operation controls are used in the operation process. An operation control diagram corresponding to the camera operation response process may be obtained in advance. The operation control diagram includes operation controls, and different camera operation response processes correspond to different operation controls. The operation control diagram and the video frame including the operation control are images under the same viewing angle, and the position of the operation control in the operation control diagram is the same as its position in the video frame. Based on this, according to the position of the operation control in the operation control diagram, region matching is performed in the video frame to obtain the region corresponding to the operation control in the video frame (i.e., the first preset region). Whether a pixel change occurs in the first preset region in adjacent video frames, that is, whether a relatively obvious difference occurs in the region where the operation control is located in adjacent video frames, thereby detecting the video frame in which the operation control begins to change, and the first frame in which the operation control begins to change is the starting frame corresponding to the camera operation response process. After the start frame is detected, pixel changes in the video frames after the start frame or in adjacent video frames in the video to be detected are detected to determine whether a picture appears or whether an image appears completely.

[0007] In the embodiment of the present application, by matching the area where the operation control is located in the video frame, targeted area detection can be performed, the detection accuracy is improved, and the amount of data when judging whether a pixel has changed is reduced, thereby improving the detection efficiency. By detecting whether a pixel change occurs in a video frame or whether a difference occurs between adjacent video frames, the end frame is detected, and the automatic detection process of the video frame is realized, thereby improving the detection efficiency and accuracy of the video frame.

[0008] Among them, for the video recording scene, the operation control is the video recording control; for the scene of starting the camera application icon to shoot, the operation control is the camera application icon; for the scene of switching between front-facing shooting and rear-facing shooting, the operation control is the flip control.

[0009] In one possible implementation, when detecting whether the operation control begins to change, the area where the operation control is located in the video frame (i.e., the first preset area) can be detected without paying attention to other areas. When the operation control of the target terminal is clicked, the control will first shrink and then return to normal. Detect pixel changes in the first preset area of ​​adjacent video frames in the video to be detected to obtain the first frame where the operation control begins to change, including: using a frame difference method to perform inter-frame differences on the area where the operation control is located in adjacent video frames in the video to be detected to obtain a frame difference between adjacent video frames; when the frame difference is greater than the preset difference value, it indicates that the operation control begins to change, and the first frame whose frame difference is greater than the preset difference value is used as the first frame where the operation control begins to change.

[0010] In an embodiment of the present application, when the frame difference method is used to detect whether there is a large difference between the pixels of adjacent video frames, targeted area detection can be performed, which improves detection accuracy and reduces the amount of data required to determine whether a pixel has changed, thereby improving detection efficiency.

[0011] In a possible implementation, inter-frame difference is performed on a first preset area of ​​adjacent video frames in the video to be detected to obtain a frame difference value, including: for each video frame in the video to be detected, the video frame can be firstly transformed into grayscale to obtain a first grayscale image corresponding to the video frame, and then inter-frame difference is performed on the area where the operation controls of the first grayscale images of adjacent video frames are located to obtain a frame difference value.

[0012] In the embodiment of the present application, the grayscale image can better reflect the difference of pixel changes. Using the grayscale image for inter-frame difference can reduce the amount of calculation and improve the data processing speed. In addition, the grayscale image can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0013] In a possible implementation, after detecting the starting frame in the video to be detected, the computer device then detects the picture in the preview interface of the target terminal. Detecting the pixel changes in the video frame after the starting frame or the adjacent video frame in the video to be detected to obtain the video frame where the response ends, including: extracting edge features of the video frames after the starting frame in the video to be detected in turn, and obtaining the line features of each video frame after the starting frame. The line feature can indicate the number of lines. When the number of lines is greater than a first number, it means that the picture appears in the preview interface and the camera operation response ends. The first frame of the video frame whose line feature indicates that the number of lines is greater than the first number is used as the first frame where the picture appears in the preview interface, and the first frame where the picture appears in the preview interface is used as the end frame.

[0014] In the embodiment of the present application, by comparing the number of lines that constitute the contour, it is determined whether a picture appears in the preview interface without the need for manual detection, thereby realizing an automatic detection process for the end frame and improving the efficiency and accuracy of video frame detection.

[0015] In a possible implementation, the video frame detection method provided by the present application is applied in the test process of the terminal, and the terminal may be operated in a relatively static state. Therefore, the method may also include: pre-acquiring an operation response diagram, which is an interface diagram displayed by the terminal in response to triggering an operation on an operation control. The operation response diagram includes a first object (for example, the first object) and a surrounding background, and the background may be black or white, etc. The operation response diagram and the video frame in which the first object is detected are images under the same viewing angle, and the position of the first object in the operation response diagram is the same as its position in the video frame. According to the positional features of the area where the first object is located in the operation response diagram, template matching is performed in the video frame located after the start frame, the position coordinates of the first object are determined in the video frame, and the area where the first object is located in the video frame (that is, the second preset area) is used as a specific area that needs attention. The edge feature extraction is performed on the video frames located after the start frame in the video to be detected in sequence, and the line features of each video frame located after the start frame are obtained, including: the edge feature extraction is performed on the second preset area of ​​the video frame located after the start frame in the video to be detected in sequence, and the line features of each video frame located after the start frame are obtained.

[0016] In the embodiment of the present application, under normal circumstances, the appearance of the picture in the preview interface is the process of the first object appearing. When detecting whether the first object appears in the preview interface, the area where the first object is located in the video frame (i.e., the second preset area) can be detected without paying attention to the background. By selecting the area corresponding to the first object in the video frame, targeted area detection can be performed, which improves the detection accuracy, reduces the edge feature extraction range, and improves the detection efficiency.

[0017] In a possible implementation, when you need to continue shooting after viewing the taken pictures, you need to click the return control on the gallery interface of the camera application, and then return to the preview interface of the camera application. The camera operation response process includes the camera application jumping from the gallery interface to the preview interface, the operation control is the return control, the first frame in which the return control starts to change is the start frame corresponding to the camera operation response process, and the first frame in which the picture appears in the preview interface is the end frame corresponding to the camera operation response process.

[0018] In a possible implementation, when switching the camera for shooting, it is necessary to click the flip control (also called the flip button) and then enter the preview interface of the front camera or the rear camera. The camera operation response process includes switching from front shooting to rear shooting or from rear shooting to front shooting. The operation control is a flip control. The first frame in which the flip control starts to change is the starting frame corresponding to the camera operation response process. The first frame in which the picture appears in the preview interface is the end frame corresponding to the camera operation response process. In the scenario of switching the camera for shooting, after detecting the starting frame in the video to be detected, the computer device then detects the picture in the preview interface that appears on the target terminal. When the picture in the preview interface appears on the target terminal, the picture in the preview interface gradually changes from clear to blurred and then becomes clear again.

[0019] In an embodiment of the present application, the computer device can realize the recognition of the picture in the preview interface by the edge detection method. The edge feature extraction is performed on the video frame located after the starting frame in the video to be detected to obtain the line feature. The line feature can indicate the number of lines. The process in which the picture in the preview interface gradually changes from clear to blurred and then becomes clear again is reflected in the gradual decrease in the number of lines and then gradually increases. In the process of detecting that the line feature indicates that the number of lines is gradually decreasing, when the number of lines is less than the second number, it means that the picture in the preview interface becomes blurred, and the number of lines is continued to be counted. In the process of detecting that the line feature of multiple video frames indicates that the number of lines is gradually increasing, when the number of lines is greater than the first number, it means that the picture in the preview interface becomes clear again. Therefore, the first frame of the video frame whose number of lines is greater than the first number is used as the first frame that appears in the preview interface, and the first frame that appears in the preview interface is used as the end frame.

[0020] In the embodiment of the present application, the computer device not only detects whether the number of lines is less than the second number to determine the process of the picture becoming blurred, but also detects whether the number of lines is greater than the first number to determine the process of the picture becoming clear again, and uses the cleared video frame as the end frame. The embodiment of the present application detects the process of blurring and becoming clear during the switching process. After detecting that the picture has become blurred, the process of the picture becoming clear is detected again to reduce the situation where the number of lines is greater than the first number due to unsuccessful switching, thereby ensuring successful switching and improving the accuracy of video frame detection. There is no need for manual detection, and the automatic detection process of the end frame is realized, which improves the efficiency and accuracy of video frame detection.

[0021] In a possible implementation, when the camera application is started for shooting, the camera application icon needs to be clicked to enter the preview interface. The camera operation response process includes starting the camera application for shooting, the operation control is the camera application icon, the first frame in which the camera application icon starts to change is the start frame corresponding to the camera operation response process, and the first frame in which the picture appears in the preview interface is the end frame corresponding to the camera operation response process.

[0022] In a possible implementation, the camera operation response process includes starting a camera application for shooting, the operation control is a camera application icon, and the first frame in which the camera application icon begins to change is the starting frame corresponding to the camera operation response process. When the camera application icon of the terminal is clicked, the color changes, and the color detection method can be used to identify whether the camera application icon begins to change. Detecting pixel changes in the first preset area of ​​adjacent video frames in the video to be detected, and obtaining the first frame in which the operation control begins to change, including: extracting color features from the first preset area of ​​the video frame in the video to be detected to obtain color features; determining the color similarity between adjacent video frames based on the color features. When the color similarity is greater than the preset similarity, it indicates that the camera application icon begins to change. The first frame of the video frame whose color similarity is greater than the preset similarity is used as the first frame in which the camera application icon begins to change, and the first frame in which the camera application icon begins to change is the starting frame.

[0023] In an embodiment of the present application, by matching the area where the camera application icon is located in the video frame, targeted area detection can be performed when detecting whether the color of the camera application icon begins to change, thereby improving detection accuracy and reducing the amount of data when judging whether a pixel has changed, thereby improving detection efficiency.

[0024] Furthermore, when detecting whether the camera application icon starts to change, color detection or frame difference method can be used for detection, which increases the diversity and richness of detection methods.

[0025] In a possible implementation, when taking photos using a terminal, it is necessary to trigger the photo control and then obtain a thumbnail, which is a thumbnail of the latest captured image and is displayed in the image echo control. The camera operation response process includes receiving a trigger operation on the photo control to generating a thumbnail corresponding to the latest captured image in the image echo control. The image echo control is used to trigger the display of the latest captured image. The operation control is a photo control. The first frame at which the photo control starts to change is the start frame corresponding to the camera operation response process, and the last frame at which the thumbnail changes is the end frame corresponding to the camera operation response process. The computer device can realize complete picture recognition by a method of picture contour detection. Detecting pixel changes in a video frame or an adjacent video frame after the start frame in the video to be detected to obtain a video frame at the end of the response includes: performing inter-frame difference on adjacent video frames after the start frame in the video to be detected to obtain first frame difference data. The first frame difference data includes the pixel difference of multiple pixels, and the pixel difference of multiple pixels constitutes the picture outline. The area of ​​the picture outline reflects the difference between adjacent video frames. The larger the area of ​​the picture outline, the greater the difference between adjacent video frames, indicating that the thumbnail is being scaled. The smaller the area of ​​the picture outline, the smaller the difference between adjacent video frames, indicating that the thumbnail scaling is completed (that is, the thumbnail stops scaling or the thumbnail appears completely). The video frame whose area of ​​the picture outline indicated by the first frame difference data is less than the first preset area is taken as the last frame of the thumbnail change (that is, the first frame where the thumbnail stops scaling), and the last frame of the thumbnail change is taken as the end frame.

[0026] In the embodiment of the present application, the frame difference variation pattern of adjacent video frames is identified, and whether a large-area difference occurs in the frame difference is judged to determine whether the thumbnail appears completely. No manual detection is required, thereby realizing an automatic detection process for the end frame and improving the video frame detection efficiency and accuracy.

[0027] In a possible implementation, the video frame detection method provided by the present application is applied in the test process of the terminal, and the terminal can be operated in a relatively static state, so the shooting interface image can be acquired in advance. The shooting interface image includes an image echo control, and the shooting interface image and the video frame where the thumbnail is detected are images under the same viewing angle, and the position of the image echo control in the shooting interface image is the same as the position of the thumbnail in the video frame. Based on this, according to the position of the image echo control in the shooting interface image, regional matching is performed in the video frame in the video to be detected, and the area corresponding to the thumbnail in the video frame (i.e., the third preset area) is obtained. For the video frame located after the starting frame in the video to be detected, the video frame can be first converted to grayscale to obtain a second grayscale image corresponding to the video frame, and then the area where the thumbnail of the second grayscale image corresponding to the adjacent video frames is located is subjected to frame difference to obtain the first frame difference data.

[0028] In the embodiment of the present application, by selecting the area corresponding to the thumbnail in the video frame, targeted area detection can be performed, which improves the detection accuracy, reduces the amount of data when judging whether the pixel has changed, and improves the detection efficiency. Using grayscale images for inter-frame difference can reduce the amount of calculation and increase the data processing speed. Moreover, grayscale images can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0029] In one possible implementation, when viewing a picture just taken, you can click on the image echo control (also referred to as the image echo button), and then the enlarged thumbnail is fully displayed in the gallery interface of the camera application (i.e., the thumbnail is fully opened to become a display image). The camera operation response process includes receiving a trigger operation on the image echo control to displaying the display image corresponding to the thumbnail in the image echo control, the operation control is an image echo control, and the image echo control is used to trigger the display of the latest captured image; the first frame in which the image echo control begins to change is the starting frame corresponding to the camera operation response process, and the first frame in which the display image is fully opened is the ending frame corresponding to the camera operation response process. Detecting pixel changes in the video frame or adjacent video frames after the starting frame in the video to be detected to obtain a video frame at the end of the response includes: performing frame difference on adjacent video frames after the starting frame in the video to be detected to obtain second frame difference data. The second frame difference data includes the pixel difference of multiple pixel points, and the pixel difference of multiple pixel points constitutes the picture outline, and the area of ​​the picture outline reflects the difference between adjacent video frames; the larger the area of ​​the picture outline, the larger the difference between adjacent video frames, indicating that the display image is being scaled; the smaller the area of ​​the picture outline, the smaller the difference between adjacent video frames, indicating that the display image is fully scaled (that is, the display image stops scaling or the display image appears completely). The video frame whose area of ​​the picture outline indicated by the second frame difference data is less than the second preset area is taken as the first frame of the display image fully opened (that is, the first frame of the display image stops scaling, or the last frame of the display image change), and the first frame of the display image fully opened is taken as the end frame.

[0030] In the embodiment of the present application, by identifying the variation pattern of the frame difference between adjacent video frames and judging whether a large area difference appears in the frame difference, it is judged whether the display image is completely opened (that is, the display image stops zooming), without the need for manual detection, thereby realizing automatic detection of the end frame and improving the efficiency and accuracy of video frame detection.

[0031] In a possible implementation, when recording a video using a terminal, it is necessary to click on the recording control and then start recording. The camera operation response process includes receiving a trigger operation on the recording control to starting recording, the operation control is a recording control, and the first frame where the recording control starts to change is the starting frame corresponding to the camera operation response process. In the recording scene, after detecting the starting frame in the video to be detected, the recording content will appear next, and the pixel changes in the first preset area of ​​adjacent video frames in the video to be detected are detected. After obtaining the first frame where the operation control starts to change, the method also includes: determining a video frame with a recording duration of a preset duration in the video frames after the starting frame in the video to be detected; and using the first frame of the video frames with a recording duration of the preset duration as the end frame corresponding to the camera operation response process.

[0032] In an embodiment of the present application, by determining the recording duration, the first frame in the video frame with a preset recording duration is used as the end frame, without the need for manual detection, thereby realizing an automatic detection process for the end frame and improving the video frame detection efficiency and detection accuracy.

[0033] In a possible implementation, the video frame detection method provided by the present application is applied in the test process of the terminal, and the process of switching from close view to distant view can be simulated by removing the baffle. The baffle is placed in front of the rear camera of the terminal, and the target component will be displayed after the baffle is completely removed. The camera operation response process includes switching from close-up shooting to distant view shooting, and the first frame in which the baffle is completely removed as the close-up is the starting frame corresponding to the camera operation response process. The method includes: obtaining a baffle map; performing regional matching on the video frame in the video to be detected according to the position of the baffle in the baffle map, and obtaining the baffle area in the video frame, and the baffle area refers to the area where the baffle is located in the video frame. The frame difference method is used to perform inter-frame difference on the baffle area of ​​adjacent video frames in the video to be detected, and obtain the third frame difference data. The third frame difference data includes the pixel difference of multiple pixel points, and the third frame difference data can reflect whether the baffle has moved. The larger the pixel difference, the greater the difference between adjacent video frames, indicating that the baffle has moved, and the smaller the pixel difference, the smaller the difference between adjacent video frames, indicating that the baffle has not moved. When the third frame difference data indicates that the baffle has moved, inter-frame difference is performed on adjacent video frames in the video to be detected to obtain fourth frame difference data, which includes pixel differences of multiple pixels, and the pixel differences of multiple pixels constitute the picture outline. When the outline of the target component is detected, it means that the baffle is completely moved away, and the first frame in which the outline of the target component appears is used as the first frame in which the baffle is completely moved away.

[0034] In the embodiment of the present application, by matching the baffle area in the video frame, when the frame difference method is used to detect whether there is a large difference in the pixels of adjacent video frames, targeted area detection can be performed, thereby improving the detection accuracy, and reducing the amount of data when judging whether the pixels have changed, thereby improving the detection efficiency. By identifying the change rules of the frame difference of adjacent video frames and judging whether there is a large area difference in the frame difference, it is judged whether the distant view has stopped zooming, without the need for manual detection, and the automatic detection process of the end frame is realized, thereby improving the video frame detection efficiency and detection accuracy.

[0035] In a possible implementation, the method further includes: obtaining a component map in advance, performing region matching on the video frame in the video to be detected according to the position of the target component in the component map, and obtaining the component region of the target component in the video frame. When the third frame difference data indicates that the baffle has moved, performing frame difference between adjacent video frames in the video to be detected to obtain fourth frame difference data, including: when the third frame difference data indicates that the baffle has moved, for each video frame in the video to be detected, performing grayscale conversion on the video frame to obtain a grayscale image (i.e., the third grayscale image) corresponding to the video frame. Then, performing frame difference between the component regions of the grayscale images of adjacent video frames to obtain fourth frame difference data.

[0036] In the embodiment of the present application, by matching the area where the target component is located in the video frame, when the frame difference method is used to detect whether the pixels of adjacent video frames are greatly different, targeted area detection can be performed, thereby improving the detection accuracy, and reducing the amount of data when judging whether the pixels have changed, thereby improving the detection efficiency. Using grayscale images for inter-frame difference can reduce the amount of calculation and increase the data processing speed. Moreover, grayscale images can eliminate the interference caused by color differences, reduce the interference of video flicker on detection, and improve the accuracy of data processing results.

[0037] In a possible implementation, the first frame where the distant view stops zooming is the end frame corresponding to the camera operation response process; the pixel changes in the video frame or the adjacent video frame located after the start frame in the video to be detected are detected to obtain the video frame where the response ends, including: performing inter-frame difference on the adjacent video frames located after the start frame in the video to be detected to obtain the fifth frame difference data. The fifth frame difference data includes the pixel difference of multiple pixels, and the pixel difference of multiple pixels constitutes the picture contour, and the area of ​​the picture contour reflects the difference between adjacent video frames; the larger the area of ​​the picture contour, the larger the difference between adjacent video frames, indicating that the distant view is zooming; the smaller the area of ​​the picture contour, the smaller the difference between adjacent video frames, indicating that the distant view is zoomed (that is, the distant view stops zooming). The video frame whose area of ​​the picture contour indicated by the fifth frame difference data is less than the third preset area is taken as the first frame where the distant view stops zooming, and the first frame where the distant view stops zooming is taken as the end frame.

[0038] In the embodiment of the present application, by identifying the change pattern of the frame difference between adjacent video frames and judging whether a large area difference appears in the frame difference, it is judged whether the distant view picture stops zooming, without the need for manual detection, thereby realizing the automatic detection process of the end frame and improving the video frame detection efficiency and detection accuracy.

[0039] In a possible implementation, before performing inter-frame difference on adjacent video frames located after the starting frame in the video to be detected to obtain the fifth frame difference data, the method also includes: pre-acquiring a distant view image, the distant view image and the video frame in which the second object is detected are images under the same viewing angle, and the position of the second object in the distant view image is the same as its position in the video frame. According to the position characteristics of the second object in the distant view image, performing regional matching on the video frames in the video to be detected to obtain a fourth preset area of ​​the second object in the video frame. Performing inter-frame difference on adjacent video frames located after the starting frame in the video to be detected to obtain the fifth frame difference data includes: for the video frames located after the starting frame in the video to be detected, performing grayscale conversion on the video frames to obtain a fourth grayscale image corresponding to the video frames; performing inter-frame difference on the area where the second object is located in the fourth grayscale image corresponding to the adjacent video frames to obtain the fifth frame difference data.

[0040] In the embodiment of the present application, by selecting the area corresponding to the second object in the video frame, targeted area detection can be performed, thereby improving detection accuracy, reducing the amount of data required to determine whether a pixel has changed, and improving detection efficiency. Using a grayscale image for inter-frame difference can reduce the amount of calculation and increase data processing speed, and the grayscale image can eliminate interference caused by color differences, reduce the interference of video flicker on detection, and improve the accuracy of data processing results.

[0041] In a possible implementation, the camera photo delay is the time difference between pressing the shutter and the camera actually taking the picture. When detecting the camera photo delay duration, it is necessary to click the photo control and then obtain the captured picture. The dynamic timer is photographed through the terminal to simulate the photo delay scene, and the captured picture includes the dynamic timer. In the case where the shooting scene includes a dynamic timer, the camera operation response process includes from receiving a trigger operation on the photo control to completing the shooting of the dynamic timer, the operation control is the photo control, and the first frame where the photo control starts to change is the starting frame corresponding to the camera operation response process. After detecting the pixel changes in the first preset area of ​​adjacent video frames in the video to be detected and obtaining the first frame where the operation control starts to change, the method also includes: determining the first time indicated by the dynamic timer in the first frame where the photo control starts to change, and determining the second time indicated by the dynamic timer in the captured picture. Subtracting the two to obtain the photo delay duration.

[0042] In a possible implementation, the dynamic timer includes a dynamically displayed light array, which includes a plurality of lights arranged in rows and columns. The dynamic timer controls the lights to display cyclically according to a preset frequency, and the outline of the lighted lights can be detected by color recognition. Determining the first time indicated by the dynamic timer in the first frame where the photo control starts to change includes: traversing each video frame in the video to be detected, performing color recognition on each video frame, and obtaining the outline of the lighted lights in each video frame; deduplicating the outlines of the lighted lights in multiple video frames of the video to be detected and superimposing them together, so as to construct the outlines of all lights in the dynamic timer and obtain the first coordinate matrix of the dynamic timer. Then, color recognition is performed on the starting frame to obtain the first outline of the lighted lights in the starting frame; according to the position of the first outline in the starting frame, position matching is performed in the first coordinate matrix to obtain the position of the first outline in the first coordinate matrix, and then the time of the dynamic timer corresponding to the starting frame is determined according to the position of the first outline in the first coordinate matrix.

[0043] In the embodiment of the present application, for low-resolution video frames, the entire video to be detected is traversed to identify the light in each video frame, thereby constructing a coordinate matrix of a dynamic timer, and then the outline of the light in the starting frame is identified, and the outline is matched with the coordinate matrix to obtain the row and column where the light in the starting frame is located, thereby obtaining the time corresponding to the starting frame. By identifying the digitized video, the row and column where the light in the starting frame is located are determined, and automatic reading is achieved. There is no need for manual detection, and the automatic detection process of the end frame is realized, which improves the efficiency and accuracy of video frame detection.

[0044] In a possible implementation, determining the second time indicated by the dynamic timer in the captured picture includes: performing shape recognition on the captured picture according to a preset shape to obtain the positions of all non-lit lamps in the captured picture, and constructing a second coordinate matrix according to the positions of all non-lit lamps. Performing color recognition on the captured picture to obtain the second outline of the lit lamp in the picture; matching the second outline of the lit lamp in the second coordinate matrix to obtain the position of the second outline in the second coordinate matrix, and then determining the time of the dynamic timer in the picture according to the position of the second outline in the second coordinate matrix.

[0045] In the embodiment of the present application, for the captured high-definition picture, the position of the non-lit lamp is directly identified, a coordinate matrix is ​​constructed, and then the position of the lit lamp is identified, and the template is matched with the coordinate matrix to obtain the row and column where the lit lamp in the picture is located, thereby obtaining the time corresponding to the captured picture. By identifying the digitized video, the row and column where the lamp in the picture is located are determined, and automatic reading is realized. There is no need for manual detection, and the automatic detection process of the end frame is realized, which improves the efficiency and accuracy of video frame detection.

[0046] In a possible implementation, after determining the first time indicated by the dynamic timer in the first frame where the photo control starts to change, and determining the second time indicated by the dynamic timer in the captured picture, the method further includes: determining the camera photo delay duration of the target terminal according to the first time and the second time. That is, the photo delay duration is obtained by subtracting the two. The simulation process of photo delay is realized, which increases the richness and comprehensiveness of the detection scene.

[0047] In a possible implementation, the camera shooting rate refers to the number of frames that the camera can shoot per second. The camera shooting rate scenario can be simulated by the display changes of the picture in the preview interface during the terminal's shooting of the dynamic timer and the display changes of the dynamic timer. In the case where the shooting scene includes a dynamic timer, the dynamic timer includes a dynamically displayed light array, which includes a plurality of lights arranged in rows and columns, and the camera operation response process includes a process for executing a trigger operation of a photo control, the operation control is a photo control, and the first frame in which the photo control starts to change is the starting frame corresponding to the camera operation response process. After detecting the starting frame in the video to be detected, the computer device then detects the picture in the preview interface of the target terminal, and there are dynamically displayed lights in the preview interface of the target terminal. Detecting pixel changes in the video frame after the start frame or in the adjacent video frame in the video to be detected, and obtaining the video frame at which the response ends, including: detecting the video frame after the start frame in the video to be detected, respectively determining the first frame in which the screen changes from black to bright in the preview interface and the last frame in which the dynamic timer displays a freeze, wherein the position of the light in the video frames before and after the dynamic timer displays a freeze changes from unchanged to changed; and using the first frame in which the screen changes from black to bright and the last frame in which the dynamic timer displays a freeze as the end frame corresponding to the camera operation response process. The camera shooting rate can be determined according to the start frame and the end frame.

[0048] In a possible implementation, the process of brightening the screen of the preview interface is a gradual process, which is brighter step by step. The behavior of the screen changing from black to bright can be identified by the brightness change of the screen from black to bright. The screen changing from black to bright can be identified in the following two ways. In the first way, a black threshold (i.e., the first threshold) is pre-set. If the brightness of the video frame is less than the black threshold, the video frame is judged as black. At the same time, a black-to-bright threshold (i.e., the second threshold) is pre-set. When the current video frame is judged as black, if the brightness difference obtained by subtracting the brightness of the previous video frame from the brightness of the current video frame is greater than the black-to-bright threshold, the current video frame is taken as the first frame changing from black to bright. In the second way, the brightness of multiple video frames in the video to be detected is counted, the brightness of multiple video frames is averaged, and the brightness threshold is determined according to the average value. In the process of the brightness of multiple video frames located after the starting frame gradually increasing, the video frame with a brightness greater than the brightness threshold is taken as the first frame of the screen changing from black to bright. This method can adapt to different lighting conditions, does not rely on a fixed threshold, and increases the applicable scenarios of the detection method.

[0049] In the embodiment of the present application, the brightness of the video frame is judged by the set black threshold, and the brightness difference of adjacent video frames is judged by the set black-to-light threshold, so that the video frame that changes from black to light can be detected in multiple video frames after the start frame. Manual detection is not required, and the automatic detection process of the video frame that changes from black to light is realized, thereby improving the efficiency and accuracy of video frame detection.

[0050] In one possible implementation, the dynamic timer controls the light to display in a cyclic manner at a preset frequency, and the outline of the lighted light can be detected by color recognition. The video frame after the start frame in the video to be detected is detected to determine the first frame in which the screen changes from black to bright in the preview interface and the last frame in which the dynamic timer display is stuck, including: In the embodiment of the present application, color recognition is performed on each video frame after the start frame to obtain the outline of the lighted light in each video frame (i.e., the third outline). When continuous video frames with the same outline are detected, it means that the dynamic timer display is stuck, and the detection is continued until a video frame with an outline different from that of the continuous video frame is detected, and multiple video frames with the same outline are obtained. The video frame with the same outline is used as the target video frame to obtain multiple target video frames, and the outline of the next video frame of the last frame in the multiple target video frames is different from the outline of the multiple target video frames. The last frame in the multiple target video frames is used as the last frame in which the dynamic timer display is stuck.

[0051] In the embodiment of the present application, the outline of the light in the video frame is identified, and then multiple target video frames with the same outline of the light are determined, and the last frame in the multiple target video frames is used as the last frame of the dynamic timer display freeze. Without manual detection, the automatic detection process of the video frame of the dynamic timer display freezes is realized, which improves the video frame detection efficiency and detection accuracy.

[0052] In a possible implementation, after detecting pixel changes in a video frame after a start frame or in an adjacent video frame in a video to be detected and obtaining a video frame at the end of a response, the method further includes: after detecting the start frame and the end frame, subtracting the frame number of the end frame from the frame number of the start frame to obtain a target frame number, and the camera shooting rate can be determined according to the target frame number. The duration of the camera operation response process can also be calculated according to the target frame number and the camera shooting frame rate.

[0053] In a second aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method involved in the first aspect or any one of the implementations of the first aspect is implemented.

[0054] In a third aspect, a computer-readable storage medium is provided, in which instructions are stored, and when the computer-readable storage medium is executed on a computer, the computer executes the method involved in the first aspect or any one of the implementations of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a chip system, which can be applied to an electronic device. The chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method described in the first aspect or any one of the implementations of the first aspect.

[0056] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the video frame inspection method described in the first aspect.

[0057] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0059] Figure 2 is a structural schematic diagram of a computer device provided in an embodiment of the present application;

[0060] Figure 3 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0061] Figure 4 This is a schematic diagram of a scene of starting a camera application for shooting provided in an embodiment of the present application;

[0062] Figure 5 This is a schematic diagram of an edge detection effect provided by an embodiment of the present application;

[0063] Figure 6 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 1 ;

[0064] Figure 7 This is a process of a video frame detection method provided in an embodiment of the present application. Figure 1 ;

[0065] Figure 8 This is a process of a video frame detection method provided in an embodiment of the present application. Figure 2 ;

[0066] Fig. 9 It is a schematic diagram of a scene from taking a photo to a thumbnail provided by an embodiment of the present application;

[0067] Fig.10 This is a schematic diagram of a control behavior detection differential effect provided by an embodiment of the present application;

[0068] Fig.11 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 2 ;

[0069] Fig.12 This is a process of a video frame detection method provided in an embodiment of the present application. Figure 3 ;

[0070] Fig.13 It is a schematic diagram of a scene from a thumbnail to a display image provided by an embodiment of the present application;

[0071] Fig.14 This is a schematic diagram of a complete screen stop zoom detection effect provided by an embodiment of the present application;

[0072] Fig.15 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 3 ;

[0073] Fig.16 This is a process of a video frame detection method provided in an embodiment of the present application. Figure 4 ;

[0074] Fig.17This is a schematic diagram of a scenario of jumping from a gallery interface to a preview interface provided in an embodiment of the present application;

[0075] Fig.18 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 4 ;

[0076] Fig.19 This is a process of a video frame detection method provided in an embodiment of the present application. Figure 5 ;

[0077] Fig. 20 It is a schematic diagram of a scene of switching from rear-view shooting to front-view shooting provided in an embodiment of the present application;

[0078] Fig.21 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 5 ;

[0079] Fig. 22 This is a process of a video frame detection method provided in an embodiment of the present application. Figure 6 ;

[0080] Fig.23 This is a schematic diagram of a video recording scene provided in an embodiment of the present application;

[0081] Fig.24 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 6 ;

[0082] Fig.25 This is a process of a video frame detection method provided in an embodiment of the present application. Figure 7 ;

[0083] Fig.26 This is a schematic diagram of a scene of switching from a near view to a far view provided in an embodiment of the present application;

[0084] Fig. 27 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 7 ;

[0085] Fig.28 This is a process of a video frame detection method provided in an embodiment of the present application. Figure 8 ;

[0086] Fig.29 is a schematic diagram of a dynamic timer provided in an embodiment of the present application;

[0087] Fig.30 It is a schematic diagram of a scene of photographing a dynamic counter provided in an embodiment of the present application;

[0088] Fig.31is a schematic diagram of a shooting auxiliary line provided in an embodiment of the present application;

[0089] Fig.32 is a schematic diagram of the outline of a range of lamps provided in an embodiment of the present application;

[0090] Fig.33 It is a schematic diagram of the position of a light-emitting lamp provided in an embodiment of the present application;

[0091] Fig.34 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 8 ;

[0092] Fig.35 This is a process of a video frame detection method provided in an embodiment of the present application. Figure 9 ;

[0093] Fig.36 is another schematic diagram of a scene of photographing a dynamic counter provided in an embodiment of the present application;

[0094] Fig.37 It is a schematic diagram of a dynamic counter display freeze provided by an embodiment of the present application;

[0095] Fig.38 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 9 ;

[0096] Fig.39 This is a process of a video frame detection method provided in an embodiment of the present application. Figure 10 ;

[0097] Fig.40 is a flow chart of a video frame detection method provided in an embodiment of the present application;

[0098] Fig.41 It is a flow chart of a detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0100] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0101] The embodiments of the present application are applied to a scenario in which a camera operation response process of a test prototype (target terminal) is detected, such as detecting specific video frames (eg, a start frame and an end frame) of the camera operation response process of the target terminal.

[0102] Figure 1 is a schematic diagram of a detection system provided in an embodiment of the present application, such as Figure 1 As shown, the detection system includes a detection device 10, a high-speed camera 20 and a target terminal 30 ( Figure 1 Take a mobile phone as an example). The target terminal 30 can shoot the shooting scene, such as Figure 1 The portrait set on the bracket in the shooting scene shown is photographed so that the portrait appears in the preview interface of the target terminal 30. The high-speed camera 20 is used to shoot the screen of the target terminal 30 to obtain a video to be detected (i.e., a video frame sequence), and the video to be detected includes multiple continuous video frames. The high-speed camera 20 is also used to transmit the captured video to be detected to the detection device 10, and the detection device 10 executes the video frame detection method provided in the embodiment of the present application to detect the start frame and the end frame from the video to be detected. The start frame and the end frame are used to indicate the key nodes where the behavior occurs in the camera operation response process of the target terminal 30. Among them, the end frame can also be called the termination frame.

[0103] Taking the target terminal 30 as a mobile phone as an example, if the camera button of the mobile phone is triggered, the video to be detected is about the process of the mobile phone taking a photo of a person. If the video button of the mobile phone is triggered, the video to be detected is about the process of the mobile phone recording a person.

[0104] The mobile phone as a test prototype can also take a photo of the dynamic timer. The light of the dynamic timer that glows cyclically appears in the preview interface of the mobile phone. The screen of the mobile phone is photographed by the high-speed camera 20 to obtain the video to be tested. If the camera button of the mobile phone is triggered, the video to be tested captured by the high-speed camera 20 is about the process of the mobile phone photographing the dynamic timer.

[0105] Of course, the mobile phone can also execute other camera operation response processes, such as starting the camera application to shoot, thumbnail to display image, camera application jumps from gallery interface to preview interface, front shooting and rear shooting switch each other, close view switch to long view. The high-speed camera 20 shoots the screen of the mobile phone to obtain the video to be detected in different scenes.

[0106] It is understandable that the high-speed camera 20 can also be integrated into the detection device 10, or the detection device 10 has the function of shooting the screen of the mobile phone. The detection device 10 shoots the screen of the target terminal 30 to obtain the video to be detected. This embodiment of the application does not limit this.

[0107] Detecting the start and end frames of the camera operation response process, obtaining the number of frames between the start and end frames to obtain the duration of each node, this test process is called frame counting. The frame counting process refers to analyzing and identifying features in continuous video frames to detect specific frames (also called key frames or key nodes). The frame counting process can be applied to multiple technical fields such as video management, traffic management, robot navigation, virtual reality (VR), etc. It involves real-time processing and understanding of video frame sequences, which is related to the system's response speed and judgment accuracy.

[0108] Currently, specific frames are selected manually from multiple video frames by counting frames, which consumes a lot of manpower and time. In addition, human subjective judgment will also cause deviations in the selection of specific frames, making it difficult to select them according to unified standards. Therefore, how to ensure the accuracy of the selected specific frames is particularly important.

[0109] At present, in the process of manually selecting a specific frame from multiple video frames, the quality of the video data needs to be manually controlled. Data quality control is usually divided into the following stages. First, before manually selecting a specific frame, the original video data needs to be manually verified to find possible errors and correct them in time. The verification process includes checking whether the original video data is complete, whether the format is correct, and whether there is any damage, so as to ensure the accuracy and integrity of the original video data and facilitate the subsequent selection of specific frames. Secondly, in order to effectively select specific frames, it is necessary to equip sufficient equipment and personnel, and ensure the quality of the equipment and the professional quality of the personnel. The equipment needs to be installed with high-quality video players or editing software so that video frames can be accurately browsed and selected. At the same time, in order to better judge and select specific frames, the equipped personnel need to have relevant professional knowledge and rich experience. In addition, it is also necessary to standardize the selection criteria to ensure the clarity and consistency of the operation steps for selecting specific frames and improve the accuracy and efficiency of the selection results. Then, after selecting a specific frame, manual data inspection is required. The data inspection process includes manual inspection of the selected specific frames to ensure that they are consistent with the required video frames in the original video data; through statistical analysis and sampling inspection, to evaluate whether the selected specific frames meet the expected requirements, so as to further improve the quality and accuracy of the selected specific frames and ensure that the manually selected specific frames can meet the needs of subsequent analysis and application.

[0110] Manual selection of specific frames from a large amount of video data will inevitably introduce errors, and manually checking the consistency of the selection results is also a time-consuming and laborious task. In this process, the operator's experience level, emotional state, noise in the selection environment, and the arrangement of the selection work will all affect the consistency judgment of the selection results.

[0111] Traditional image processing technology is used to analyze static images, but has great limitations in processing continuous dynamic images (i.e., continuous video frames). Since the position, shape, and external environment of target objects in dynamic scenes change over time, the algorithm is required to comprehensively consider the continuity and variability in the time dimension while analyzing a single frame. Therefore, traditional image processing technology cannot be directly applied to the selection process of a specific frame. It is urgent to provide an effective frame counting process to detect the key start and end frames by identifying and analyzing the key node features of the video data.

[0112] In view of the above problems, the present application provides a method for automatically detecting specific video frames in camera videos, which realizes automatic detection based on template matching and edge detection, obtains the operation control diagram of the target terminal when executing different camera operation response processes in advance, and provides configuration parameters for the judgment conditions of detecting the start frame and the end frame under different camera operation response processes in advance. The template matching method is used to match the area where the operation control is located in the video frame, detect whether the pixels in the area where the operation control is located have changed, and select the video frame whose pixel change meets the configuration parameters. Through edge detection, the video frame whose edge meets the configuration parameters can be selected in the video frame, that is, to determine whether the picture appears in the preview interface, or the frame difference method can be used to select the video frame whose pixel change meets the configuration parameters in the video frame, that is, to determine whether the image appears completely or stops zooming, so as to automatically select a specific frame in the continuous video frame. By adopting this detection method, manual operation can be reduced, detection efficiency can be improved, and the subjective influence of manual factors on the selection results can be reduced, and the detection accuracy and consistency can be improved. This detection method can quickly and accurately identify specific frames in the video to be detected, and ensure that they meet the standard requirements, providing a reliable data basis for subsequent analysis and application.

[0113] Before explaining in detail the video frame detection method provided in the embodiment of the present application, the computer device and terminal involved in the embodiment of the present application are first described respectively.

[0114] Figure 2 is a schematic diagram of a computer device provided in an embodiment of the present application, and the computer device can be the above Figure 1 Detection equipment 10, see Figure 2 The computer device includes at least one processor 201 , a communication bus 202 , a memory 203 and at least one communication interface 204 .

[0115] Processor 201 may be a microprocessor (including a central processing unit (CPU) etc.), an application-specific integrated circuit (ASIC), or may be one or more integrated circuits for controlling the execution of the program of the present application.

[0116] The communication bus 202 may include a pathway for transmitting information between the above-mentioned components.

[0117] The memory 203 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 203 may exist independently and be connected to the processor 201 via the communication bus 202. The memory 203 may also be integrated with the processor 201.

[0118] The communication interface 204 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0119] In a specific implementation, as an embodiment, the processor 201 may include one or more CPUs, such as Figure 2 CPU0 and CPU1 are shown in the figure.

[0120] In a specific implementation, as an embodiment, a computer device may include multiple processors, such as Figure 2 201 and processor 205 shown in FIG. Each of these processors may be a single-core processor or a multi-core processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0121] In a specific implementation, as an embodiment, the computer device may further include an output device 206 and an input device 207. The output device 206 communicates with the processor 201 and may display information in a variety of ways. For example, the output device 206 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 207 communicates with the processor 201 and may receive user input in a variety of ways. For example, the input device 207 may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0122] The above-mentioned computer device can be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device can be a desktop computer, a portable computer, a network server, a handheld computer, a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer device.

[0123] The memory 203 is used to store the program code 210 for executing the solution of the present application, and the processor 201 is used to execute the program code 210 stored in the memory 203. The computer device can implement the video frame detection method provided in the embodiment of the present application through the processor 201 and the program code 210 in the memory 203.

[0124] Figure 3 is a schematic diagram of a terminal structure provided in an embodiment of the present application. The terminal 100 may be the above-mentioned Figure 1 The target terminal 30 in Figure 3The terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0125] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0126] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0127] The controller may be the nerve center and command center of the terminal 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0128] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0129] The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0130] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0131] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the terminal 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0132] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0133] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0134] In some embodiments, antenna 1 of terminal 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that terminal 100 can communicate with the network and other devices through wireless communication technology.

[0135] The terminal 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0136] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal 100 may include 1 or N display screens 194, where N is an integer greater than 1.

[0137] The terminal 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

[0138] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0139] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the terminal 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0140] The terminal 100 can implement audio functions, such as music playing and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D and the application processor.

[0141] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the terminal 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than a pressure threshold acts on a short message application icon, an instruction to view a short message is executed. When a touch operation with a touch operation intensity greater than or equal to the pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.

[0142] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be arranged on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be arranged on the surface of the terminal 100, which is different from the position of the display screen 194.

[0143] In the embodiment of the present application, the camera operation response process includes at least the following scenarios: scenario 1, starting the camera application to shoot; scenario 2, from receiving the trigger operation of the photo control to generating the thumbnail corresponding to the latest shot image in the image echo control (shot2see); scenario 3, from receiving the trigger operation of the image echo control to displaying the display image corresponding to the thumbnail in the image echo control (see2review); scenario 4, the camera application jumps from the gallery interface to the preview interface (Gallery2Preview); scenario 5, switching between front shooting and rear shooting; scenario 6, from receiving the trigger operation of the video control to starting the video recording; scenario 7, switching from close-up shooting to long-range shooting; scenario 8, from receiving the trigger operation of the photo control to completing the dynamic timer shooting (shutterlag); scenario 9, executing the process in response to the trigger operation of the photo control (shot2Preview). The following takes the target terminal as a mobile phone as an example to explain the above scenarios one by one.

[0144] Below through Figure 4-Figure 8 Scenario 1 (scenario of starting a camera application to take a picture) is described.

[0145] When the camera application is started for shooting, the mobile phone triggers the camera application icon (for example, clicks) based on the test instruction to realize the automatic triggering operation of the camera application icon. The test instruction can be initiated by the user or sent to the mobile phone by other devices, and the embodiment of the present application is not limited to this. In response to the triggering operation of the camera application icon, the mobile phone starts the camera application and displays the picture in the preview interface. The user-perceivable process duration refers to the time from the camera application icon starting to change to the appearance of the outline in the preview interface. In this process, the camera application icon starts to change and the outline appears in the preview interface as a key node. Based on this, the operation response process of the mobile phone to start the camera application for shooting can be shot by a high-speed camera to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0146] Take the example of using a mobile phone to shoot a still portrait pasted on the wall. Figure 4 As shown, Figure 4 Four representative video frames in the video to be detected corresponding to the scene of starting the camera application for shooting are shown. It can be understood that the video to be detected also includes other video frames. The process of starting the camera application for shooting on a mobile phone is as follows: using an automated method, click the camera application icon on the display interface of the mobile phone, such as Figure 4 As shown in Figure A in the figure, the camera application icon changes color and shrinks. Figure 4 As shown in Figure B, then Figure 4 As shown in Figure C, the camera application icon returns to normal; then the phone will display the screen in the preview interface. When the screen appears in the preview interface, it will go through the following steps: Figure 4 The slow emergence process shown in Figure C until Figure 4 Figure D in .

[0147] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (indicating that the camera application icon begins to change) and the end frame (indicating the first frame that the picture appears in the preview interface). The camera operation response process involves camera application icon recognition and picture recognition in the preview interface. The changes in the camera application icon can be identified by the frame difference method or the color detection method; when the picture appears in the preview interface, characteristic content (for example, white lines, etc.) will appear in the picture. The white lines can be identified by the edge detection operator extraction method, thereby detecting whether the picture appears in the preview interface. The edge detection operator includes but is not limited to the Canny operator, Laplacian operator, Sobel operator, Prewitt operator, Roberts operator, Marr-Hildreth operator, etc.

[0148] The video frame detection method provided by the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, an operation control diagram can be obtained in advance. The operation control diagram includes a camera application icon. The camera application icon is an operation control, such as Figure 4 As shown in Figure A in , the computer device pre-acquires an operation control diagram with a camera application icon. The operation control diagram and the video frame including the camera application icon are images under the same viewing angle, and the position of the camera application icon in the operation control diagram is the same as its position in the video frame. When detecting whether the camera application icon begins to change, the area where the camera application icon is located in the video frame (i.e., the first preset area) can be detected without paying attention to other areas. Based on the positional features of the area where the camera application icon is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the camera application icon in the video frame, and the area where the camera application icon is located in the video frame is treated as a specific area that needs attention.

[0149] When the phone's camera app icon is clicked, it changes color, shrinks, and then returns to normal. Figure 4 As shown in Figure B, Figure 4 Compared to Figure A in the figure, the camera app icon is smaller and grayed out, and then Figure 4 As shown in Figure C in the figure, the camera application icon returns to normal. Based on this, color detection or frame difference method can be used to detect whether the camera application icon starts to change. The following are respectively explained.

[0150] The color representation mode can use the RGB color space or the HSV color space, where RGB indicates red, green and blue, and HSV indicates hue, saturation and value. The color feature is obtained by extracting the color feature of the area where the camera application icon of the video frame in the video to be detected is located. The color feature can be expressed as a color histogram, a color matrix, a color set, a color aggregation vector or a color correlation graph. When judging whether the color change occurs between adjacent video frames, it can be achieved by comparing the similarity relationship between the color features of adjacent video frames. For example, the color similarity between the color features of adjacent video frames is calculated. When the color similarity is greater than the preset similarity, it indicates that the camera application icon begins to change. The first frame of the video frame whose color similarity is greater than the preset similarity is used as the first frame where the camera application icon begins to change, and the first frame where the camera application icon begins to change is used as the starting frame.

[0151] The frame difference method (i.e., the inter-frame difference method) is to subtract the pixel values ​​of the corresponding pixel points of the frames to obtain the pixel difference between the frames, and average the pixel differences of multiple pixel points in the frame as the frame difference. The frame difference method can also be to first average the pixel values ​​of multiple pixel points included in the frame to obtain the brightness of the frame, and then subtract the brightness of the frame from the frame as the frame difference. In the present application, the computer device uses the frame difference method to perform inter-frame difference on the area where the camera application icon of adjacent video frames in the video to be detected to obtain the frame difference. When the frame difference is greater than the preset difference value, it means that there is a more obvious difference between the adjacent video frames, that is, the camera application icon begins to change. The first frame whose frame difference is greater than the preset difference value is used as the first frame where the camera application icon begins to change, and the first frame where the camera application icon begins to change is used as the starting frame.

[0152] When using the frame difference method to perform inter-frame difference on adjacent video frames, the inter-frame difference can be performed on the grayscale images of adjacent video frames, and the grayscale image can better reflect the difference in pixel changes. Based on this, for each video frame in the video to be detected, the video frame can be first gray-scale transformed to obtain the grayscale image corresponding to the video frame, and then the inter-frame difference is performed on the area where the camera application icon is located in the grayscale image of the adjacent video frames to obtain the frame difference value.

[0153] Using grayscale images for inter-frame difference can reduce the amount of calculation and improve the data processing speed. Grayscale images can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results. Pay attention to the changes in the light and dark of the camera application icon and enhance the contrast of the area where the camera application icon is located, so that the camera application icon is easier to detect when it changes.

[0154] Among them, the preset similarity and preset difference value are configuration parameters set by technical personnel in this field based on the similarity threshold and difference threshold adopted in a large number of experimental processes. As long as it can be determined whether the camera application icon starts to change, the embodiment of the present application is not limited to this.

[0155] In addition, the preset difference value may also be positively correlated with the degree of flicker of the camera in the mobile phone. The degree of flicker of the camera refers to the change in the brightness of the flash, that is, the intensity of the light emitted by the flash when taking a photo. The brighter the flash is (i.e., the camera is in high-brightness mode), the brighter the captured image is, and the preset difference value for judging whether the camera application icon begins to change should be set larger. The smaller the flash is (i.e., the camera is in low-brightness mode), the softer or dimmer the captured image is, and the preset difference value for judging whether the camera application icon begins to change should be set smaller. Setting the preset difference value according to the brightness of the camera improves the flexibility of the judgment conditions and further improves the accuracy of the video frame detection results.

[0156] In the embodiment of the present application, by matching the area where the camera application icon is located in the video frame, when detecting whether the color of the camera application icon begins to change, or using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, targeted area detection can be performed, thereby improving the detection accuracy, reducing the amount of data when judging whether the pixels have changed, and improving the detection efficiency. Furthermore, when detecting whether the camera application icon begins to change, color detection or frame difference method can be used for detection, increasing the diversity and richness of detection methods.

[0157] In the scenario where the camera application is started to shoot, after detecting the start frame in the video to be detected, the computer device then detects the picture in the preview interface of the mobile phone. When the picture in the preview interface appears, the mobile phone will go through the following steps: Figure 4 The slow emergence process shown in Figure C until Figure 4 Figure D in the figure. In the present application, the computer device can realize the recognition of the picture in the preview interface by the edge detection method. The edge feature of the video frame located after the start frame in the video to be detected is extracted to obtain the line feature. The line feature can indicate the number of lines. When the number of lines is greater than the first number, it means that the picture appears in the preview interface and the camera operation response ends. The first frame of the video frame whose line feature indicates that the number of lines is greater than the first number is used as the first frame of the picture appearing in the preview interface, and the first frame of the picture appearing in the preview interface is used as the end frame. By comparing the number of lines that constitute the contour, it is determined whether the picture appears in the preview interface, without the need for manual detection, and the automatic detection process of the end frame is realized, thereby improving the efficiency and accuracy of video frame detection.

[0158] The image recognition detection effect in the preview interface is as follows Figure 5As shown, Figure 5 This is merely an exemplary description of edge detection. Figure 5 Figure A and Figure 5 The B picture in is the adjacent video frame. Figure 5 The preview interface in Figure A shows that there are no detected lines, that is, no picture outline appears. Figure 5 The preview interface shown in Figure B detects lines, that is, the outline of the screen appears. The outline of the screen is a white line. Therefore, Figure 5 Figure B is the key node, and in practical applications, it is used as the end frame.

[0159] Among them, the first number is a configuration parameter set by technical personnel in this field based on a number threshold adopted in a large number of experiments. As long as it can be determined whether a picture appears in the preview interface, this embodiment of the present application is not limited to this.

[0160] The video frame detection method provided by the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, an operation response diagram can be obtained in advance. The operation response diagram is an interface diagram displayed by the mobile phone in response to a trigger operation (for example, a click operation) on an operation control. Figure 4 The D graph in the operation response graph. The operation response graph includes a still portrait (i.e., the first object) and the surrounding background, and the background can be black or white, etc. The operation response graph and the video frame in which the still portrait is detected are images under the same viewing angle, and the position of the still portrait in the operation response graph is the same as its position in the video frame. When detecting whether a still portrait appears in the preview interface, the area where the still portrait is located in the video frame (i.e., the second preset area) can be detected without paying attention to other areas. According to the positional features of the area where the still portrait is located in the operation response graph, template matching is performed in the video frame after the starting frame, the position coordinates of the still portrait are determined in the video frame, and the area where the still portrait is located in the video frame is used as a specific area that needs to be paid attention to. Then, edge feature extraction is performed on the second preset area of ​​the video frame after the starting frame to obtain the line features of the video frame after the starting frame.

[0161] In the embodiments of the present application, under normal circumstances, the appearance of the picture in the preview interface is the process of the appearance of the still portrait, and there is no need to pay attention to the background. By selecting the area corresponding to the still portrait in the video frame, targeted area detection can be performed, thereby improving the detection accuracy, reducing the edge feature extraction range, and improving the detection efficiency.

[0162] In the embodiment of the present application, the duration of the camera operation response process includes the camera application startup duration. After detecting the start frame and the end frame, the two are subtracted to obtain the shooting frame number, and the camera application startup duration is calculated according to the shooting frame number and the camera shooting frame rate.

[0163] The above scenario 1 involves the interaction between mobile phones, high-speed cameras and computer equipment, such as Figure 6 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the present application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Figure 6 ,The interaction process includes the following steps.

[0164] S11. A high-speed camera shoots the screen of the mobile phone.

[0165] Aim the high-speed camera at the screen of the mobile phone, then start the high-speed camera. In response to the start operation, the high-speed camera starts to shoot the screen of the mobile phone.

[0166] S12. Trigger the phone’s camera app icon.

[0167] S13. In response to the triggering operation, the mobile phone starts the camera application and displays the picture in the preview interface.

[0168] S14. The high-speed camera sends the captured video to be detected to the computer device.

[0169] S15. The computer device identifies whether the camera application icon starts to change in the video to be detected through a frame difference method or a color detection method, takes the first frame where the camera application icon starts to change as the starting frame, and identifies the picture in the preview interface, and takes the first frame where the picture appears in the preview interface as the ending frame.

[0170] As an example, the video frame detection method performed by the computer device in scenario 1 is as follows: Figure 7 Please refer to Figure 7 , the method comprises the following steps:

[0171] S151. Obtain the video to be detected and the corresponding operation control diagram and operation response diagram.

[0172] The operation control diagram includes the camera application icon, such as Figure 4 The operation response diagram is an interface diagram displayed by the mobile phone in response to a trigger operation on the operation control. The operation response diagram includes a still portrait (ie, the first object), such as Figure 4 As shown in Figure D.

[0173] S152. According to the position of the camera application icon in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the camera application icon in the video frame.

[0174] Based on the positional features of the area where the camera application icon is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the camera application icon in the video frame, and the area where the camera application icon is located in the video frame (i.e., the first preset area) is taken as a specific area that needs attention.

[0175] S153. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0176] S154. Perform inter-frame difference on a first preset area of ​​a first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0177] S155. Determine whether the frame difference is greater than a preset difference value?

[0178] If yes, execute S156, if no, execute S152.

[0179] S156. The first frame whose frame difference is greater than the preset difference value is used as the first frame at which the camera application icon starts to change.

[0180] S157. According to the position of the first object in the operation response graph, perform region matching on the video frame after the start frame in the video to be detected to obtain a second preset region of the first object in the video frame.

[0181] According to the position features of the area where the first object is located in the operation response graph, template matching is performed in the video frame located after the start frame to determine the position coordinates of the first object in the video frame, and the area where the first object is located in the video frame (i.e., the second preset area) is taken as a specific area that needs attention.

[0182] S158. Extract edge features from second preset areas of the video frames after the start frame in the video to be detected, and obtain line features of each video frame after the start frame.

[0183] S159. Determine whether the number of lines indicated by the line feature is greater than the first number?

[0184] If yes, execute S161, if no, execute S157.

[0185] S161. The first frame of the video frame in which the number of line feature indication lines is greater than the first number is used as the first frame of the picture appearing in the preview interface.

[0186] As an example, the video frame detection method performed by the computer device in scenario 1 can also be as follows: Figure 8 Please refer to Figure 8 , the method comprises the following steps:

[0187] S151. Obtain the video to be detected and the corresponding operation control diagram and operation response diagram.

[0188] S152. According to the position of the camera application icon in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the camera application icon in the video frame.

[0189] S162. Perform color extraction on a first preset area of ​​a video frame in the video to be detected to obtain a color feature.

[0190] S163. Determine the color similarity between adjacent video frames based on the color features.

[0191] S164. Determine whether the color similarity is greater than a preset similarity?

[0192] If yes, execute S164, if no, execute S152.

[0193] S165. Using the first frame of the video frame whose color similarity is greater than the preset similarity as the first frame at which the camera application icon starts to change.

[0194] S157. According to the position of the first object in the operation response graph, perform region matching on the video frame after the start frame in the video to be detected to obtain a second preset region of the first object in the video frame.

[0195] S158. Extract edge features from second preset areas of the video frames after the start frame in the video to be detected, and obtain line features of each video frame after the start frame.

[0196] S159. Determine whether the number of lines indicated by the line feature is greater than the first number?

[0197] If yes, execute S161, if no, execute S157.

[0198] S161. The first frame of the video frame in which the number of line feature indication lines is greater than the first number is used as the first frame of the picture appearing in the preview interface.

[0199] In this application Figure 6-Figure 8 This section introduces the scenario of launching the camera application to take pictures. The implementation process and the beneficial effects achieved can be found in the above Figure 4 and Figure 5 The description will not be repeated here.

[0200] Below through Figure 9-12 Scenario 2 (the scenario from receiving a trigger operation on a photo control to generating a thumbnail corresponding to the latest captured image in the image echo control) is described.

[0201] When taking photos with a mobile phone, the mobile phone triggers the photo control (also called the photo button) in the camera application interface based on the test instruction (for example, clicks) to realize the automatic triggering operation of the photo control. In response to the triggering operation of the photo control, the mobile phone performs the shooting operation and displays the thumbnail of the latest shot image in the image echo control of the camera application interface. The user-perceivable process duration refers to the time from the beginning of the change of the photo control to the complete appearance of the thumbnail in the image echo control. In this process, the beginning of the change of the photo control and the complete appearance of the thumbnail in the image echo control are key nodes. Based on this, the operation response process of the mobile phone performing the photo shooting can be filmed by a high-speed camera to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0202] like Fig. 9 As shown, Fig. 9 Four video frames in the photo shooting scene are shown. It is understandable that the video to be detected in this scene also includes other video frames. The process of triggering the photo shooting control to shoot is as follows: using an automated method to click the photo shooting control on the shooting interface of the camera application, such as Fig. 9 As shown in Figure A; the camera control is reduced, as shown in Fig. 9 As shown in Figure B, then Fig. 9 As shown in Figure C, the camera controls return to normal; then the phone will display thumbnails, which will go through the following steps: Fig. 9 The zooming process shown in Figure C in the figure is repeated until the thumbnail zooming is completed. Fig. 9 Figure D in .

[0203] The video frame detection method provided in the present application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the photo control starts to change) and the end frame (the last frame indicating that the thumbnail appears completely). The camera operation response process involves photo control recognition and complete picture recognition. The changes in the photo control can be identified by the frame difference method; the complete picture recognition can be achieved by finding the frame difference. When the picture is zoomed, the frame difference is large, and the frame difference is small after the zoom is completed. Through the change law of the frame difference amplitude, it is recognized that the picture display is complete and the zooming has stopped. In other words, when a large area difference appears in the frame difference, it means that the picture is zooming, and when a large area difference stops appearing in the frame difference, it means that the picture zooming is complete, and the last zoomed frame is the frame where the picture stops zooming. The following describes the photo control recognition and complete picture recognition.

[0204] The video frame detection method provided by the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, an operation control diagram can be obtained in advance. The operation control diagram includes a camera control. Fig. 9As shown in Figure A in the figure, the computer device pre-acquires an operation control diagram with a photo control. The operation control diagram and the video frame including the photo control are images under the same viewing angle, and the position of the photo control in the operation control diagram is the same as its position in the video frame. When detecting whether the photo control begins to change, the area where the photo control is located in the video frame (i.e., the first preset area) can be detected without paying attention to other areas. Based on the positional features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the photo control are determined in the video frame, and the area where the photo control is located in the video frame is used as a specific area that needs to be paid attention to.

[0205] When the camera control of the mobile phone is triggered, the control will shrink first and then return to normal. Fig. 9 As shown in Figure B, Fig. 9 Compared with Figure A in the figure, the camera control is reduced, and then Fig. 9 As shown in Figure C in the figure, the camera control returns to normal. Based on this, the present application uses the frame difference method to perform inter-frame differences on the area where the camera control of adjacent video frames in the video to be detected is located to obtain a frame difference value. When the frame difference value is greater than the preset difference value, it indicates that there is a relatively obvious difference between adjacent video frames, that is, the camera control begins to change. The first frame whose frame difference value is greater than the preset difference value is taken as the first frame where the camera control begins to change, and the first frame where the camera control begins to change is taken as the starting frame.

[0206] When using the frame difference method to perform inter-frame difference on adjacent video frames, the video frames can be first converted to grayscale and then inter-frame difference can be performed. Using grayscale images for inter-frame difference can reduce the amount of calculation and increase the data processing speed. Moreover, grayscale images can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0207] Among them, the recognition process of the photo control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0208] The difference effect of control behavior detection is as follows Fig.10 As shown, Fig.10 It is just an example of the camera controls for a certain phone. Fig.10 Indicates the frame difference between the current video frame and the previous video frame after inter-frame difference, represented by Fig.10 It can be seen that the camera control starts to change in the current video frame. In the present application, the computer device determines whether the camera control starts to change by determining whether the frame difference is greater than a preset difference value. Fig.10 It is a key node and is used as the starting frame in practical applications.

[0209] In an embodiment of the present application, by matching the area where the photo control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving detection accuracy and reducing the amount of data required to determine whether the pixels have changed, thereby improving detection efficiency.

[0210] In the scene from taking a photo to thumbnail, after detecting the starting frame in the video to be detected, the mobile phone will display a thumbnail. When the thumbnail appears, it can be generated by sliding in animation or zooming animation. The video frame detection method provided by this application can be applied to a variety of thumbnail generation schemes. This example uses the generation of thumbnails in the form of zooming animation as an example. When an incorrect thumbnail appears, it can be generated by sliding in animation or zooming animation. Fig. 9 The zooming process shown in Figure C in the figure is repeated until the thumbnail zooming is completed. Fig. 9 Figure D in the figure. In the present application, for the process of displaying thumbnails in the form of slide-in animation and zoom animation, the computer device can realize complete picture recognition by the method of picture contour detection. The adjacent video frames located after the start frame in the video to be detected are inter-frame differentiated to obtain the first frame difference data. The first frame difference data includes the pixel difference of multiple pixels, and the pixel difference of multiple pixels constitutes the picture contour. The area of ​​the picture contour reflects the difference between adjacent video frames; the larger the area of ​​the picture contour, the larger the difference between adjacent video frames, indicating that the thumbnail is being scaled; the smaller the area of ​​the picture contour, the smaller the difference between adjacent video frames, indicating that the thumbnail scaling is completed (that is, the thumbnail stops scaling or the thumbnail appears completely). The video frame whose area of ​​the picture contour indicated by the first frame difference data is less than the first preset area is used as the last frame of the thumbnail change (that is, the first frame where the thumbnail stops scaling), and the last frame of the thumbnail change is used as the end frame.

[0211] In an embodiment of the present application, the frame difference variation pattern of adjacent video frames is identified, and whether a large-area difference appears in the frame difference is determined to determine whether the thumbnail appears completely (i.e., the zooming stops). No manual detection is required, thereby realizing an automatic detection process for the end frame and improving the video frame detection efficiency and accuracy.

[0212] Among them, the first preset area is a configuration parameter set by technical personnel in this field based on an area threshold adopted in a large number of experiments. As long as it can be determined whether the thumbnail appears completely or stops zooming, the embodiment of the present application is not limited to this.

[0213] The video frame detection method provided in the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, a shooting interface image can be obtained in advance, such as Fig. 9 Figure C and Fig. 9As shown in Figure D in the figure. The shooting interface diagram includes an image echo control, and the shooting interface diagram and the video frame in which the thumbnail is detected are images under the same viewing angle, and the position of the image echo control in the shooting interface diagram is the same as the position of the thumbnail in the video frame. When detecting whether the thumbnail appears completely, the area where the thumbnail is located in the video frame (i.e., the third preset area) can be detected without paying attention to other areas. According to the positional characteristics of the area where the image echo control is located in the shooting interface diagram, template matching is performed in the video frame, the position coordinates of the thumbnail are determined in the video frame, and the area where the thumbnail is located in the video frame is used as a specific area that needs attention. Then, the area where the thumbnail of the adjacent video frames located after the starting frame in the video to be detected is subjected to inter-frame difference, and the first frame difference data is obtained, and the video frame whose area of ​​the picture contour indicated by the first frame difference data is less than the first preset area is used as the last frame of the thumbnail change.

[0214] In an embodiment of the present application, targeted area detection can be performed by selecting an area in a video frame corresponding to a thumbnail, thereby improving detection accuracy and reducing the amount of data required to determine whether a pixel has changed, thereby improving detection efficiency.

[0215] When the frame difference method is used to perform inter-frame difference on adjacent video frames, for each video frame located after the start frame in the video to be detected, the video frame can be firstly converted into grayscale to obtain a grayscale image (i.e., a second grayscale image) corresponding to the video frame, and then the inter-frame difference is performed on the area where the thumbnails of the grayscale images of the adjacent video frames are located to obtain the first frame difference data. The use of grayscale images for inter-frame difference can reduce the amount of calculation and improve the data processing speed. Moreover, the grayscale image can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of the data processing results.

[0216] In the embodiment of the present application, the duration of the camera operation response process includes the photo response duration. After detecting the start frame and the end frame, the two are subtracted to obtain the number of shooting frames, and the photo response duration is calculated according to the number of shooting frames and the camera shooting frame rate.

[0217] The above scenario 2 involves the interaction between mobile phones, high-speed cameras and computer equipment, such as Fig.11 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the present application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Fig.11 ,The interaction process includes the following steps.

[0218] S21. A high-speed camera shoots the screen of the mobile phone.

[0219] S22. Trigger the photo control in the mobile phone shooting interface.

[0220] S23. In response to the trigger operation, the mobile phone displays a thumbnail corresponding to the latest captured image in the shooting interface.

[0221] S24. The high-speed camera sends the captured video to be detected to the computer device.

[0222] S25. The computer device uses the frame difference method to identify whether the photo control starts to change in the video to be detected, and uses the first frame where the photo control starts to change as the start frame. The computer device also uses the frame difference method to identify whether the thumbnail in the image echo control appears completely, and uses the last frame where the thumbnail changes as the end frame.

[0223] As an example, the video frame detection method performed by the computer device in scenario 2 is as follows: Fig.12 Please refer to Fig.12 , the method comprises the following steps:

[0224] S251. Obtain the video to be detected and the corresponding operation control diagram and shooting interface diagram.

[0225] The operation control diagram includes camera controls, such as Fig. 9 The shooting interface diagram includes image echo controls, such as Fig. 9 As shown in Figure C.

[0226] S252. According to the position of the camera control in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the camera control in the video frame.

[0227] Based on the position features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the photo control in the video frame, and the area where the photo control is located in the video frame (i.e., the first preset area) is taken as a specific area that needs attention.

[0228] S253. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0229] S254. Perform inter-frame difference on a first preset area of ​​a first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0230] S255. Determine whether the frame difference is greater than a preset difference value?

[0231] If yes, execute S256, if no, execute S252.

[0232] S256. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the camera control to start changing.

[0233] S257. According to the position of the image echo control in the shooting interface diagram, perform region matching on the video frames after the start frame in the video to be detected to obtain a third preset region of the thumbnail in the video frame.

[0234] According to the positional features of the area where the image echo control is located in the shooting interface image, template matching is performed in the video frame after the start frame to determine the position coordinates of the thumbnail in the video frame, and the area where the thumbnail is located in the video frame (i.e., the third preset area) is taken as a specific area that needs attention.

[0235] S258. Perform grayscale conversion on the video frame after the start frame in the video to be detected to obtain a second grayscale image corresponding to the video frame.

[0236] S259. Perform inter-frame difference on a third preset area of ​​the second grayscale image corresponding to adjacent video frames to obtain first frame difference data.

[0237] S260. Determine whether the area of ​​the picture outline indicated by the first frame difference data is smaller than a first preset area?

[0238] S261. Use the video frame whose first frame difference data indicates that the area of ​​the picture outline is smaller than the first preset area as the last frame of the thumbnail change.

[0239] In this application Fig.11 and Fig.12 This paper introduces the scene from receiving the trigger operation of the photo control to generating the thumbnail (shot2see) corresponding to the latest shot image in the image echo control. The scene can include front highlight, front low light, rear highlight and rear low light. The implementation process and the beneficial effects achieved can be seen in the above Fig. 9 and Fig.10 The description will not be repeated here.

[0240] Below through Figure 13-Figure 16 The scenario 3 (the scenario from receiving a trigger operation on the image echo control to displaying a presentation image corresponding to the thumbnail in the image echo control) is described.

[0241] When viewing the picture just taken, the mobile phone triggers the image echo control (also referred to as the image echo button) based on the test instruction (for example, clicks on it), thereby realizing the automatic triggering operation process of the image echo control. In response to the triggering operation of the image echo control, the mobile phone fully displays the enlarged thumbnail in the gallery interface of the camera application (that is, the thumbnail is fully opened to become a display image). The user-perceivable process duration refers to the time from the beginning of the change of the image echo control to the full opening of the display image. In this process, the beginning of the change of the image echo control and the complete appearance of the display image are key nodes. The display image corresponding to the thumbnail is the latest image taken, and triggering the image echo control can display the latest image taken. Based on this, the high-speed camera can be used to shoot the operation response process of the mobile phone to view the thumbnail to obtain the video to be detected, and the video to be detected includes multiple continuous video frames.

[0242] Take the example of using a mobile phone to shoot a still portrait pasted on the wall. Fig.13 As shown, Fig.13 The following figure shows four representative video frames in the video to be detected corresponding to the scene from thumbnail to display image. It can be understood that the video to be detected in this scene also includes other video frames. The process from thumbnail to display image is as follows: Use an automated method to click the image echo control on the shooting interface of the camera application, such as Fig.13 As shown in Figure A in the figure; the image echo control is reduced, that is, the thumbnail displayed in the image echo control will be as follows Fig.13 The zooming process is shown in Figure B in the figure. Next, the mobile phone will display the display image. When the display image appears, it will go through the following steps: Fig.13 The zooming process shown in Figure C in the figure is continued until the display stops zooming, as shown in Figure C. Fig.13 Figure D in .

[0243] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the image echo control begins to change) and the end frame (indicating that the display image is fully opened or appears completely). The camera operation response process involves image echo control recognition and complete screen recognition. The changes in the image echo control can be identified by the frame difference method; complete screen recognition can be achieved by calculating the frame difference. When the picture is scaled, the frame difference is large, and the frame difference is small after the scaling is completed. Through the change pattern of the frame difference amplitude, it is recognized that the picture display is complete and the scaling has stopped. The image echo control recognition and complete screen recognition are explained below.

[0244] The video frame detection method provided by the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, an operation control diagram can be acquired in advance. The operation control diagram includes an image echo control. Fig.13As shown in Figure A in , the computer device pre-acquires an operation control diagram with an image echo control. The operation control diagram and the video frame including the image echo control are images under the same viewing angle, and the position of the image echo control in the operation control diagram is the same as its position in the video frame. When detecting whether the image echo control begins to change, the area where the image echo control is located in the video frame (i.e., the first preset area) can be detected without paying attention to other areas. Based on the positional features of the area where the image echo control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the image echo control are determined in the video frame, and the area where the image echo control is located in the video frame is used as a specific area that needs attention.

[0245] When the image echo control of the mobile phone is clicked, the control will shrink, such as Fig.13 As shown in Figure B, Fig.13 Compared with Figure A in the video, the image echo control is reduced. That is, it is necessary to detect the zooming behavior of the thumbnail to detect the first frame of clicking the image echo control. Based on this, the present application uses a frame difference method to perform inter-frame differences on the area where the image echo control of adjacent video frames in the video to be detected is located to obtain a frame difference. When the frame difference is greater than the preset difference value, it means that there is a more obvious difference between adjacent video frames, that is, the image echo control begins to change. The first frame with a frame difference value greater than the preset difference value is used as the first frame in which the image echo control begins to change, and the first frame in which the image echo control begins to change is used as the starting frame.

[0246] When using the frame difference method to perform inter-frame difference on adjacent video frames, the video frames can be first converted to grayscale and then inter-frame difference can be performed. Using grayscale images for inter-frame difference can reduce the amount of calculation and increase the data processing speed. Moreover, grayscale images can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0247] Among them, the recognition process of the image echo control is consistent with the recognition process of the camera application icon using the frame difference method mentioned above. Its implementation process and the technical effect achieved can be referred to the recognition process of the camera application icon using the frame difference method mentioned above, and will not be repeated here.

[0248] In an embodiment of the present application, by matching the area where the image echo control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy and reducing the amount of data required to determine whether the pixels have changed, thereby improving the detection efficiency.

[0249] In the scenario from thumbnail to display image, after the start frame in the video to be detected is detected, the enlarged thumbnail (i.e., display image) will appear next. When the display image appears, it will go through the following steps: Fig.13The scaling process shown in Figure C until Fig.13 D figure in. In the present application, the computer device can realize complete picture recognition by the method of picture contour detection. Inter-frame difference is performed on the adjacent video frames located after the starting frame in the video to be detected to obtain second frame difference data. The second frame difference data includes the pixel difference of multiple pixels, and the pixel difference of multiple pixels constitutes the picture contour. The area of ​​the picture contour reflects the difference between adjacent video frames; the larger the area of ​​the picture contour, the larger the difference between adjacent video frames, indicating that the display image is being scaled; the smaller the area of ​​the picture contour, the smaller the difference between adjacent video frames, indicating that the display image is scaled (that is, the display image stops scaling or the display image appears completely). The video frame whose area of ​​the picture contour indicated by the second frame difference data is less than the second preset area is used as the first frame of the display image fully opened (that is, the first frame of the display image stops scaling, or, the last frame of the display image changes), and the first frame of the display image fully opened is used as the end frame.

[0250] The full screen stops zooming detection effect is as follows Fig.14 As shown, Fig.14 It is merely an example of the appearance of a complete picture. Fig.14 The frame difference binary image after the current video frame and the previous video frame of the current video frame are subjected to the frame difference is shown in FIG. Fig.14 It can be seen that the difference between adjacent video frames is small, indicating that the complete picture is scaled. In the present application, the computer device determines whether the complete picture is scaled by determining whether the area of ​​the frame difference binary image is smaller than a preset area. Fig.14 It is a key node and is used as the end frame in practical applications.

[0251] By identifying the changing pattern of frame differences between adjacent video frames and judging whether a large area difference occurs in the frame difference, it is possible to determine whether the display image is fully opened (i.e., the display image stops scaling). This eliminates the need for manual detection and automatically detects the end frame, thereby improving the efficiency and accuracy of video frame detection.

[0252] Among them, the second preset area is a configuration parameter set by technical personnel in this field based on the area threshold adopted in a large number of experimental processes. As long as it can be determined whether the display image is fully opened or the display image stops zooming, the embodiment of the present application is not limited to this.

[0253] When the frame difference method is used to perform inter-frame difference on adjacent video frames, for each video frame located after the start frame in the video to be detected, the video frame can be first converted to grayscale to obtain a grayscale image corresponding to the video frame, and then the grayscale images of adjacent video frames are inter-frame differentiated to obtain the second frame difference data. Using grayscale images for inter-frame difference can reduce the amount of calculation and increase the data processing speed. In addition, grayscale images can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0254] In the embodiment of the present application, the duration of the camera operation response process includes the echo duration. After detecting the start frame and the end frame, the two are subtracted to obtain the shooting frame number, and the echo duration is calculated according to the shooting frame number and the camera shooting frame rate.

[0255] The above scenario 3 involves the interaction between the mobile phone, high-speed camera and computer equipment, such as Fig.15 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the present application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Fig.15 ,The interaction process includes the following steps.

[0256] S31. A high-speed camera shoots the screen of the mobile phone.

[0257] S32. Trigger the image echo control of the mobile phone shooting interface.

[0258] S33. In response to the trigger operation, the mobile phone displays the full image corresponding to the thumbnail in the shooting interface.

[0259] S34. The high-speed camera sends the captured video to be detected to the computer device.

[0260] S35. The computer device uses a frame difference method to identify whether the image echo control starts to change in the video frame to be detected, and uses the first frame where the image echo control starts to change as the start frame. The computer device also uses a frame difference method to identify whether the display image corresponding to the thumbnail is fully opened, and uses the first frame where the display image is fully opened as the end frame.

[0261] As an example, the video frame detection method performed by the computer device in scenario 3 is as follows: Fig.16 Please refer to Fig.16 , the method comprises the following steps:

[0262] S351. Obtain the video to be detected and the corresponding operation control diagram.

[0263] The operation control diagram includes image echo controls, such as Fig.13As shown in Figure A.

[0264] S352. According to the position of the image echo control in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the image echo control in the video frame.

[0265] According to the positional features of the area where the image echo control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the image echo control in the video frame, and the area where the image echo control is located in the video frame (i.e., the first preset area) is taken as a specific area that needs attention.

[0266] S353. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0267] S354. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0268] S355. Determine whether the frame difference is greater than a preset difference value?

[0269] If yes, execute S356, if no, execute S352.

[0270] S356. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the image echo control to start changing.

[0271] S357. Perform grayscale conversion on the video frames after the start frame in the video to be detected to obtain a grayscale image corresponding to the video frame.

[0272] S358. Perform inter-frame difference on the grayscale images of adjacent video frames located after the start frame in the video to be detected to obtain second frame difference data.

[0273] S359. Determine whether the area of ​​the picture outline indicated by the second frame difference data is smaller than a second preset area?

[0274] If yes, execute S360, if no, execute S357.

[0275] S360. Use the video frame whose second frame difference data indicates that the area of ​​the picture outline is smaller than the second preset area as the first frame of the fully opened display image.

[0276] In this application Fig.15 and Fig.16 This article introduces the scenario from receiving a trigger operation on the image echo control to displaying the presentation image (see2review) corresponding to the thumbnail in the image echo control. The implementation process and the beneficial effects achieved can be found in the above Fig.13 and Fig.14 The description will not be repeated here.

[0277] Below through Figure 17-Figure 19 Scenario 4 (the scenario in which the camera application jumps from the gallery interface to the preview interface) is described.

[0278] When you need to continue shooting after viewing the taken pictures, the mobile phone triggers the return control (also known as the return button) based on the test instruction (for example, clicks on it) to realize the automatic triggering operation process of the return control. In response to the triggering operation of the return control, the mobile phone returns to the preview interface of the camera application. The user-perceivable process duration refers to the time from the beginning of the change of the return control to the appearance of the outline in the preview interface. In this process, the beginning of the change of the return control and the appearance of the outline in the preview interface are key nodes. Based on this, the operation response process of the mobile phone returning to continue shooting after viewing the taken pictures can be captured by a high-speed camera to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0279] Take the example of using a mobile phone to shoot a still portrait pasted on the wall. Fig.17 As shown, Fig.17 The following figure shows four representative video frames in the video to be detected corresponding to the scene of jumping from the gallery interface of the camera application to the preview interface. It can be understood that the video to be detected in this scene also includes other video frames. The process of jumping from the gallery interface of the camera application to the preview interface is as follows: Use an automated method to click the return control on the gallery interface of the camera application, such as Fig.17 As shown in Figure A; the return control is reduced, as shown in Figure Fig.17 As shown in Figure B; the phone will then display the preview screen. When the preview screen appears, it will go through the following steps: Fig.17 The fuzzy process shown in Figure C until Fig.17 Figure D in .

[0280] The video to be detected is detected using the video frame detection method provided in this application to identify the key nodes where the behavior occurs in the camera operation response process. The key nodes include the start frame (the first frame indicating that the return control begins to change) and the end frame (the first frame indicating that the picture appears in the preview interface). The camera operation response process involves return control identification and picture identification in the preview interface. The change of the return control can be identified by the frame difference method; the white lines are identified by the edge detection operator extraction method to detect whether the picture appears in the preview interface. The return control identification and picture identification in the preview interface are explained below.

[0281] The video frame detection method provided by the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, an operation control diagram can be obtained in advance. The operation control diagram includes a return control, such as Fig.17As shown in Figure A in , the computer device pre-acquires an operation control diagram with a return control. The operation control diagram and the video frame including the return control are images under the same perspective, and the position of the return control in the operation control diagram is the same as its position in the video frame. When detecting whether the return control begins to change, the area where the return control is located in the video frame (i.e., the first preset area) can be detected without paying attention to other areas. Based on the positional features of the area where the return control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the return control are determined in the video frame, and the area where the return control is located in the video frame is treated as a specific area that needs to be paid attention to.

[0282] When the mobile phone's back control is clicked, the control will shrink, such as Fig.17 As shown in Figure B, Fig.17 Compared with Figure A in the video to be detected, the return control is reduced. Based on this, the present application uses a frame difference method to perform inter-frame differences on the area where the return control of adjacent video frames in the video to be detected is located to obtain a frame difference value. When the frame difference value is greater than the preset difference value, it indicates that there is a more obvious difference between adjacent video frames, that is, the return control begins to change. The first frame whose frame difference value is greater than the preset difference value is used as the first frame where the return control begins to change, and the first frame where the return control begins to change is used as the starting frame.

[0283] When using the frame difference method to perform inter-frame difference on adjacent video frames, the video frames can be first converted to grayscale and then inter-frame difference can be performed. Using grayscale images for inter-frame difference can reduce the amount of calculation and increase the data processing speed. Moreover, grayscale images can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0284] Among them, the recognition process of the returned control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0285] In an embodiment of the present application, by matching the area where the control is returned in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving detection accuracy and reducing the amount of data required to determine whether the pixels have changed, thereby improving detection efficiency.

[0286] In the scene from the gallery interface of the camera application to the preview interface, after detecting the starting frame in the video to be detected, the computer device then detects the screen of the preview interface of the mobile phone. When the screen in the preview interface appears, the mobile phone will go through the following steps: Fig.17 The fuzzy process shown in Figure C until Fig.17Figure D in the figure. In the present application, the computer device can realize the recognition of the picture in the preview interface by the edge detection method. The recognition process of the picture in the preview interface is consistent with the recognition process of the picture in the preview interface in the scene of starting the camera application for shooting. The implementation process and the technical effect achieved can be referred to the recognition process of the picture in the preview interface above, which will not be repeated here.

[0287] By comparing the number of lines that make up the outline, it is determined whether the picture appears in the preview interface without manual detection, thus realizing the automatic detection process of the end frame and improving the efficiency and accuracy of video frame detection.

[0288] In the embodiment of the present application, the duration of the camera operation response process includes the interface jump duration. After detecting the start frame and the end frame, the two are subtracted to obtain the shooting frame number, and the interface jump duration is calculated according to the shooting frame number and the camera shooting frame rate.

[0289] Scenario 4 above involves the interaction between mobile phones, high-speed cameras, and computer devices, such as Fig.18 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the present application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Fig.18 ,The interaction process includes the following steps.

[0290] S41. A high-speed camera takes a picture of the screen of the mobile phone.

[0291] S42. Trigger the return control in the gallery interface.

[0292] S43. The mobile phone displays the picture in the preview interface in response to the trigger operation.

[0293] S44. The high-speed camera sends the captured video to be detected to the computer device.

[0294] S45. The computer device identifies whether the return control starts to change in the video to be detected by using a frame difference method, and uses the first frame where the return control starts to change as the start frame. It also identifies whether a picture appears in the preview interface, and uses the first frame where the picture appears in the preview interface as the end frame.

[0295] As an example, the video frame detection method performed by the computer device in scenario 4 is as follows: Fig.19 Please refer to Fig.19 , the method comprises the following steps:

[0296] S451. Obtain the video to be detected and the corresponding operation control diagram and operation response diagram.

[0297] The operation control diagram includes return controls, such as Fig.17 The operation response diagram is an interface diagram displayed by the mobile phone in response to a trigger operation on the operation control. The operation response diagram includes a still portrait (ie, the first object), such as Fig.17 As shown in Figure D.

[0298] S452. According to the position of the return control in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the return control in the video frame.

[0299] Based on the position features of the area where the return control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the return control in the video frame, and the area where the return control is located in the video frame (i.e., the first preset area) is taken as a specific area that needs attention.

[0300] S453. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0301] S454. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0302] S455. Determine whether the frame difference is greater than a preset difference value?

[0303] If yes, execute S456, if no, execute S452.

[0304] S456. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the return control to start changing.

[0305] S457. According to the position of the first object in the operation response graph, perform region matching on the video frame after the start frame in the video to be detected to obtain a second preset region of the first object in the video frame.

[0306] According to the position features of the area where the first object is located in the operation response graph, template matching is performed in the video frame located after the start frame to determine the position coordinates of the first object in the video frame, and the area where the first object is located in the video frame (i.e., the second preset area) is taken as a specific area that needs attention.

[0307] S458. Extract edge features from the second preset areas of the video frames after the start frame in the video to be detected, and obtain line features of each video frame after the start frame.

[0308] S459. Determine whether the number of lines indicated by the line feature is greater than the first number?

[0309] If yes, execute S460, if no, execute S457.

[0310] S460. Use the first frame of the video frame in which the number of line feature indication lines is greater than the first number as the first frame of the picture appearing in the preview interface.

[0311] In this application Fig.18 and Fig.19 This section introduces the scene where the camera application jumps from the gallery interface to the preview interface (Gallery2Preview). The implementation process and the beneficial effects achieved can be found in the above Fig.17 The description will not be repeated here.

[0312] Below through Figure 20-Figure 22 Scene 5 (the scene of switching between the front-camera shooting and the rear-camera shooting) is described.

[0313] This scenario includes switching from front-facing shooting to rear-facing shooting and from rear-facing shooting to front-facing shooting. When switching the camera for shooting, the mobile phone triggers the flip control (also referred to as a flip button) based on the test instruction (for example, a click operation) to realize the automatic triggering operation of the flip control. In response to the triggering operation of the flip control, the mobile phone displays the preview interface of the front camera or the rear camera. The user-perceivable process duration refers to the time from the beginning of the change of the flip control to the appearance of the outline in the preview interface. In this process, the beginning of the change of the flip control and the appearance of the outline in the preview interface are key nodes. Based on this, the process of switching the front camera or the rear camera of the mobile phone can be filmed by a high-speed camera to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0314] For example, the rear camera of a mobile phone takes a still portrait pasted on the wall, and the front camera of the mobile phone takes a still picture. Fig. 20 As shown, Fig. 20 The figure shows five representative video frames in the video to be detected corresponding to the scene of switching from rear-facing shooting to front-facing shooting. It can be understood that the video to be detected in this scene also includes other video frames. The process of switching from the rear camera to the front camera is as follows: Use an automated method to click the flip control on the shooting interface of the camera application, such as Fig. 20 As shown in Figure A; flip the control to zoom out, as shown in Figure Fig. 20 As shown in Figure B, then Fig. 20 As shown in Figure C, the flip control returns to normal; then the phone will display the screen in the preview interface, and when the screen in the preview interface appears, it will go through the following steps: Fig. 20 Figure C and Fig. 20 The fuzzy process shown in Figure D until Fig. 20 Figure E in the figure means that the picture in the preview interface gradually changes from clear to blurred and then becomes clear again.

[0315] The video to be detected is detected using the video frame detection method provided in this application to identify the key nodes where the behavior occurs in the camera operation response process. The key nodes include the start frame (the first frame indicating that the flip control begins to change) and the end frame (the first frame indicating that the picture appears in the preview interface). The camera operation response process involves flip control recognition and picture recognition in the preview interface. The changes in the flip control can be identified by the frame difference method; the white lines are identified by the edge detection operator extraction method to detect whether the picture appears in the preview interface. The following is an explanation of flip control recognition and picture recognition in the preview interface.

[0316] The video frame detection method provided by the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, an operation control diagram can be obtained in advance. The operation control diagram includes a flip control. Fig. 20 As shown in Figure A in , the computer device pre-acquires an operation control diagram with a flip control. The operation control diagram and the video frame including the flip control are images under the same viewing angle, and the position of the flip control in the operation control diagram is the same as its position in the video frame. When detecting whether the flip control begins to change, the area where the flip control is located in the video frame (i.e., the first preset area) can be detected without paying attention to other areas. Based on the positional features of the area where the flip control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the flip control are determined in the video frame, and the area where the flip control is located in the video frame is used as a specific area that needs to be paid attention to.

[0317] When the flip control of the mobile phone is clicked, the control will shrink first and then return to normal, such as Fig. 20 As shown in Figure B, Fig. 20 Compared to Figure A in Figure 1, the flip control is zoomed out, and then Fig. 20 As shown in Figure C in the figure, the flip control returns to normal. Based on this, the present application uses the frame difference method to perform inter-frame differences on the area where the flip control of adjacent video frames in the video to be detected is located to obtain a frame difference value. When the frame difference value is greater than the preset difference value, it means that there is a more obvious difference between adjacent video frames, that is, the flip control begins to change. The first frame whose frame difference value is greater than the preset difference value is used as the first frame where the flip control begins to change, and the first frame where the flip control begins to change is used as the starting frame.

[0318] When using the frame difference method to perform inter-frame difference on adjacent video frames, the video frames can be first converted to grayscale and then inter-frame difference can be performed. Using grayscale images for inter-frame difference can reduce the amount of calculation and increase the data processing speed. Moreover, grayscale images can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0319] Among them, the recognition process of the flip control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0320] In an embodiment of the present application, by matching the area where the flip control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving detection accuracy and reducing the amount of data required to determine whether the pixels have changed, thereby improving detection efficiency.

[0321] In the scenario of switching from rear-camera shooting to front-camera shooting, after detecting the start frame in the video to be detected, the computer device then detects the picture in the preview interface of the mobile phone. When the picture in the preview interface appears, the mobile phone will go through the following steps: Fig. 20 Figure C and Fig. 20 The fuzzy process shown in Figure D until Fig. 20 Figure E in the figure, that is, the picture in the preview interface gradually changes from clear to blurred and then becomes clear again. In the present application, the computer device can realize the recognition of the picture in the preview interface by the edge detection method. The edge feature extraction is performed on the video frame located after the starting frame in the video to be detected to obtain the line feature. The line feature can indicate the number of lines. The process in which the picture in the preview interface gradually changes from clear to blurred and then becomes clear again is reflected in the gradual decrease in the number of lines and then gradually increases. In the process of detecting that the line feature indicates that the number of lines is gradually decreasing, when the number of lines is less than the second number, it means that the picture in the preview interface becomes blurred, and the number of lines is continued to be counted. In the process of detecting that the line feature of multiple video frames indicates that the number of lines is gradually increasing, when the number of lines is greater than the first number, it means that the picture in the preview interface becomes clear again. Therefore, the first frame of the video frame whose number of lines is greater than the first number is used as the first frame that appears in the preview interface, and the first frame that appears in the preview interface is used as the end frame. By comparing the number of lines that make up the contour, and judging whether the corresponding picture becomes blurred or becomes clear again according to whether the number of lines is lower than the second number or higher than the first number, the cleared video frame is used as the end frame without manual detection, thus realizing automatic detection of the end frame and improving the detection efficiency and accuracy of the video frame.

[0322] The second number is a configuration parameter set by a technician in this field based on a number threshold adopted in a large number of experiments. As long as it can be determined that the picture becomes blurred, the embodiment of the present application does not limit this.

[0323] When switching from front-facing camera to rear-facing camera or from rear-facing camera to front-facing camera, the preview interface switches twice. Fig. 20 Taking the switching from the rear camera to the front camera shown in the figure as an example, the picture in the preview interface changes from clear to blurred as the rear camera is switched, and then changes from blurred to clear as the front camera is switched. Therefore, it is necessary not only to detect whether the number of lines is less than the second number to determine the process of the picture becoming blurred, but also to detect whether the number of lines is greater than the first number to determine the process of the picture becoming clear again. The embodiment of the present application detects the process of becoming blurred and clear during the switching process. After detecting that the picture has become blurred, the process of the picture becoming clear is detected again to reduce the situation where the number of lines is greater than the first number due to unsuccessful switching, thereby ensuring successful switching and improving the accuracy of video frame detection.

[0324] The video frame detection method provided in the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, for switching to front-camera shooting or switching to rear-camera shooting, an operation response graph can be acquired in advance. For the sake of distinction, they can be called front-camera operation response graph and rear-camera operation response graph.

[0325] For the scenario of switching from rear-view shooting to front-view shooting, the front-view operation response image may include a still picture and a surrounding background, and the background may be black or white, etc. Fig. 20 As shown in Figure E in . The pre-operation response graph and the video frame where the still picture is detected are images under the same viewing angle, and the position of the still picture in the pre-operation response graph is the same as its position in the video frame. When detecting whether the still picture appears, the detection can be performed on the area where the still picture is located in the video frame, without paying attention to other areas. According to the positional features of the area where the still picture is located in the pre-operation response graph, template matching is performed in the video frame after it is detected that the number of lines is less than the second number, the position coordinates of the still picture are determined in the video frame, and the area where the still picture is located in the video frame is taken as a specific area that needs to be paid attention to. Then, edge features are extracted from the area where the still picture is located in the video frame after it is detected that the number of lines is less than the second number, and line features are obtained. It is judged whether the still picture becomes clear based on the line features.

[0326] For the scene switching from front-facing to rear-facing shooting, that is, Fig. 20 The post-operation response image may include a still portrait and the surrounding background, and the background may be black or white, etc. Fig. 20As shown in Figure A in , the post-operation response graph and the video frame in which the still portrait is detected are images under the same viewing angle, and the position of the still portrait in the post-operation response graph is the same as its position in the video frame. When detecting whether the still portrait becomes blurred, the detection can be performed on the area where the still portrait is located in the video frame, without paying attention to other areas. According to the positional features of the area where the still portrait is located in the post-operation response graph, template matching is performed in the video frame, the position coordinates of the still portrait are determined in the video frame, and the area where the still portrait is located in the video frame is taken as a specific area that needs to be paid attention to. Then, edge features are extracted from the area where the still portrait is located in the video frame after the starting frame to obtain line features. It is judged whether the still portrait becomes blurred based on the line features.

[0327] In the embodiment of the present application, by selecting the area corresponding to the still portrait in the video frame after the starting frame, and selecting the area corresponding to the still picture in the video frame after the number of lines is detected to be less than the second number, targeted area detection can be performed, thereby improving detection accuracy, reducing the edge feature extraction range, and improving detection efficiency.

[0328] In the embodiment of the present application, the duration of the camera operation response process includes the camera switching duration. After detecting the start frame and the end frame, the two are subtracted to obtain the shooting frame number, and the camera switching duration is calculated according to the shooting frame number and the camera shooting frame rate.

[0329] Scenario 5 above involves the interaction between a mobile phone, a high-speed camera, and a computer device, e.g. Fig.21 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the present application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Fig.21 ,The interaction process includes the following steps.

[0330] S51. A high-speed camera shoots the screen of the mobile phone.

[0331] S52. Trigger the flip control in the mobile phone shooting interface.

[0332] S53. The mobile phone responds to the trigger operation and displays the picture in the preview interface.

[0333] S54. The high-speed camera sends the captured video to be detected to the computer device.

[0334] S55. The computer device identifies whether the flip control starts to change in the video to be detected by using a frame difference method, and uses the first frame where the flip control starts to change as the start frame. The computer device also identifies whether the picture reappears in the preview interface, and uses the first frame where the picture reappears in the preview interface as the end frame.

[0335] As an example, the video frame detection method performed by the computer device in scenario 5 is as follows: Fig. 22 Please refer to Fig. 22 , the method comprises the following steps:

[0336] S551. Obtain the video to be detected and the corresponding operation control diagram and operation response diagram.

[0337] The operation control diagram includes a flip control, such as Fig. 20 The operation response diagram is an interface diagram displayed by the mobile phone in response to a trigger operation on the operation control. The operation response diagram may include a still picture (ie, a first object), such as Fig. 20 As shown in Figure E.

[0338] S552. According to the position of the flip control in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the flip control in the video frame.

[0339] According to the position features of the area where the flip control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the flip control in the video frame, and the area where the flip control is located in the video frame (i.e., the first preset area) is taken as a specific area that needs attention.

[0340] S553. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0341] S554. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0342] S555. Determine whether the frame difference is greater than a preset difference value?

[0343] S556. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the flip control to start changing.

[0344] S557. According to the position of the first object in the operation response graph, perform region matching on the video frame after the start frame in the video to be detected to obtain a second preset region of the first object in the video frame.

[0345] According to the position features of the area where the first object is located in the operation response graph, template matching is performed in the video frame located after the start frame to determine the position coordinates of the first object in the video frame, and the area where the first object is located in the video frame (i.e., the second preset area) is taken as a specific area that needs attention.

[0346] S558. Extract edge features from the second preset areas of the video frames after the start frame in the video to be detected, and obtain line features of each video frame after the start frame.

[0347] S559. In the process in which the number of lines indicated by the line feature of the plurality of video frames after the start frame gradually decreases, determine whether the number of lines indicated by the line feature is less than a second number?

[0348] If yes, execute S560, if no, execute S557.

[0349] S560. In the process in which the number of lines indicated by the line feature of the plurality of video frames after the start frame gradually increases, determine whether the number of lines indicated by the line feature is greater than a first number?

[0350] If yes, execute S561, if no, continue to execute S560.

[0351] S561. Use the first frame of the video frame in which the number of line feature indication lines is greater than the first number as the first frame of the picture appearing in the preview interface.

[0352] In this application Fig.21 and Fig. 22 This section introduces the scenarios of switching from front-facing to rear-facing or from rear-facing to front-facing. The implementation process and the beneficial effects achieved can be found in the above Fig. 20 The description will not be repeated here.

[0353] Below through Figure 23-Figure 25 Scene 6 (from receiving a trigger operation on a video recording control to starting video recording) is described.

[0354] When recording a video using a mobile phone, the mobile phone triggers the video control (also called the video button) based on the test instruction (for example, clicks on it) to realize the automatic triggering operation of the video control. In response to the triggering operation of the video control, the mobile phone starts recording. The user-perceivable process duration refers to the duration from the change of the video control to the start of the recording (for example, the recording duration is a preset duration). In this process, the start of the change of the video control and the start of the recording are key nodes. Based on this, the operation response process of the mobile phone performing the recording can be captured by a high-speed camera to obtain a video to be tested, and the video to be tested includes multiple continuous video frames.

[0355] Take the example of using a mobile phone to record a still portrait pasted on the wall. Fig.23 As shown, Fig.23 The following figure shows three representative video frames in the video to be detected corresponding to the recording scene. It can be understood that the video to be detected in this scene also includes other video frames. The recording process is as follows: Use an automated method to click the recording control on the shooting interface of the camera application, such as Fig.23 As shown in Figure A; the video control is reduced, as shown in Fig.23 As shown in Figure B; then the phone will display the video recording interface, such as Fig.23 As shown in Figure C.

[0356] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the video control starts to change) and the end frame (indicating that the video duration is the preset duration). The camera operation response process involves the identification of the video control and the determination of the video duration. The change of the video control can be identified by the frame difference method. The identification of the video control and the determination of the video duration are explained below.

[0357] The video frame detection method provided in the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, an operation control diagram can be obtained in advance. The operation control diagram includes a video recording control, such as Fig.23 As shown in Figure A in , the computer device pre-acquires an operation control diagram with a video control. The operation control diagram and the video frame including the video control are images under the same viewing angle, and the position of the video control in the operation control diagram is the same as its position in the video frame. When detecting whether the video control begins to change, the detection can be performed on the area where the video control is located in the video frame (i.e., the first preset area), without paying attention to other areas. Based on the positional features of the area where the video control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the video control are determined in the video frame, and the area where the video control is located in the video frame is used as a specific area that needs to be paid attention to.

[0358] When the recording control of the mobile phone is clicked, the control will shrink, such as Fig.23 As shown in Figure B, Fig.23 Compared with Figure A in the video, the video control is smaller. Based on this, the present application uses the frame difference method to perform inter-frame differences on the area where the video control of adjacent video frames in the video to be detected is located to obtain a frame difference value. When the frame difference value is greater than the preset difference value, it indicates that there is a more obvious difference between adjacent video frames, that is, the video control begins to change. The first frame where the frame difference value is greater than the preset difference value is taken as the first frame where the video control begins to change, and the first frame where the video control begins to change is taken as the starting frame.

[0359] When using the frame difference method to perform inter-frame difference on adjacent video frames, the video frames can be first converted to grayscale and then inter-frame difference can be performed. Using grayscale images for inter-frame difference can reduce the amount of calculation and increase the data processing speed. Moreover, grayscale images can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0360] Among them, the recognition process of the video recording control is consistent with the recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the recognition process of the camera application icon using the frame difference method, which will not be repeated here.

[0361] In an embodiment of the present application, by matching the area where the recording control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy and reducing the amount of data required to determine whether the pixels have changed, thereby improving the detection efficiency.

[0362] The video frames with a recording time of a preset duration can be determined from the video frames after the start frame in the video to be detected, and the first frame of the video frames with a recording time of the preset duration is used as the end frame.

[0363] The preset duration is a configuration parameter appropriately set by those skilled in the art according to actual conditions, for example, 00:01, 00:02, etc. Fig.23 As shown in Figure C, Fig.23 Figure C in the figure shows the interface with a recording duration of 00:01.

[0364] By determining the recording duration, the first frame of the video frame with a preset recording duration is used as the end frame, without the need for manual detection, thereby realizing the automatic detection process of the end frame and improving the video frame detection efficiency and accuracy.

[0365] In the embodiment of the present application, the duration of the camera operation response process includes the video recording response duration. After detecting the start frame and the end frame, the two are subtracted to obtain the shooting frame number, and the video recording response duration is calculated according to the shooting frame number and the camera shooting frame rate.

[0366] The above scenario 6 involves the interaction between the mobile phone, high-speed camera and computer equipment, such as Fig.24 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the present application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Fig.24 ,The interaction process includes the following steps.

[0367] S61. A high-speed camera shoots the screen of the mobile phone.

[0368] S62. Trigger the video recording control in the shooting interface of the mobile phone.

[0369] S63. The mobile phone displays a video recording interface in response to the trigger operation.

[0370] S64. The high-speed camera sends the captured video to be detected to the computer device.

[0371] S65. The computer device identifies whether the video control starts to change in the video to be detected by using a frame difference method, and takes the first frame where the video control starts to change as the start frame, and takes the first frame of the video frame with a preset recording duration as the end frame.

[0372] As an example, the video frame detection method performed by the computer device in scenario 6 is as follows: Fig.25 Please refer to Fig.25 , the method comprises the following steps:

[0373] S651. Obtain the video to be detected and the corresponding operation control diagram.

[0374] The operation control diagram includes video recording controls, such as Fig.23 As shown in Figure A.

[0375] S652. Perform region matching on the video frame in the video to be detected according to the position of the video recording control in the operation control diagram, and obtain a first preset region of the video recording control in the video frame.

[0376] According to the position features of the area where the video control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the video control in the video frame, and the area where the video control is located in the video frame (i.e., the first preset area) is taken as a specific area that needs attention.

[0377] S653. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0378] S654. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0379] S655. Determine whether the frame difference is greater than a preset difference value?

[0380] If yes, execute S656, if no, execute S652.

[0381] S656. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the video recording control to start changing.

[0382] S657. Determine, among the video frames after the start frame in the video to be detected, video frames whose recording duration is the preset duration.

[0383] S658. Use the first frame of the video frames whose recording duration is the preset duration as the end frame corresponding to the camera operation response process.

[0384] In this application Fig.24 and Fig.25 This section describes the process from receiving a trigger operation on the video control to starting the video recording. The implementation process and the beneficial effects achieved can be found in the above Fig.23 The description will not be repeated here.

[0385] Below through Figure 26-Figure 28 Scene 7 (the scene switching from close-up shooting to long-range shooting) will be described.

[0386] The video frame detection method provided in the present application is applied in the testing process of a mobile phone, and the process of switching from a close view to a distant view can be simulated by removing the baffle. The baffle is placed in front of the rear camera of the mobile phone, and the distant view screen appears zoomed after the baffle is completely removed. After the baffle is completely removed, a component will be displayed, and the component may be a trapezoidal component, a triangular component, a square component or a button, etc., which is not limited to the embodiments of the present application. Whether the baffle is completely removed can be detected by detecting whether the outline of the component appears. When the outline of the component is detected, it means that the baffle is completely removed. The user-perceivable process duration refers to the time from when the baffle is completely removed to when the distant view screen stops zooming. In this process, the outline that appears after the baffle is completely removed and the picture in the distant view stops zooming are key nodes. Based on this, the operation response process of the mobile phone to perform a photo-taking operation can be captured by a high-speed camera to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0387] Take the example of taking a photo of a still person pasted on a wall in the distance with a mobile phone. Fig.26 As shown, Fig.26 The figure shows four representative video frames in the video to be detected corresponding to the scene of switching from close view to distant view. It can be understood that the video to be detected in this scene also includes other video frames. The process of switching from close view to distant view is as follows: a baffle as close view is placed in front of the camera, and the picture in the preview interface of the mobile phone is the baffle, such as Fig.26 As shown in Figure A; remove the baffle, such as Fig.26 As shown in Figure B in the figure; when the baffle is completely removed, parts will appear, such as Fig.26 As shown in Figure C in the figure, the phone focuses on the distant view, and the distant view will appear blurred and zoomed. Fig.26 The zooming process shown in Figure C in the figure is continued until the distant scene stops zooming. Fig.26Figure D in the figure. The video frame detection method provided in the present application is used to detect the video to be detected, and whether the baffle moves. When the movement of the baffle is detected, the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the baffle is completely removed) and the end frame (indicating that the picture in the distant view stops zooming). The camera operation response process involves the identification of the complete removal of the baffle and the identification of the stopped zooming of the picture. The frame difference method is used to identify whether the baffle has moved. After the movement of the baffle is identified, the change in the contour of the component is identified to determine whether the baffle is completely removed. The identification of the stopped zooming of the picture can be achieved by frame difference. When the picture is zoomed, the frame difference is large, and the frame difference is small after the zoom is completed. The change pattern of the frame difference amplitude is used to identify whether the picture has stopped zooming. The component identification and the identification of the stopped zooming of the picture are explained below.

[0388] The video frame detection method provided in the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the baffle image can be obtained in advance, such as Fig.26 As shown in Figure A in the figure, the baffle is placed on the rear camera of the camera, and the baffle appears in the preview interface. The baffle image includes the baffle, and the baffle image and the video frame including the baffle are images under the same perspective, and the position of the baffle in the baffle image is the same as its position in the video frame. When detecting whether the baffle has moved, the area where the baffle is located in the video frame (i.e., the baffle area) can be detected without paying attention to other areas. According to the position characteristics of the area where the baffle is located in the baffle image, template matching is performed in the video frame, the position coordinates of the baffle are determined in the video frame, and the baffle area in the video frame is used as a specific area that needs to be paid attention to.

[0389] The present application uses a frame difference method to perform inter-frame difference on the baffle area of ​​adjacent video frames in the video to be detected, and obtains third frame difference data. The third frame difference data includes the pixel difference of multiple pixels, and the third frame difference data can reflect whether the baffle has moved. The larger the pixel difference, the greater the difference between adjacent video frames, indicating that the baffle has moved. The smaller the pixel difference, the smaller the difference between adjacent video frames, indicating that the baffle has not moved. Whether the baffle has moved is determined based on the third frame difference data, and if the baffle has moved, whether the baffle has been completely removed is identified.

[0390] When detecting whether the baffle has moved, the detection can also be performed on the area where the mobile control is located. The mobile control is used to move the baffle. When the mobile control is clicked, the baffle starts to move. The recognition process of the mobile control is consistent with the recognition process of the camera application icon using the frame difference method. The implementation process and the technical effect achieved can be referred to the recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0391] In the embodiment of the present application, the mobile phone can be operated in a relatively static state, so the component diagram can be acquired in advance. Fig.26 As shown in Figure B, the baffle moves to the left. After the baffle is completely moved away, a component appears at the back, such as Fig.26 As shown in Figure C in the figure, the component is shown as a trapezoidal component. The component diagram includes a trapezoidal component, and the component diagram and the video frame including the trapezoidal component are images under the same viewing angle, and the position of the trapezoidal component in the component diagram is the same as its position in the video frame. When detecting whether the baffle is completely removed, the area where the trapezoidal component is located in the video frame (i.e., the component area) can be detected without paying attention to other areas. Based on the positional features of the area where the trapezoidal component is located in the component diagram, template matching is performed in the video frame, the position coordinates of the trapezoidal component are determined in the video frame, and the area where the trapezoidal component is located in the video frame is treated as a specific area that needs attention.

[0392] The component regions of adjacent video frames in the video to be detected are subjected to inter-frame difference to obtain fourth frame difference data. The fourth frame difference data includes pixel differences of a plurality of pixels, and the pixel differences of the plurality of pixels constitute the outline of the picture. When the outline of the trapezoidal component is detected, it indicates that the baffle is completely removed. The first frame in which the outline of the trapezoidal component appears is used as the first frame in which the baffle is completely removed, and the first frame in which the baffle is completely removed is used as the starting frame.

[0393] In an embodiment of the present application, by matching the baffle area and the component area in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy and reducing the amount of data required to determine whether the pixels have changed, thereby improving the detection efficiency.

[0394] When the frame difference method is used to perform inter-frame difference on the component areas of adjacent video frames, the video frame can be firstly gray-converted for each video frame in the video to be detected to obtain a gray-scale image (i.e., a third gray-scale image) corresponding to the video frame. Then, the component areas of the gray-scale images of adjacent video frames are inter-frame-differentiated to obtain fourth frame difference data. The use of gray-scale images for inter-frame difference can reduce the amount of calculation and improve the data processing speed. Moreover, the gray-scale image can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0395] In the scene switching from close-up to distant view, after detecting the starting frame in the video to be detected, that is, after the baffle is completely removed, the phone focuses on the distant view, which will be blurred and scaled. Fig.26 The scaling process shown in Figure C until Fig.26Figure D in the figure. Based on this, inter-frame difference is performed on adjacent video frames located after the start frame in the video to be detected to obtain fifth frame difference data. The fifth frame difference data includes pixel differences of multiple pixel points, and the pixel differences of multiple pixel points constitute the outline of the picture. The area of ​​the outline of the picture reflects the difference between adjacent video frames; the larger the area of ​​the outline of the picture, the larger the difference between adjacent video frames, indicating that the distant picture is being scaled; the smaller the area of ​​the outline of the picture, the smaller the difference between adjacent video frames, indicating that the scaling of the distant picture is completed (that is, the distant picture stops scaling). The video frame whose area of ​​the outline of the picture indicated by the fifth frame difference data is less than the third preset area is used as the first frame for stopping scaling of the distant picture, and the first frame for stopping scaling of the distant picture is used as the end frame.

[0396] Among them, the third preset area is a configuration parameter set by technical personnel in this field based on an area threshold adopted in a large number of experiments. As long as it can determine whether the distant view picture stops zooming, the embodiment of the present application does not limit this.

[0397] By identifying the changing pattern of frame differences between adjacent video frames and judging whether a large area difference appears in the frame difference, it is possible to determine whether the distant view has stopped zooming. This eliminates the need for manual detection and realizes automatic detection of the end frame, thereby improving the efficiency and accuracy of video frame detection.

[0398] The video frame detection method provided by the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, a distant view image can be acquired in advance. The distant view image includes a static portrait (i.e., the second object) and the surrounding background. The background can be black or white, etc. Fig.26 As shown in Figure D in . The distant view image and the video frame in which the still portrait is detected are images under the same viewing angle, and the position of the still portrait in the distant view image is the same as its position in the video frame. When detecting whether the still portrait stops zooming, the area where the still portrait is located in the video frame (i.e., the fourth preset area) can be detected without paying attention to other areas. According to the position characteristics of the area where the still portrait is located in the distant view image, template matching is performed in the video frame, the position coordinates of the still portrait are determined in the video frame, and the area where the still portrait is located in the video frame is taken as a specific area that needs attention. Then, inter-frame difference is performed on the area where the still portrait is located in the adjacent video frames after the starting frame in the video to be detected, and the fifth frame difference data is obtained, and the video frame whose area of ​​the picture contour indicated by the fifth frame difference data is less than the third preset area is taken as the first frame where the distant view picture stops zooming.

[0399] In the embodiment of the present application, by selecting the area in the video frame corresponding to the still portrait, targeted area detection can be performed, thereby improving detection accuracy, reducing the amount of data required to determine whether a pixel has changed, and improving detection efficiency.

[0400] When the frame difference method is used to perform inter-frame difference on adjacent video frames, for each video frame located after the starting frame in the video to be detected, the video frame can be first converted to grayscale to obtain a grayscale image corresponding to the video frame (i.e., the fourth grayscale image). Then, inter-frame difference is performed on the region where the still portrait of the grayscale image of the adjacent video frame is located to obtain the fifth frame difference data. Using the grayscale image for inter-frame difference can reduce the amount of calculation and increase the data processing speed. In addition, the grayscale image can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of the data processing results.

[0401] In the embodiment of the present application, the duration of the camera operation response process includes the near-off-far response duration. After detecting the start frame and the end frame, the two are subtracted to obtain the shooting frame number, and the near-off-far response duration is calculated according to the shooting frame number and the camera shooting frame rate.

[0402] The above scenario 7 involves the interaction between the mobile phone, the high-speed camera and the computer equipment, such as Fig. 27 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the present application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Fig. 27 ,The interaction process includes the following steps.

[0403] S71. A high-speed camera shoots the screen of the mobile phone.

[0404] S72. Remove the baffle for the close view.

[0405] S73. Display the picture in the preview interface of the mobile phone.

[0406] S74. The high-speed camera sends the captured video to be detected to the computer device.

[0407] S75. The computer device identifies whether the baffle is completely removed in the video to be detected by using a frame difference method, and uses the first frame in which the baffle is completely removed as the start frame. The computer device also identifies whether the image in the preview interface is zoomed by using the frame difference method, and uses the first frame in which the image in the preview interface stops zooming as the end frame.

[0408] As an example, the video frame detection method performed by the computer device in scene 7 is as follows: Fig.28 Please refer to Fig.28 , the method comprises the following steps:

[0409] S751. Obtain the video to be detected and the corresponding baffle image, component image, and perspective image.

[0410] The baffle diagram includes baffles, such as Fig.26As shown in Figure A in . The component diagram includes a trapezoidal component (i.e., the target component), such as Fig.26 As shown in Figure C in FIG. The distant view includes a still portrait (ie, the second object), such as Fig.26 As shown in Figure D.

[0411] S752. Perform region matching on the video frame in the video to be detected according to the position of the baffle in the baffle map to obtain the baffle region in the video frame.

[0412] According to the position features of the area where the baffle is located in the baffle map, template matching is performed in the video frame to determine the position coordinates of the baffle in the video frame, and the area where the baffle is located in the video frame (ie, the baffle area) is taken as a specific area that needs attention.

[0413] S753. Perform inter-frame difference on the baffle regions of adjacent video frames in the video to be detected to obtain third frame difference data.

[0414] S754. When the third frame difference data indicates that the baffle has moved, perform area matching on the video frame in the video to be detected according to the position of the target component in the component map to obtain the component area of ​​the target component in the video frame, and perform grayscale conversion on the video frame to obtain a third grayscale map corresponding to the video frame.

[0415] According to the positional features of the area where the target component is located in the component map, template matching is performed in the video frame to determine the position coordinates of the target component in the video frame, and the area where the target component is located in the video frame (ie, the component area) is regarded as a specific area that needs attention.

[0416] S755. Perform inter-frame difference on the component areas of the third grayscale image corresponding to adjacent video frames to obtain fourth frame difference data.

[0417] S756. The first frame in which the fourth frame difference data indicates that the outline of the target component appears is used as the first frame in which the baffle is completely removed. The target component is the component displayed after the baffle is completely removed.

[0418] S757. According to the position of the second object in the distant view, perform region matching on the video frames after the start frame in the video to be detected to obtain a fourth preset region of the second object in the video frame.

[0419] According to the position features of the area where the second object is located in the distant view, template matching is performed in the video frame to determine the position coordinates of the second object in the video frame, and the area where the second object is located in the video frame (i.e., the fourth preset area) is taken as a specific area that needs attention.

[0420] S758. Perform grayscale conversion on the video frame after the start frame in the video to be detected to obtain a fourth grayscale image corresponding to the video frame.

[0421] S759. Perform inter-frame difference on a fourth preset area of ​​a fourth grayscale image corresponding to adjacent video frames to obtain fifth frame difference data.

[0422] S760. Determine whether the area of ​​the picture outline indicated by the fifth frame difference data is smaller than a third preset area?

[0423] If yes, execute S761, if no, execute S757.

[0424] S761. Use the video frame whose fifth frame difference data indicates that the area of ​​the picture outline is smaller than the third preset area as the first frame for stopping zooming of the distant picture.

[0425] In this application Fig. 27 and Fig.28 Introducing the scene of switching from close-up shooting to long-range shooting. The implementation process and the beneficial effects achieved can be found in the above Fig.26 The description will not be repeated here.

[0426] Below through Figure 29-Figure 35 Scene 8 (the scene from receiving a trigger operation on the photo control to completing the shooting of the dynamic timer) is described.

[0427] By testing the video to be tested corresponding to the scene from when the mobile phone receives the trigger operation of the photo control to when the dynamic timer completes the shooting, the camera delay of the mobile phone can be detected. In the actual test process, the test of the camera delay scene is achieved by detecting the time corresponding to the starting frame and the time corresponding to the captured picture. The dynamic timer is a timing tool that turns on and off the lights regularly according to a time period. The dynamic timer includes a dynamically displayed light array, which includes multiple lights arranged in rows and columns. The dynamic timer controls different lights to emit light and display in a cycle according to a preset frequency. The dynamic timer can also be called a revolving lantern. The display time corresponding to the dynamic timer can be obtained by identifying the revolving lantern time.

[0428] Each light in the dynamically displayed light array may be a light emitting diode (LED), etc. The light array may be timed by the position of the dynamically displayed light. The light array may have different refresh frequencies, and the display area of ​​the light array may have different light number specifications. The embodiment of the present application does not limit the specifications and operating frequency of the light array. Fig.29 Introduce one of the light arrays. Fig.29A schematic diagram of the dynamic display of the light array in the dynamic timer is shown. The dynamic timer includes a plurality of lights displayed in rows and columns. The dynamic timer can control the lights in different positions to dynamically illuminate according to a preset frequency. The lighting of lights in different rows and columns can correspond to different times. The dynamic timer is divided into two display areas, the upper display area includes 100 lights, and the lower display area includes 10 lights. The upper display area displays the lights in a cycle of 5 consecutive lights; the lower display area displays the lights in a cycle of 1 light. Five illuminated lights are displayed in a cycle in the upper display area, corresponding to one illuminated light in the lower display area moving one position to the right. At the same moment, there are 5 illuminated lights in the upper display area and 1 illuminated light in the lower display area. The current moment can be determined based on the row and column positions of the 5 illuminated lights in the upper display area and the position of the 1 illuminated light in the lower display area. Fig.29 Taking the dynamic display in Figure A as an example, the first display moment is as follows Fig.29 As shown in Figure A, the lights with row number 90, column number 3 to row number 90, column number 7 in the upper display area light up, and the light with number 2 in the lower display area lights up, and the other lights do not light up. The time corresponding to the lights that light up is 2×100+90+3; the second display time is cycled to Fig.29 In Figure B, the lights in row number 90, column number 8 to row number 90, column number 9 in the upper display area light up, and the lights in row number 0, column number 0 to row number 0, column number 2 light up, and the light in the lower display area number 2 lights up, and the other lights do not light up. The time corresponding to the lights that are lit is 2×100+90+8; the third display time cycle is to Fig.29 In Figure C, the lights in the upper display area with row number 0, column number 3 to row number 0, column number 7 are lit, and the light with number 3 in the lower display area is lit, and the other lights are not lit. The time corresponding to the lights that are lit is 3×100+0+3; the fourth display time is cycled to Fig.29 In Figure D, the lights in the area with row number 0, column number 8 to row number 0, column number 9 in the upper display area are lit, as well as the lights in row number 1, column number 0 to row number 1, column number 2 are lit. The light with number 3 in the area below is lit, and the other lights are not lit. The time corresponding to the lit lights is 3×100+0+8.

[0429] The camera photo delay is the time difference between pressing the shutter button and the camera actually taking the picture. When detecting the camera's photo delay, the mobile phone triggers the photo control based on the test instruction to realize the automatic triggering operation of the photo control. In response to the triggering operation of the photo control, the mobile phone obtains the captured picture. In this process, the corresponding time when the photo control starts to change and the corresponding time on the captured picture are key nodes. Based on this, a dynamic timer can be set in the shooting scene, and the dynamic timer can be used by the mobile phone to take a photo to simulate the photo delay scene, and the picture taken by the mobile phone includes the dynamic timer.

[0430] The time corresponding to the dynamic timer when the photo control starts to change and the time corresponding to the dynamic timer on the captured image are obtained, and the two are subtracted to obtain the photo delay duration. The operation response process of the mobile phone to take photos of the dynamic timer can be captured by a high-speed camera to obtain a video to be detected, which includes multiple continuous video frames.

[0431] For example, Fig.30 As shown, Fig.30 Two representative video frames in the video to be detected corresponding to the scene of taking a photo of the dynamic timer are shown. It can be understood that the video to be detected in this scene also includes other video frames. The process of triggering the photo control to take a photo of the dynamic timer is as follows: using an automated method to click the photo control on the shooting interface of the camera application, such as Fig.30 As shown in Figure A; the camera control is reduced, as shown in Fig.30 As shown in Figure B; the phone will then take a picture, such as Fig.30 Figure C in .

[0432] The video to be detected is detected using the video frame detection method provided in this application to identify the key nodes where the behavior occurs in the camera operation response process, and the key nodes include the start frame (the first frame indicating that the photo control starts to change). Then determine the time of the dynamic timer corresponding to the start frame, and identify the light that glows in the captured picture to determine the time of the dynamic timer in the captured picture. The camera operation response process involves the recognition of photo controls, as well as the time recognition of the light in the video frame and the captured picture. The changes in the photo controls can be identified by the frame difference method. The following describes the recognition of photo controls and the recognition of the time of the revolving lantern.

[0433] The video frame detection method provided in the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, an operation control diagram can be obtained in advance. The operation control diagram includes a photo taking control, such as Fig.30As shown in Figure A in . The operation control diagram and the video frame including the photo control are images under the same viewing angle, and the position of the photo control in the operation control diagram is the same as its position in the video frame. When detecting whether the photo control begins to change, the area where the photo control is located in the video frame (i.e., the first preset area) can be detected without paying attention to other areas. Based on the positional features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the photo control are determined in the video frame, and the area where the photo control is located in the video frame is used as a specific area that needs to be paid attention to.

[0434] When the camera control of the mobile phone is clicked, the control will shrink, such as Fig.30 As shown in Figure B, Fig.30 Compared with Figure A in the figure, the photo control is reduced. Based on this, the present application uses the frame difference method to perform inter-frame differences on the area where the photo control of adjacent video frames in the video to be detected is located to obtain a frame difference value. When the frame difference value is greater than the preset difference value, it indicates that there is a more obvious difference between adjacent video frames, that is, the photo control begins to change. The first frame where the frame difference value is greater than the preset difference value is used as the first frame where the photo control begins to change, and the first frame where the photo control begins to change is used as the starting frame.

[0435] When using the frame difference method to perform inter-frame difference on adjacent video frames, the video frames can be first converted to grayscale and then inter-frame difference can be performed. Using grayscale images for inter-frame difference can reduce the amount of calculation and increase the data processing speed. Moreover, grayscale images can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0436] Among them, the recognition process of the photo control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0437] In an embodiment of the present application, by matching the area where the photo control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving detection accuracy and reducing the amount of data required to determine whether the pixels have changed, thereby improving detection efficiency.

[0438] There are two types of revolving lantern time recognition: one is the revolving lantern time recognition in the first frame (i.e., the starting frame) when the camera control starts to change; the other is the revolving lantern time recognition of a given high-definition picture (e.g., the captured picture); the starting frame is a video frame, and its resolution is lower than that of the captured picture. Therefore, different methods are required for the two types of revolving lantern time recognition.

[0439] The first method, for video frames, has low resolution, low brightness, unclear outlines of lights that are not emitting light, and interference such as focus lines on the dynamic timer, such as Fig.31 As shown, the focus line is a shooting auxiliary line used to help determine the horizontal degree of shooting. Fig.31 , it is shown that when shooting a dynamic timer, shooting auxiliary lines, such as horizontal lines, grid lines, etc., will appear in the preview interface. Fig.31 The shooting auxiliary lines shown in the figure will be superimposed on the preview interface. To facilitate viewing the attached image, Fig.30 Figure A in the figure does not show the shooting auxiliary lines, which will interfere with the light recognition process. Therefore, non-luminous lights are difficult to identify, while luminous lights are easy to identify. Based on this, the algorithm for identifying the time of the carousel in the video frame is as follows: first read the entire video frame of the video to be detected, save the video frame where the camera control begins to change (i.e., the starting frame), and identify the position of the luminous light in each video frame (i.e., the outline of the lit light). Then all the identified light positions are screened and deduplicated to obtain the positions of all the lights, and the carousel position marking is completed (i.e., the first coordinate matrix). The carousel position marking is as follows: Fig.32 As shown, Fig.32 This is only an exemplary description of the outline of a certain revolving light. By identifying the position of the lights in each video frame and combining the identified positions of the lights, the following can be obtained: Fig.32 The outline of the area where the revolving lantern is shown.

[0440] Then, the light position in the saved starting frame is identified to obtain the light position in the starting frame, such as Fig.33 The schematic diagram of the position of the light emitting lamp shown in FIG. Fig.33 Other places with bright lights are also shown in the figure. In practical applications, we can only focus on the lights that are shining. Match the position of the lights in the starting frame with the position mark of the revolving lantern above to get the row and column where the position of the lights in the starting frame is located. Fig.33 The light position in the start frame is placed at Fig.32 The row and column where the light is on in the start frame are obtained. The time is determined according to the row and column where the light is on in the start frame, that is, the time of the dynamic timer corresponding to the start frame is obtained.

[0441] The second method is to take pictures, such as Fig.30 As shown in Figure C, the resolution is high, the outline of the lamp is clear, and there are no interference factors. The position of the non-emitting lamp can be directly identified and the position marking is completed. Then the position of the emitting lamp is identified, the position matching is completed, and the row and column where the emitting lamp is located are obtained. Fig.30 The time corresponding to the light shown in Figure C is 5×100+90+3.

[0442] In the embodiment of the present application, the first method is used to identify the time of the dynamic timer in the video frame. The dynamic timer controls the light to display in a cycle according to a preset frequency, and the outline of the lighted light can be detected by color recognition. Each video frame in the video to be detected is traversed, and color recognition is performed on each video frame to obtain the outline of the lighted light in each video frame; the outlines of the lighted lights in multiple video frames of the video to be detected are deduplicated and superimposed together to construct the outlines of all the lights in the dynamic timer and obtain the first coordinate matrix of the dynamic timer. Then, color recognition is performed on the starting frame to obtain the outline of the lighted light in the starting frame; the position of the outline in the starting frame is matched with the first coordinate matrix to obtain the position of the outline in the first coordinate matrix, and then the time of the dynamic timer corresponding to the starting frame is determined.

[0443] In the embodiment of the present application, the second method is used to identify the time of the dynamic timer in the captured picture. The captured picture is subjected to shape recognition according to a preset shape to obtain the positions of all unlit lamps in the captured picture, and a second coordinate matrix is ​​constructed according to the positions of all unlit lamps. The captured picture is subjected to color recognition to obtain the outline of the lit lamp in the picture; the outline of the lit lamp is matched in the second coordinate matrix to obtain the position of the outline in the second coordinate matrix, and then the time of the dynamic timer in the picture is determined.

[0444] Among them, the preset shape is a configuration parameter set by technical personnel in this field based on the physical shape of the lamp used in a large number of experiments, such as a triangle, square, rectangle, circle, ellipse, etc. The lamp can only be identified based on the shape, and the embodiments of the present application are not limited to this.

[0445] In the embodiment of the present application, different methods are used to identify the state (glowing or not) and time of the lamp for low-resolution video frames and high-definition pictures taken, respectively. For low-resolution video frames, the entire video to be detected is traversed to identify the glowing lamp in each video frame, so as to construct the coordinate matrix of the dynamic timer, and then the contour of the glowing lamp in the starting frame is identified, and the contour is matched with the coordinate matrix to obtain the row and column where the glowing lamp in the starting frame is located, so as to obtain the time corresponding to the starting frame. For the high-definition pictures taken, the position of the non-glowing lamp is directly identified, the coordinate matrix is ​​constructed, and then the position of the glowing lamp is identified, and the template is matched with the coordinate matrix to obtain the row and column where the glowing lamp in the picture is located, so as to obtain the time corresponding to the taken picture. By identifying the digitized video, the row and column where the lamp in the picture and the starting frame is located are determined, and automatic reading is realized. No manual detection is required, and the automatic detection process of the end frame is realized, which improves the efficiency and accuracy of video frame detection.

[0446] The above scenario 8 involves the interaction between the mobile phone, high-speed camera and computer equipment, such as Fig.34 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the present application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Fig.34 ,The interaction process includes the following steps.

[0447] S81. A high-speed camera films the screen of a mobile phone.

[0448] S82. Trigger the photo control in the mobile phone shooting interface.

[0449] S83. The mobile phone takes a picture in response to the trigger operation.

[0450] S84. The high-speed camera sends the captured video to be detected to the computer device.

[0451] S85. The mobile phone sends the captured image to the computer device.

[0452] S86. The computer device identifies whether the camera control starts to change in the video to be detected by using a frame difference method, and uses the first frame in which the camera control starts to change as the starting frame. The computer device also identifies the time of the revolving lantern in the starting frame and the time of the revolving lantern in the captured picture.

[0453] As an example, the video frame detection method performed by the computer device in scene 8 is as follows: Fig.35 Please refer to Fig.35 , the method comprises the following steps:

[0454] S861. Obtain the video to be detected and the corresponding operation control diagram.

[0455] The operation control diagram includes camera controls, such as Fig.30 As shown in Figure A.

[0456] S862. According to the position of the camera control in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the camera control in the video frame.

[0457] Based on the position features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the photo control in the video frame, and the area where the photo control is located in the video frame (i.e., the first preset area) is taken as a specific area that needs attention.

[0458] S863. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0459] S864. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0460] S865. Determine whether the frame difference is greater than a preset difference value?

[0461] If yes, execute S866, if no, execute S862.

[0462] S866. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the camera control to start changing.

[0463] S867. Perform color recognition on each video frame in the video to be detected to obtain the outline of the light in each video frame.

[0464] S868. Construct the first coordinate matrix of the dynamic timer according to the outline of the glowing lights in each video frame.

[0465] S869. Perform color recognition on the first frame in which the camera control starts to change, and obtain a first outline; the first outline is the outline of the light that emits light in the first frame in which the camera control starts to change.

[0466] S870. Perform position matching in the first coordinate matrix according to the position of the first contour in the first frame where the camera control starts to change, and obtain the position of the first contour in the first coordinate matrix.

[0467] S871. Determine, according to the position of the first contour in the first coordinate matrix, the first time indicated by the dynamic timer in the first frame in which the camera control starts to change.

[0468] S872. Perform shape recognition on the captured image according to a preset shape to obtain a second coordinate matrix of the dynamic timer; the preset shape is the physical shape of the lamp, and the second coordinate matrix indicates the position of the unlit lamp.

[0469] S873. Perform color recognition on the captured image to obtain a second outline; the second outline is the outline of the light in the captured image.

[0470] S874. Perform position matching in the second coordinate matrix according to the position of the second contour in the captured image to obtain the position of the second contour in the second coordinate matrix.

[0471] S875. Determine the second time indicated by the dynamic timer in the captured image according to the position of the second contour in the second coordinate matrix.

[0472] In this application Fig.34 and Fig.35The following describes the process from receiving a trigger operation on the camera control to completing the dynamic timer shooting (shutterlag), which may include high brightness and low brightness. The implementation process and the beneficial effects achieved can be found in the above Figure 29-Figure 33 The description will not be repeated here.

[0473] Below through Figure 36-Figure 39 Scenario 9 (scenario of executing a process in response to a trigger operation on a photo-taking control) is described.

[0474] The camera shooting rate refers to the number of frames that the camera can shoot per second. The camera shooting rate scenario can be simulated by the changes in the screen display in the preview interface and the display changes of the dynamic timer during the mobile phone's dynamic timer shooting process. In the actual test process, the shooting rate scenario is tested by detecting the start frame and the end frame. When detecting the camera shooting rate, the mobile phone triggers the photo control based on the test instruction to realize the automatic triggering operation of the photo control. In response to the triggering operation of the photo control, the screen display will change from black to bright (or from black to bright) and the dynamic timer display will freeze in the preview interface of the mobile phone. The order of the screen display changing from black to bright and the dynamic timer display freezing is uncertain. Therefore, it is necessary to identify the screen display changing from black to bright and the dynamic timer display freezing. In this process, the key nodes are the start of the change of the photo control, the screen changing from black to bright, and the dynamic timer display freezing. The dynamic timer includes a dynamically displayed light array, which includes a plurality of lights arranged in rows and columns. For the description of the dynamic timer, please refer to the above Fig.29 The description in will not be repeated here.

[0475] In the embodiment of the present application, the first frame where the photo control starts to change is taken as the starting frame, and the video frame sorted after the first frame where the screen changes from black to bright and the last frame where the dynamic timer is stuck is taken as the ending frame, and the camera shooting rate can be determined according to the starting frame and the ending frame. The present application can use a high-speed camera to shoot the operation response process of the mobile phone executing the dynamic timer to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0476] For example, Fig.36 As shown, Fig.36 Four representative video frames in the video to be detected corresponding to the scene of taking a photo of the dynamic timer are shown. It can be understood that the video to be detected in this scene also includes other video frames. The process of triggering the photo control to take a photo of the dynamic timer is as follows: using an automated method to click the photo control on the shooting interface of the camera application, such as Fig.36 As shown in Figure A; the camera control is reduced, as shown in Fig.36 As shown in Figure B, then Fig.36As shown in Figure C, the camera controls return to normal; next, the preview interface of the phone will have black frames and frames that change from black to light. The preview interface will go through the following steps: Fig.36 The darkening process shown in Figure C, and the Fig.36 The brightening process is shown in Figure D. The lights displayed dynamically in the preview interface may appear to be stuck, such as Fig.37 As shown, Fig.37 Four consecutive video frames are shown, Fig.37 Figure A in Fig.37 Figure B and Fig.37 The video frame shown in Figure C shows that the lights displayed dynamically in the video frame appear to be stuck until Fig.37 As shown in Figure D.

[0477] The video to be detected is detected by the video frame detection method provided in the present application, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the photo control starts to change) and the end frame (the first frame indicating that the screen changes from black to bright and the video frame sorted after the last frame where the dynamic timer displays a freeze). The camera operation response process involves the identification of photo controls, the identification of the screen changing from black to bright, and the identification of the dynamic timer display freeze. The changes in the photo controls can be identified by the frame difference method. The process of the screen changing from black to bright is identified by brightness comparison. The following describes the identification of photo controls, the identification of the screen changing from black to bright, and the identification of the dynamic timer display freeze.

[0478] The video frame detection method provided in the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, an operation control diagram can be obtained in advance. The operation control diagram includes a photo taking control, such as Fig.36 As shown in Figure A in the figure, the computer device pre-acquires an operation control diagram with a photo control. The operation control diagram and the video frame including the photo control are images under the same viewing angle, and the position of the photo control in the operation control diagram is the same as its position in the video frame. When detecting whether the photo control begins to change, the area where the photo control is located in the video frame (i.e., the first preset area) can be detected without paying attention to other areas. Based on the positional features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the photo control are determined in the video frame, and the area where the photo control is located in the video frame is used as a specific area that needs to be paid attention to.

[0479] When the camera control of the mobile phone is clicked, the control will shrink first and then return to normal. Fig.36 As shown in Figure B, Fig.36 Compared with Figure A in the figure, the camera control is reduced, and then Fig.36As shown in Figure C in the figure, the camera control returns to normal. Based on this, the present application uses the frame difference method to perform inter-frame differences on the area where the camera control of adjacent video frames in the video to be detected is located to obtain a frame difference value. When the frame difference value is greater than the preset difference value, it indicates that there is a relatively obvious difference between adjacent video frames, that is, the camera control begins to change. The first frame whose frame difference value is greater than the preset difference value is taken as the first frame where the camera control begins to change, and the first frame where the camera control begins to change is taken as the starting frame.

[0480] When using the frame difference method to perform inter-frame difference on adjacent video frames, the video frames can be first converted to grayscale and then inter-frame difference can be performed. Using grayscale images for inter-frame difference can reduce the amount of calculation and increase the data processing speed. Moreover, grayscale images can eliminate the interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0481] Among them, the recognition process of the photo control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0482] In an embodiment of the present application, by matching the area where the photo control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving detection accuracy and reducing the amount of data required to determine whether the pixels have changed, thereby improving detection efficiency.

[0483] In the scene where the camera takes pictures of the dynamic timer, after detecting the starting frame in the video to be detected, the computer device will then detect the picture in the preview interface of the mobile phone, which includes the dynamic display light. The preview interface will have black frames and frames that change from black to light. The preview interface will go through the following steps: Fig.36 The darkening process shown in Figure C, and the Fig.36 The brightening process is shown in Figure D. The lights displayed dynamically in the preview interface may appear to be stuck, such as Fig.37 As shown, Fig.37 Four consecutive video frames are shown, Fig.37 Figure A in Fig.37 Figure B and Fig.37 The video frame shown in Figure C shows that the lights displayed dynamically in the video frame appear to be stuck until Fig.37 As shown in Figure D. Compared with Fig.37 Figure C in the figure, Fig.37 The D diagram in the figure shows that the position of the light is changing from unchanged to changed. Fig.37 Figure C in the figure shows the last frame of the dynamic timer display freeze.

[0484] It should be noted that Fig.37 is located in Fig.36 The video frame after the B picture in can be Fig.36 After the C figure in Fig.36 That is, the video frame where the screen changes from black to bright may be before or after the video frame where the dynamic timer shows a freeze.

[0485] The process of brightening the screen of the preview interface is a gradual process, which is brightening step by step. The behavior of the screen changing from black to bright can be identified by the brightness change of the screen from black to bright. The following two methods can be used to identify the screen changing from black to bright. The first method is to pre-set a black threshold (i.e., the first threshold). If the brightness of the video frame is less than the black threshold, the video frame is judged as black. At the same time, a black-to-bright threshold (i.e., the second threshold) is pre-set. When the current video frame is judged as black, if the brightness difference obtained by subtracting the brightness of the previous video frame from the brightness of the current video frame is greater than the black-to-bright threshold, the current video frame is used as the first frame that changes from black to bright. The second method is to count the brightness of multiple video frames in the video to be detected, average the brightness of multiple video frames, and determine the brightness threshold based on the average value. For example, the average value is used as the brightness threshold, or the brightness threshold is obtained by multiplying the average value by a coefficient. In the process of the brightness of multiple video frames after the starting frame gradually increasing, the video frame with a brightness greater than the brightness threshold is used as the first frame of the screen changing from black to bright. This method can adapt to different lighting conditions, does not rely on a fixed threshold, and increases the applicable scenarios of the detection method.

[0486] Among them, the first threshold and the second threshold are configuration parameters set by technical personnel in this field based on the black threshold and the black-to-light threshold adopted in a large number of experimental processes. As long as it is possible to determine whether a video frame is a black video frame and whether a video frame is a black-to-light video frame, the embodiments of the present application are not limited to this.

[0487] In the embodiment of the present application, the brightness of the video frame is judged by the set black threshold, and the brightness difference of adjacent video frames is judged by the set black-to-light threshold, so that the video frame that changes from black to light can be detected in multiple video frames after the start frame. Manual detection is not required, and the automatic detection process of the video frame that changes from black to light is realized, thereby improving the efficiency and accuracy of video frame detection.

[0488] Furthermore, a brightness threshold can be set according to the brightness of multiple video frames, and the brightness threshold is related to the brightness of multiple video frames. For example, a video frame that changes from black to bright can be detected in multiple video frames after the starting frame, thereby increasing the diversity of implementation methods of the judgment process.

[0489] The dynamic timer controls the light to display in a cyclic manner at a preset frequency, and the outline of the lighted light can be detected by color recognition. Color recognition is performed on each video frame after the start frame to obtain the outline of the lighted light in each video frame (i.e., the third outline). When continuous video frames with the same outline are detected, it means that the dynamic timer display is stuck, and the detection is continued until a video frame with an outline different from the continuous video frame is detected, and multiple video frames with the same outline are obtained. The video frames with the same outline are used as target video frames to obtain multiple target video frames, and the outline of the next video frame of the last frame of the multiple target video frames is different from the outline of the multiple target video frames. The last frame of the multiple target video frames is used as the last frame of the dynamic timer display stuck.

[0490] In the embodiment of the present application, the outline of the light in the video frame is identified, and then multiple target video frames with the same outline of the light are determined, and the last frame in the multiple target video frames is used as the last frame of the dynamic timer display freeze. Without manual detection, the automatic detection process of the video frame of the dynamic timer display freezes is realized, which improves the video frame detection efficiency and detection accuracy.

[0491] After detecting the first frame where the screen changes from black to bright and the last frame where the dynamic timer displays a freeze, the video frame that is sorted later than the first frame where the screen changes from black to bright and the last frame where the dynamic timer displays a freeze is taken as the end frame. No manual detection is required, thus realizing an automatic detection process for the end frame and improving the efficiency and accuracy of video frame detection.

[0492] In the embodiment of the present application, after the start frame and the end frame are detected, the two are subtracted to obtain the number of shooting frames, and the camera shooting rate is determined according to the number of shooting frames.

[0493] The above scenario 9 involves the interaction between the mobile phone, the high-speed camera and the computer equipment, such as Fig.38 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the present application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Fig.38 ,The interaction process includes the following steps.

[0494] S91. A high-speed camera shoots the screen of a mobile phone.

[0495] S92. Trigger the photo control in the mobile phone shooting interface.

[0496] S93. The mobile phone responds to the trigger operation and displays the picture in the preview interface.

[0497] S94. The high-speed camera sends the captured video to be detected to the computer device.

[0498] S95. The computer device identifies whether the camera control starts to change in the video to be detected by using a frame difference method, and uses the first frame where the camera control starts to change as the start frame. The computer device also identifies the first frame where the screen changes from black to bright in the preview interface and the last frame where the dynamic timer displays a freeze, and uses the video frame that is sorted later as the end frame.

[0499] As an example, the video frame detection method performed by the computer device in scene 9 is as follows: Fig.39 Please refer to Fig.39 , the method comprises the following steps:

[0500] S951. Obtain the video to be detected and the corresponding operation control diagram.

[0501] The operation control diagram includes camera controls, such as Fig.36 As shown in Figure A.

[0502] S952. Perform region matching on the video frame in the video to be detected according to the position of the camera control in the operation control diagram, and obtain a first preset region of the camera control in the video frame.

[0503] Based on the position features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the photo control in the video frame, and the area where the photo control is located in the video frame (i.e., the first preset area) is taken as a specific area that needs attention.

[0504] S953. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0505] S954. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0506] S955. Determine whether the frame difference is greater than a preset difference value?

[0507] If yes, execute S956, if no, execute S952.

[0508] S956. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the camera control to start changing.

[0509] S957. Detect the video frame located after the starting frame in the video to be detected. When the brightness of the current video frame is less than a first threshold and the brightness difference between the current video frame and the previous video frame of the current video frame is greater than a second threshold, use the current video frame as the first frame where the picture changes from black to bright; or, determine the brightness threshold according to the brightness of each video frame in the video to be detected; in the process where the brightness of multiple video frames located after the starting frame gradually increases, use the video frame whose brightness is greater than the brightness threshold as the first frame where the picture changes from black to bright.

[0510] S958. Perform color recognition on each video frame after the start frame in the video to be detected to obtain a third outline of each video frame; the third outline is the outline of the light in the video frame after the start frame.

[0511] S959. Determine multiple target video frames in each video frame after the start frame; wherein the third contours of the multiple target video frames are the same, and the third contour of the next video frame of the last frame in the multiple target video frames is different from the third contours of the multiple target video frames.

[0512] S960. Use the last frame among the multiple target video frames as the last frame of the dynamic timer display freeze.

[0513] S961. The video frame that is sorted after the first frame where the screen changes from black to bright and the last frame where the dynamic timer displays a freeze is used as the end frame corresponding to the camera operation response process.

[0514] In this application Fig.38 and Fig.39 Introducing the scene of executing the process (shot2preview) in response to the trigger operation of the photo control, which can include high light and low light. The implementation process and the beneficial effects achieved can be seen in the above Figure 36-Figure 37 The description will not be repeated here.

[0515] Fig.40 An exemplary flow chart of a video frame detection method provided in an embodiment of the present application.

[0516] Please refer to Fig.40 , Fig.40 is a flow chart of a video frame detection method provided in an embodiment of the present application, such as Fig.40 As shown, the method comprises the following steps:

[0517] S111. Obtain a video to be detected and a corresponding operation control diagram. The video to be detected is obtained by photographing the screen of a target terminal that executes a camera operation response process. The operation controls in the operation control diagram are used to start the camera operation response process.

[0518] In a possible implementation, the operation controls include a camera application icon, a photo taking control, an image echo control, a return control, a flip control, and a video recording control. Figure 4 Figure A in the above Fig. 9 Figure A in the above Fig.13 Figure A in the above Fig.17 Figure A in the above Fig. 20 Figure A and the above Fig.23 Figure A in .

[0519] S112. Performing region matching on the video frame in the video to be detected according to the position of the operation control in the operation control map, and obtaining a first preset region of the operation control in the video frame.

[0520] In a possible implementation, the operation control diagram and the video frame including the operation control are images under the same viewing angle, and the position of the operation control in the operation control diagram is the same as its position in the video frame. Based on the positional features of the area where the operation control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the operation control are determined in the video frame, and the area where the operation control is located in the video frame is used as a specific area that needs attention (i.e., the first preset area). When detecting whether the operation control begins to change, the area where the operation control is located in the video frame (i.e., the first preset area) can be detected without paying attention to other areas.

[0521] The first preset area can refer to the above Figure 4 The area where the camera application icon is located in Figure A, the above Fig. 9 The area where the camera control is located in Figure A, the above Fig.13 The area where the image echo control is located in Figure A, the above Fig.17 In the A figure in the figure, the area where the control is located is returned. Fig. 20 The area where the flip control is located in Figure A and the above Fig.23 The area where the video control is located in Figure A.

[0522] S113. Detect pixel changes in a first preset area of ​​adjacent video frames in the video to be detected, and obtain the first frame where the operation control starts to change. The first frame where the operation control starts to change is the starting frame corresponding to the camera operation response process.

[0523] In one possible implementation, a frame difference method is used to perform inter-frame difference on a first preset area of ​​adjacent video frames in the video to be detected to obtain a frame difference value, which reflects the pixel change in the first preset area, thereby detecting the video frame where the operation control begins to change, and taking the first frame where the operation control begins to change as the starting frame corresponding to the camera operation response process.

[0524] S114. Detect pixel changes in the video frame after the start frame or in the adjacent video frames in the video to be detected to obtain a response end video frame, where the response end video frame is an end frame corresponding to the camera operation response process.

[0525] In a possible implementation, after the start frame is detected, the pixel changes in the video frame after the start frame in the video to be detected are detected to determine whether a picture appears, or the pixel changes in the adjacent video frames after the start frame in the video to be detected are detected to detect whether there is a difference between adjacent video frames and determine whether the image appears completely. For different detection scenarios, a detection method that matches the detection scenario can be selected to detect the end frame.

[0526] The relevant contents of S111-S114 can refer to the description of the above video frame detection process, which will not be repeated here. Figure 4-Figure 39 The description of the relevant contents will not be repeated here.

[0527] In the present application, a video to be detected is obtained by shooting the screen of the target terminal when executing the camera operation response process, and the video to be detected includes multiple continuous video frames. The camera operation response process is the process of the target terminal operating different shooting functions. Different operation controls will be used in the operation process, and the operation control diagram corresponding to the camera operation response process can be obtained in advance. The operation control diagram includes operation controls, and different camera operation response processes correspond to different operation controls. The operation control diagram and the video frame including the operation control are images under the same viewing angle, and the position of the operation control in the operation control diagram is the same as its position in the video frame. Based on this, according to the position of the operation control in the operation control diagram, regional matching is performed in the video frame to obtain the first preset area of ​​the operation control in the video frame. Whether a pixel change occurs in the first preset area in adjacent video frames, that is, whether a relatively obvious difference occurs in the area where the operation control is located in the adjacent video frames, so as to detect the video frame where the operation control starts to change, and the first frame where the operation control starts to change is the starting frame corresponding to the camera operation response process. By matching the area where the operation control is located in the video frame, targeted regional detection can be achieved, the detection accuracy is improved, and the amount of data when judging whether the pixel changes is reduced, thereby improving the detection efficiency. After detecting the starting frame, the pixel changes in the video frames after the starting frame or the adjacent video frames in the detected video are detected to determine whether there is a picture or whether the image appears completely. The ending frame is detected by detecting whether there is a pixel change in the video frame or whether there is a difference between adjacent video frames, thereby realizing the automatic detection process of the video frames and improving the detection efficiency and accuracy of the video frames.

[0528] Combine the following Fig.41 The video frame detection process is introduced.

[0529] For the above different scenarios, an operation control diagram including different operation controls can be pre-acquired, and the operation control includes one or more of a camera application icon, a photo control, an image echo control, a return control, a flip control, and a video control. One or more of an operation response diagram, a shooting interface diagram, a baffle diagram, a component diagram, and a distant view diagram can also be pre-acquired. Among them, the operation response diagram, the shooting interface diagram, the baffle diagram, the component diagram, and the distant view diagram can be screenshots of the shooting content in the detection scene. Different configuration parameters are pre-set, and the configuration parameters include: a preset difference value for judging whether the operation control starts to change, a preset similarity for judging whether the color of the camera application icon starts to change, a first number for judging whether the picture appears in the preview interface, a second number for judging whether the picture is blurred in the preview interface, a first preset area for judging whether the thumbnail appears completely or stops zooming, a second preset area for judging whether the display image is completely opened or stops zooming, a preset duration, a third preset area for judging whether the distant view picture in the preview interface stops zooming, a preset shape, a first threshold for judging whether the picture is a black frame, and a second threshold for judging whether the picture changes from black to bright frame. The preset difference value is proportional to the flicker degree of the camera in the terminal.

[0530] In this application, the above-mentioned camera operation response process involves multiple scenes, and each scene needs to identify the start frame and the end frame. The recognition of video frames can be divided into the following types: operation control recognition, picture recognition in the preview interface (referring to the first frame of the picture that has just appeared, and the picture has not been fully displayed), complete picture recognition (referring to the picture has stopped zooming), picture recognition from black to bright, baffle complete removal recognition, and carousel recognition (including recognition of lights in video frames and lights in captured pictures).

[0531] The operation control recognition process is as follows: through frame difference analysis, the computer device recognizes the state change of the operation control, and takes the first frame where the operation control starts to change as the starting frame. When the operation control changes from a static state to a clicked state, the area where the operation control is located will change significantly. The frame difference method is used to calculate the difference between adjacent video frames to detect this change and complete the recognition of the video frame where the operation control starts to change. There is no need for manual template matching based on the judgment criteria, and it is not affected by the style and position changes of the operation control. It is simple and efficient.

[0532] The picture recognition process in the preview interface is as follows: edge detection is performed using an edge detection operator to identify the characteristic content of the picture in the preview interface, for example, white lines, and the first frame of the video frame whose line feature indicates that the number of lines is greater than the first number is used as the first frame of the picture appearing in the preview interface. When the picture just appears in the preview interface, a large amount of edge information will appear in the picture. The Canny edge detection algorithm can be used to detect the lines of the picture. This detection algorithm relies on edge features and can effectively detect these edges. It is not affected by the specific picture content, thereby improving the accuracy of detection.

[0533] The complete picture recognition process is as follows: By analyzing the frame difference, the computer device can detect the picture zooming process. When it is detected that the large-area frame difference stops appearing, it is judged that the picture zooming is completed. When the picture is zooming, the difference between adjacent video frames is large, and after the zooming is completed, the difference between adjacent video frames is small. The frame difference method can be used to detect the video frame where the picture stops zooming. There is no need for manual template matching based on the judgment criteria, and it can adapt to video frames of different resolutions, which increases the applicable scenarios of the detection method.

[0534] The process of identifying the image changing from black to bright is as follows: by setting the threshold of the black frame (i.e., the first threshold) and the threshold of the image changing from black to bright, the brightness difference of adjacent video frames can be judged to identify the transition frame where the image changes from black to bright. Alternatively, by counting the brightness information of all video frames, setting the brightness threshold, and detecting the brightness of the current frame, when it exceeds a certain proportion of the brightness threshold, it means that the image has become brighter, and then the video frame where the image changes from black to bright is detected. This method can adapt to different lighting conditions, does not rely on a fixed threshold, and increases the applicable scenarios of the detection method.

[0535] The identification process of the baffle being completely removed is as follows: the movement process of the baffle is detected by grayscale analysis and frame difference method, and the change of the component state is detected when the baffle starts to move. The video frame can be converted into a grayscale image, and the grayscale images of adjacent video frames are inter-frame differentiated to detect whether the baffle has moved and identify the behavior of the baffle being completely removed from the screen of the preview interface. If the baffle moves and the component outline does not appear, the video frame is continued to be processed to detect whether the component outline appears. If the baffle moves and the component outline appears, the video frame is continued to be processed to detect whether the screen appears in the preview interface.

[0536] The process of identifying the time of the revolving lantern is as follows: different methods are used to identify the state of the revolving lantern (glowing lights and non-glowing lights) and the time of the revolving lantern for low-resolution video frames and high-definition pictures respectively.

[0537] Based on the multiple recognition processes and various detection scenarios introduced above, the key node detection method implemented by the computer device is further explained. The method is applied to the camera frame counting process and automatically detects the key nodes in the camera video through feature analysis and recognition algorithms.

[0538] Please refer to Fig.41 , Fig.41 is a flow chart of a detection method provided in an embodiment of the present application, such as Fig.41 As shown, the method comprises the following steps:

[0539] S211, obtaining a target video, an operation control diagram, and configuration parameters.

[0540] The input part of the computer device that executes the video frame detection method provided by the present application includes: target video, detection scene (i.e., camera operation response process), scene screenshots corresponding to each detection scene, such as operation control diagram, operation response diagram, shooting interface diagram, baffle diagram, component diagram and perspective diagram. The input part may also include configuration parameters and related interfaces that need to be adjusted for actual detection. The configuration parameters are thresholds obtained by pre-training in a large number of experimental processes.

[0541] The target video may be a video to be detected in the application phase or a video in the pre-training phase.

[0542] The configuration parameters include a threshold for determining whether the pixels between adjacent video frames have changed, so as to determine whether important visual changes have occurred in adjacent video frames. For example, a preset difference value for determining whether the operation control has begun to change, a preset similarity for determining whether the color of the camera application icon has begun to change, a first preset area for determining whether the thumbnail has appeared completely or has stopped zooming, a second preset area for determining whether the display image has been fully opened or has stopped zooming, a third preset area for determining whether the distant view in the preview interface has stopped zooming, and a second threshold for determining whether the image has changed from a black frame to a bright frame. The configuration parameters also include a threshold for determining whether the pixels in the video frame have changed, so as to determine whether important visual changes have occurred in the video frame. For example, a first number for determining whether the image appears in the preview interface, a second number for determining whether the image is blurred in the preview interface, and a first threshold for determining whether the image is a black frame.

[0543] S212: Match a first preset area of ​​the operation control in the video frame of the target video according to the position of the operation control in the operation control diagram.

[0544] The first preset area is a specific area that needs to be paid attention to. Targeted area detection is performed according to the coordinates of the specific area, which can improve detection accuracy and detection efficiency.

[0545] S213, reading the video frames in the target video frame by frame, performing grayscale processing on the video frames, and obtaining a grayscale image.

[0546] The target video is read frame by frame, decomposed into separate video frames, and processed. Before inter-frame difference is needed, the video frame is converted into a grayscale image, which can reduce the computational complexity and effectively suppress the stroboscopic phenomenon, thereby bypassing the more complex destroboscopic algorithm for subsequent processing.

[0547] S214: Using a frame difference method, perform inter-frame difference on a first preset area of ​​the grayscale image of adjacent video frames to obtain a frame difference value.

[0548] According to different detection scenarios, key features can be extracted. The frame difference method is used to calculate the inter-frame difference between the first preset areas of the grayscale images of adjacent video frames to detect whether the area where the operation control is located has changed. The first frame with a frame difference value greater than the preset difference value is taken as the first frame where the operation control begins to change.

[0549] For the camera application startup scenario, the color change of the video frame can be detected before the grayscale conversion is performed on the video frame to detect whether the color of the camera application icon starts to change.

[0550] S215: Determine whether adjacent video frames have changed according to the frame difference value and the preset difference value in the configuration parameters?

[0551] If not, continue to execute S214. If yes, execute S216.

[0552] Detect whether there is a change between adjacent video frames, compare the frame differences between consecutive video frames, that is, determine whether the pixels in adjacent video frames have changed, and compare the feature data of the frame differences with the configuration parameters to detect whether there is a significant change between adjacent video frames. If a change is detected between adjacent video frames, it means that the operation control begins to change. The first frame where the operation control begins to change is taken as the starting frame, and the video frames after the starting frame are further analyzed.

[0553] S216 , continue to perform grayscale processing on the video frames after the first frame where the operation control starts to change, to obtain a grayscale image.

[0554] S217: Detect pixel changes in the grayscale image of the video frame or the adjacent video frame after the first frame where the operation control starts to change, and determine whether a picture appears or whether the image appears completely.

[0555] If not, continue to execute S216, and if so, execute S218.

[0556] According to different detection scenarios, key features can be extracted. For example, edge detection algorithms are used to extract line features to determine whether a picture appears; frame difference methods are used to calculate the inter-frame differences between adjacent video frames to determine whether the image appears completely.

[0557] S218: Output the frame number of the first frame where the operation control starts to change, and the frame number of the video frame where the picture appears or the frame number of the video frame where the image appears completely.

[0558] The first frame where the operation control starts to change is the start frame. The frame number of the video frame where the screen appears or the video frame where the image appears completely is taken as the video frame where the response ends, and the video frame where the response ends is taken as the end frame.

[0559] By judging whether the pixels in the video frame have changed, it is detected whether the picture appears in the preview interface. If the number of lines corresponding to the pixel changes in the video frame exceeds the first number, the frame number of the video frame is output. The frame difference method is used to judge whether the image appears completely. If it is detected that the frame difference data between adjacent video frames indicates that the area of ​​the picture outline is smaller than the preset area, it means that the difference between adjacent video frames is smaller, the image appears completely, that is, the image stops scaling, and the frame number of the video frame is output.

[0560] Exemplarily, taking the switch from rear-camera shooting to front-camera shooting as an example, the switching scene is explained in combination with the above S211-S218. The key nodes that need to be detected in the switching scene include the first frame where the flip control begins to change, and the first frame where the picture appears in the preview interface. For the switching scene, a screenshot of the detection target is obtained, which is used for regional positioning and identification in video frame processing. The detection targets are matched, and the detection targets include the flip control and the detection objects in the switched picture.

[0561] According to the position of the flip control in the operation control diagram, the flip control is regionally located. According to the position of the detection object (for example, a still picture) in the switched screen, the detection object is regionally located. After completing the regional matching, the video to be detected is traversed frame by frame, and after the adjacent video frames are grayed, the frame difference is calculated for the positioning area of ​​the adjacent video frames. When the frame difference value is greater than the preset difference value, it means that the flip control begins to change, and the first frame where the flip control begins to change is used as the starting frame. After the start frame detection is completed, for the video frame located after the start frame, the edge detection algorithm is used to draw the differential edge, the white lines in the video frame are detected, and the number of lines is calculated by the relevant computer vision method. When the number of lines is lower than the lower threshold (that is, less than the second number), it is determined that the corresponding image becomes blurred, indicating that the screen starts to switch from the screen of the rear camera. When the number of lines is higher than the upper threshold (that is, greater than the first number), it is determined that the corresponding image is clear again, indicating that the screen is switched to the screen of the front camera, and the clear first frame is used as the end frame.

[0562] Furthermore, the detection scenario also includes the implementation of specific functions. Specific functions include the time recognition of the revolving lantern and the state recognition of the revolving lantern. For low-resolution video frames, attention is paid to brightness changes and specific shapes (for example, the circular outline of the lamp); for high-resolution images (such as high-definition pictures taken), the state of the lamp (that is, whether the lamp is lit) and the position are directly recognized. For a description of the time recognition of the revolving lantern and its application scenarios, please refer to the above Figure 29-Figure 39 , I will not go into details here.

[0563] Specific functions also include baffle movement recognition, which can determine whether the baffle is completely removed by analyzing the difference in images before and after the baffle. For a description of baffle recognition and application scenarios, please refer to the above Figure 26-Figure 28 , I will not go into details here.

[0564] The configuration parameters are threshold values ​​obtained through pre-training during a large number of experiments. They can be obtained by executing S211-S218 above. In S211, the pre-training stage needs to collect videos in various detection scenarios. Collect digital image data containing the detection target, such as screenshots of operating controls, pictures of the first and second objects detected, screenshots of baffles, screenshots of components, distant pictures, etc. These data are used for subsequent automated target capture and recognition. The collected video data has an authentication label of the detection scene, and the video data in each detection scene should cover various possible test environments, such as different lighting conditions, different backgrounds, etc.

[0565] In the pre-training stage, based on the digital image data of the detection target collected in the video collection process, the coordinates of the specific areas of interest in the video frame are automatically calibrated manually or through methods such as template matching. Regional positioning can reduce the amount of data processing when analyzing specific areas in the video frame later.

[0566] After the target video is processed through the above S211-S218, the recognition result output by S218 can be compared with the actual situation to verify the accuracy of the recognition result. According to the difference between the recognition result and the actual situation, the algorithm parameters are adjusted, for example, the threshold value, color feature extraction method, area matching method, etc. set in the configuration parameters, to improve the recognition accuracy and processing speed.

[0567] The embodiment of the present application provides a camera video key node matching algorithm based on template matching and edge detection, which realizes the automatic detection process of camera video key nodes. Based on image grayscale processing and feature matching, feature analysis and recognition are performed on video data, and automatic detection can be performed for the occurrence of some specific events in the video data. When detecting key nodes, video frames with pixel changes in specific areas can be detected by automatically matching specific areas in the video data. Compared with manual detection, the detection efficiency and adaptability of the detection scene are improved, and the accuracy and consistency of the key nodes are improved. The detected key nodes can be used for camera video key duration testing, which improves the reliability and convenience of the camera performance test process, and contributes to the automation of the camera performance test process (for example, the duration of the photo process).

[0568] In the embodiments of the present application, subjective and objective tests and log analysis can be used for relevant detection scenarios to determine whether the video frame detection scheme provided in the present application is the same. For test schemes similar to the detection and identification of the duration of a photo process and a video frame automation test system, the same fault or specific sample is input into the detection system and the computer device of the present application. The specific sample is detected by the video frame detection method provided in the present application to obtain the detection result, and the detection result of the test system is captured. The detection results of the two are compared. If the detection results are highly consistent, it can be determined that the test system may have adopted the video frame detection method provided in the present application. Furthermore, the log of the test system can be obtained, and by analyzing the log, it can be determined whether the test scheme adopted by the test system is the same as the video frame detection method provided in the present application.

[0569] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0570] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. A video frame detection method, characterized in that: The method comprises: Acquire a video to be detected and a corresponding operation control diagram, wherein the video to be detected is obtained by shooting a screen of a target terminal that executes a camera operation response process, and the operation control in the operation control diagram is used to start the camera operation response process; According to the position of the operation control in the operation control map, performing region matching on the video frame in the video to be detected to obtain a first preset region of the operation control in the video frame; Detect pixel changes in a first preset area of ​​adjacent video frames in the video to be detected to obtain a first frame in which the operation control starts to change, where the first frame in which the operation control starts to change is a starting frame corresponding to the camera operation response process; Detect pixel changes in a video frame after the start frame or in an adjacent video frame in the video to be detected to obtain a response-ending video frame, where the response-ending video frame is an end frame corresponding to the camera operation response process.

2. The method according to claim 1, characterized in that The detecting pixel changes in the first preset area of ​​adjacent video frames in the video to be detected to obtain the first frame where the operation control starts to change includes: Performing inter-frame difference on a first preset area of ​​adjacent video frames in the video to be detected to obtain a frame difference value; The first frame in which the frame difference value is greater than the preset difference value is used as the first frame in which the operation control starts to change.

3. The method according to claim 2, characterized in that The performing inter-frame difference on a first preset area of ​​adjacent video frames in the video to be detected to obtain a frame difference value includes: Performing grayscale transformation on a video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame; An inter-frame difference is performed on a first preset area of ​​a first grayscale image corresponding to adjacent video frames to obtain the frame difference value.

4. The method according to any one of claims 1 to 3, characterized in that: The detecting pixel changes in the video frame after the start frame or the adjacent video frame in the video to be detected to obtain the video frame at the end of the response includes: Extracting edge features of the video frames after the starting frame in the video to be detected in sequence to obtain line features of each video frame after the starting frame; The first frame of the video frame in which the number of the line feature indicating lines is greater than the first number is used as the video frame in which the response ends.

5. The method according to claim 4, characterized in that Before sequentially extracting edge features from the video frames after the start frame in the video to be detected to obtain line features of each video frame after the start frame, the method further includes: Obtaining an operation response diagram corresponding to the camera operation response process; According to the position of the first object in the operation response graph, performing region matching on the video frame in the video to be detected to obtain a second preset region of the first object in the video frame; The step of sequentially extracting edge features from the video frames after the start frame in the video to be detected to obtain line features of each video frame after the start frame includes: Edge features are extracted sequentially from second preset areas of video frames located after the start frame in the video to be detected to obtain line features of each video frame located after the start frame.

6. The method according to any one of claims 1 to 5, characterized in that: The camera operation response process includes the camera application jumping from the gallery interface to the preview interface, the operation control is a return control, the first frame in which the return control starts to change is the start frame corresponding to the camera operation response process, and the first frame in which the picture appears in the preview interface is the end frame corresponding to the camera operation response process.

7. The method according to claim 4 or 5, characterized in that The camera operation response process includes switching from front-end shooting to rear-end shooting or from rear-end shooting to front-end shooting, the operation control is a flip control, the first frame in which the flip control starts to change is the start frame corresponding to the camera operation response process, and the first frame in which the picture appears in the preview interface is the end frame corresponding to the camera operation response process; The step of taking the first frame of the video frame in which the number of the line feature indicating lines is greater than the first number as the video frame in which the response ends, comprises: In a process in which the number of lines indicated by the line features of multiple video frames located after the starting frame gradually decreases, it is detected that the number of lines is less than a second number, and in a process in which the number of lines indicated by the line features of the multiple video frames gradually increases, it is detected that the number of lines is greater than the first number, the first frame of the video frames whose number of lines is greater than the first number is used as the first frame of the picture appearing in the preview interface.

8. The method according to any one of claims 1 to 5, characterized in that: The camera operation response process includes starting a camera application for shooting, the operation control is a camera application icon, the first frame in which the camera application icon starts to change is the start frame corresponding to the camera operation response process, and the first frame in which the picture appears in the preview interface is the end frame corresponding to the camera operation response process.

9. The method according to any one of claims 1, 4 and 5, characterized in that The camera operation response process includes starting a camera application to shoot, the operation control is a camera application icon, and the first frame in which the camera application icon starts to change is the starting frame corresponding to the camera operation response process; The detecting pixel changes in the first preset area of ​​adjacent video frames in the video to be detected to obtain the first frame where the operation control starts to change includes: Performing color extraction on a first preset area of ​​a video frame in the video to be detected to obtain a color feature; Determining color similarity between adjacent video frames according to the color feature; The first frame of the video frame whose color similarity is greater than the preset similarity is used as the first frame at which the camera application icon starts to change.

10. The method according to any one of claims 1 to 3, characterized in that: The camera operation response process includes receiving a trigger operation on a photo control to generating a thumbnail corresponding to the latest captured image in an image echo control, the image echo control is used to trigger the display of the latest captured image, the operation control is a photo control, the first frame in which the photo control starts to change is the start frame corresponding to the camera operation response process, and the last frame in which the thumbnail changes is the end frame corresponding to the camera operation response process; The detecting pixel changes in the video frame after the start frame or the adjacent video frame in the video to be detected to obtain the video frame at the end of the response includes: Performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain first frame difference data; The video frame in which the area of ​​the picture outline indicated by the first frame difference data is smaller than the first preset area is used as the last frame of the thumbnail change.

11. The method according to claim 10, characterized in that Before performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain first frame difference data, the method further includes: Get the shooting interface image; According to the position of the image echo control in the shooting interface diagram, performing region matching on the video frame in the video to be detected to obtain a third preset region of the thumbnail in the video frame; The step of performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain first frame difference data includes: Performing grayscale conversion on a video frame located after the start frame in the video to be detected to obtain a second grayscale image corresponding to the video frame; Inter-frame difference is performed on a third preset area of ​​the second grayscale image corresponding to adjacent video frames to obtain the first frame difference data.

12. The method according to any one of claims 1 to 3, characterized in that: The camera operation response process includes receiving a trigger operation on an image echo control to displaying a display image corresponding to a thumbnail in the image echo control, wherein the operation control is the image echo control, and the image echo control is used to trigger the display of the latest captured image; the first frame in which the image echo control starts to change is the start frame corresponding to the camera operation response process, and the first frame in which the display image is fully opened is the end frame corresponding to the camera operation response process; The detecting pixel changes in the video frame after the start frame or the adjacent video frame in the video to be detected to obtain the video frame at the end of the response includes: Performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain second frame difference data; The video frame in which the area of ​​the picture outline indicated by the second frame difference data is smaller than the second preset area is used as the first frame of the fully opened display image.

13. The method according to any one of claims 1 to 3, characterized in that: The camera operation response process includes from receiving a trigger operation on a video recording control to starting video recording, the operation control is a video recording control, and the first frame in which the video recording control starts to change is the start frame corresponding to the camera operation response process; After detecting pixel changes in a first preset area of ​​adjacent video frames in the video to be detected and obtaining a first frame where the operation control starts to change, the method further includes: Determine, among the video frames after the start frame in the video to be detected, video frames whose recording duration is a preset duration; The first frame of the video frames whose recording duration is the preset duration is used as the end frame corresponding to the camera operation response process.

14. The method according to claim 1, wherein: The camera operation response process includes switching from close-up shooting to long-range shooting, and the first frame in which the baffle of the close-up is completely removed is the starting frame corresponding to the camera operation response process; The method further comprises: Get the baffle map; According to the position of the baffle in the baffle map, performing region matching on the video frame in the video to be detected to obtain the baffle region in the video frame; Performing inter-frame difference on the baffle regions of adjacent video frames in the video to be detected to obtain third frame difference data; When the third frame difference data indicates that the baffle has moved, performing inter-frame difference on adjacent video frames in the video to be detected to obtain fourth frame difference data; The first frame in which the fourth frame difference data indicates that the outline of the target component appears is used as the first frame in which the baffle is completely removed, and the target component is the component displayed after the baffle is completely removed.

15. The method according to claim 14, characterized in that The method further comprises: Get the parts diagram; According to the position of the target component in the component map, performing region matching on the video frame in the video to be detected to obtain the component region of the target component in the video frame; When the third frame difference data indicates that the baffle has moved, performing inter-frame difference on adjacent video frames in the video to be detected to obtain fourth frame difference data includes: When the third frame difference data indicates that the baffle has moved, grayscale conversion is performed on the video frame in the video to be detected to obtain a third grayscale image corresponding to the video frame; Inter-frame difference is performed on the component area of ​​the third grayscale image corresponding to adjacent video frames to obtain the fourth frame difference data.

16. The method according to claim 14 or 15, characterized in that The first frame where the distant view image stops zooming is the end frame corresponding to the camera operation response process; The detecting pixel changes in the video frame after the start frame or the adjacent video frame in the video to be detected to obtain the video frame at the end of the response includes: Performing inter-frame difference on adjacent video frames after the start frame in the video to be detected to obtain fifth frame difference data; The video frame in which the area of ​​the picture outline indicated by the fifth frame difference data is smaller than the third preset area is used as the first frame for stopping zooming of the distant picture.

17. The method according to claim 16, characterized in that Before performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain fifth frame difference data, the method further includes: Get a vision map; According to the position of the second object in the distant view, performing region matching on the video frame in the video to be detected to obtain a fourth preset region of the second object in the video frame; The step of performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain fifth frame difference data includes: Performing grayscale conversion on a video frame located after the start frame in the video to be detected to obtain a fourth grayscale image corresponding to the video frame; Inter-frame difference is performed on a fourth preset area of ​​a fourth grayscale image corresponding to adjacent video frames to obtain the fifth frame difference data.

18. The method according to any one of claims 1 to 3, characterized in that: In the case where the shooting scene includes a dynamic timer, the camera operation response process includes from receiving a trigger operation on a shooting control to completing shooting of the dynamic timer, the operation control is a shooting control, and the first frame in which the shooting control starts to change is the starting frame corresponding to the camera operation response process; After detecting pixel changes in a first preset area of ​​adjacent video frames in the video to be detected and obtaining a first frame where the operation control starts to change, the method further includes: Determine a first time indicated by the dynamic timer in the first frame in which the photographing control begins to change, and determine a second time indicated by the dynamic timer in the captured picture.

19. The method according to claim 18, characterized in that The dynamic timer includes a dynamically displayed light array, the light array including a plurality of lights arranged in rows and columns; The determining the first time indicated by the dynamic timer in the first frame in which the photographing control starts to change includes: Performing color recognition on each video frame in the video to be detected to obtain the outline of the light emitting in each video frame; Constructing a first coordinate matrix of the dynamic timer according to the outlines of the lights emitting light in each of the video frames; Performing color recognition on the first frame in which the photographing control starts to change, to obtain a first outline; the first outline is the outline of the light emitting in the first frame in which the photographing control starts to change; According to the position of the first contour in the first frame in which the photographing control starts to change, position matching is performed in the first coordinate matrix to obtain the position of the first contour in the first coordinate matrix; The first time is determined according to the position of the first contour in the first coordinate matrix.

20. The method according to claim 18 or 19, characterized in that The determining the second time indicated by the dynamic timer in the captured picture includes: Performing shape recognition on the image according to a preset shape to obtain a second coordinate matrix of the dynamic timer; the preset shape is a physical shape of the lamp, and the second coordinate matrix indicates the position of the unlit lamp; Performing color recognition on the picture to obtain a second outline; the second outline is an outline of a light emitting in the picture; Performing position matching in the second coordinate matrix according to the position of the second contour in the picture to obtain the position of the second contour in the second coordinate matrix; The second time is determined according to the position of the second contour in the second coordinate matrix.

21. The method according to any one of claims 18 to 20, characterized in that: After determining the first time indicated by the dynamic timer in the first frame in which the photographing control starts to change, and determining the second time indicated by the dynamic timer in the captured picture, the method further includes: Determine a delay time of a camera photographing of the target terminal according to the first time and the second time.

22. The method according to any one of claims 1 to 3, characterized in that: In the case where the shooting scene includes a dynamic timer, the dynamic timer includes a dynamically displayed light array, the light array includes a plurality of lights arranged in rows and columns, the camera operation response process includes executing a process in response to a trigger operation on a photo control, the operation control is a photo control, and the first frame in which the photo control starts to change is the start frame corresponding to the camera operation response process; The detecting pixel changes in the video frame after the start frame or the adjacent video frame in the video to be detected to obtain the video frame at the end of the response includes: Detect the video frames after the start frame in the video to be detected, determine the first frame in which the screen in the preview interface changes from black to bright and the last frame in which the dynamic timer displays a freeze, wherein the position of the light in the video frames before and after the dynamic timer displays a freeze changes from unchanged to changed; The video frame that is sorted after the first frame in which the picture changes from black to bright and the last frame in which the dynamic timer displays a freeze is used as the end frame corresponding to the camera operation response process.

23. The method of claim 22, wherein: The detecting of the video frames after the start frame in the video to be detected, and determining the first frame in which the screen in the preview interface changes from black to bright and the last frame in which the dynamic timer displays a freeze, comprises: When the brightness of the current video frame is less than a first threshold value and the brightness difference between the current video frame and a previous video frame of the current video frame is greater than a second threshold value, the current video frame is used as the first frame of the picture changing from black to bright; Alternatively, a brightness threshold is determined according to the brightness of each video frame in the video to be detected; in the process of gradually increasing brightness of multiple video frames located after the starting frame, the video frame with brightness greater than the brightness threshold is used as the first frame of the picture changing from black to bright.

24. The method according to claim 22 or 23, characterized in that The detecting of the video frames after the start frame in the video to be detected, and determining the first frame in which the screen in the preview interface changes from black to bright and the last frame in which the dynamic timer displays a freeze, comprises: Performing color recognition on each video frame located after the starting frame in the video to be detected to obtain a third outline of each video frame; the third outline is the outline of the light emitting in the video frame located after the starting frame; Determine a plurality of target video frames in each video frame after the start frame; wherein the third contours of the plurality of target video frames are the same, and the third contour of the next video frame of the last frame in the plurality of target video frames is different from the third contours of the plurality of target video frames; The last frame among the multiple target video frames is used as the last frame of the dynamic timer display freeze.

25. The method according to any one of claims 1 to 24, characterized in that After detecting pixel changes in a video frame or an adjacent video frame located after the start frame in the video to be detected and obtaining a video frame for which a response ends, the method further includes: Determine a target frame number according to the start frame and the end frame; The duration or shooting rate of the camera operation response process is determined according to the target frame number.

26. A computer device, characterized in that: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 25 when executed by the processor.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 25.

28. A computer program product comprising instructions, characterized in that When the method is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 25.

Images