Video frame detection method, device and storage medium

Abstract

The present application discloses a video frame detection method, device and storage medium, which belong to the field of image processing technology. The method includes: obtaining a video to be detected by shooting the screen of the target terminal when executing the camera operation response process, starting the camera operation response process will use the operation control, and pre-acquiring the corresponding operation control diagram. According to the position of the operation control in the operation control diagram, regional matching is performed in the video frame to obtain the first preset area of ​​the operation control in the video frame. Whether pixel changes occur in the first preset area in the adjacent video frames, the first frame where the detected operation control begins to change is used as the starting frame. Then, the pixel changes in the video frame or the adjacent video frame after the starting frame are detected to determine whether there is a picture or whether the image appears completely, so as to detect the video frame where the response ends, and use it as the end frame, thereby realizing an automatic detection process and improving detection efficiency and detection accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video frame detection method, device, and storage medium. Background Art

[0002] In recent years, with the rapid development of the device manufacturing industry, the user's photography experience on the device has become increasingly important. In this regard, the user's perceived photography process duration is a key factor affecting the user experience.

[0003] To evaluate the duration of the photo process, a common approach is to capture the test prototype's screen with a high-speed camera, capturing the camera operation response process as it executes, and obtaining a video frame sequence. Based on given start and end criteria for key nodes, the video frame sequence is manually judged to identify the start and end frames of the photo process. The number of frames between the start and end frames can then be determined, and performance parameters such as the photo process duration can be determined based on the number of frames captured.

[0004] Currently, the detection process of the start frame and the end frame requires manual comparison of each video frame. Manual data comparison is often inefficient, and the detection results are subjective, which reduces the efficiency and accuracy of video frame detection. Summary of the Invention

[0005] This application provides a video frame detection method, device, and storage medium, which can improve the efficiency and accuracy of video frame detection. The technical solution is as follows:

[0006] In a first aspect, a video frame detection method is provided. The method may include: obtaining a video to be detected by capturing a screen of a target terminal while executing a camera operation response process, the video to be detected comprising multiple consecutive video frames. The camera operation response process is the process by which the target terminal operates different shooting functions, and different operation controls are used during the operation process. An operation control diagram corresponding to the camera operation response process may be obtained in advance. The operation control diagram includes the operation controls, and different camera operation response processes correspond to different operation controls. The operation control diagram and the video frame including the operation controls are images from the same perspective, and the position of the operation controls in the operation control diagram is the same as its position in the video frame. Based on this, region matching is performed in the video frame based on the position of the operation controls in the operation control diagram to obtain the region corresponding to the operation controls in the video frame (i.e., a first preset region). Whether pixel changes occur in the first preset region in adjacent video frames, that is, whether there are significant differences in the regions containing the operation controls in adjacent video frames, is detected, thereby detecting the video frame where the operation controls begin to change. The first frame where the operation controls begin to change is the starting frame of the camera operation response process. After the start frame is detected, pixel changes in the video frames after the start frame or in adjacent video frames in the video to be detected are detected to determine whether a picture appears or whether the image appears completely.

[0007] In the embodiments of the present application, by matching the area where the control is located in the video frame, targeted regional detection can be achieved, thereby improving detection accuracy. Furthermore, the amount of data required to determine whether a pixel has changed is reduced, thereby improving detection efficiency. By detecting whether a pixel change has occurred in the video frame or whether a difference has occurred between adjacent video frames, the end frame is detected, thereby achieving an automatic detection process for the video frame, thereby improving both detection efficiency and accuracy.

[0008] Among them, for the video recording scene, the operation control is the video recording control; for the scene of starting the camera application icon to shoot, the operation control is the camera application icon; for the scene of switching between front-facing and rear-facing shooting, the operation control is the flip control.

[0009] In one possible implementation, when detecting whether the operation control begins to change, the area where the operation control is located in the video frame (i.e., the first preset area) can be detected without paying attention to other areas. When the operation control of the target terminal is clicked, the control will first shrink and then return to normal. Detecting the pixel changes in the first preset area of ​​adjacent video frames in the video to be detected, and obtaining the first frame where the operation control begins to change, including: using a frame difference method to perform inter-frame difference on the area where the operation control is located in adjacent video frames in the video to be detected, and obtaining the frame difference between adjacent video frames; when the frame difference is greater than the preset difference value, it indicates that the operation control begins to change, and the first frame where the frame difference is greater than the preset difference value is used as the first frame where the operation control begins to change.

[0010] In an embodiment of the present application, when the frame difference method is used to detect whether there are large differences in the pixels of adjacent video frames, targeted area detection can be performed, which improves the detection accuracy and reduces the amount of data used to determine whether the pixels have changed, thereby improving the detection efficiency.

[0011] In one possible implementation, inter-frame difference is performed on a first preset area of ​​adjacent video frames in the video to be detected to obtain a frame difference value, including: for each video frame in the video to be detected, the video frame can be firstly transformed into grayscale to obtain a first grayscale image corresponding to the video frame, and then inter-frame difference is performed on the area where the operation control of the first grayscale image of the adjacent video frames is located to obtain a frame difference value.

[0012] In the embodiments of the present application, grayscale images can better reflect the differences in pixel changes. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0013] In one possible implementation, after detecting the starting frame in the video to be detected, the computer device then detects the picture that appears in the preview interface of the target terminal. Detecting pixel changes in the video frame after the starting frame or in adjacent video frames in the video to be detected to obtain a video frame where the response ends, including: sequentially extracting edge features from the video frames after the starting frame in the video to be detected to obtain line features of each video frame after the starting frame. The line feature can indicate the number of lines. When the number of lines is greater than a first number, it indicates that a picture appears in the preview interface and the camera operation response ends. The first frame of the video frame whose line feature indicates that the number of lines is greater than the first number is used as the first frame that appears in the preview interface, and the first frame that appears in the preview interface is used as the end frame.

[0014] In an embodiment of the present application, by comparing the number of lines that constitute the outline, it is determined whether the picture appears in the preview interface without manual detection, thereby realizing an automatic detection process for the end frame and improving the efficiency and accuracy of video frame detection.

[0015] In one possible implementation, the video frame detection method provided by the present application is applied during the testing process of the terminal, and the terminal may be operated in a relatively static state. Therefore, the method may further include: pre-acquiring an operation response diagram, which is an interface diagram displayed by the terminal in response to triggering an operation on an operation control. The operation response diagram includes a first object (for example, the first object) and a surrounding background, and the background may be black or white, etc. The operation response diagram and the video frame in which the first object is detected are images from the same perspective, and the position of the first object in the operation response diagram is the same as its position in the video frame. Based on the positional features of the area where the first object is located in the operation response diagram, template matching is performed in the video frame located after the starting frame, the position coordinates of the first object are determined in the video frame, and the area where the first object is located in the video frame (i.e., the second preset area) is used as a specific area that needs attention. Edge feature extraction is performed on the video frames located after the starting frame in the video to be detected in sequence to obtain line features of each video frame located after the starting frame, including: edge feature extraction is performed on the second preset area of ​​the video frame located after the starting frame in the video to be detected in sequence to obtain line features of each video frame located after the starting frame.

[0016] In the embodiments of the present application, the appearance of the image in the preview interface typically corresponds to the appearance of the first object. When detecting whether the first object appears in the preview interface, detection can be performed on the area in the video frame where the first object is located (i.e., the second preset area), without paying attention to the background. By selecting the area in the video frame corresponding to the first object, targeted regional detection can be performed, thereby improving detection accuracy, reducing the range of edge feature extraction, and improving detection efficiency.

[0017] In one possible implementation, after viewing a captured image and needing to continue shooting, the camera app's gallery interface needs to be clicked to return to the preview interface. The camera operation response process involves the camera app jumping from the gallery interface to the preview interface, with the control being the return control. The first frame in which the return control begins to change is the start frame of the camera operation response process, and the first frame in which the image appears in the preview interface is the end frame of the camera operation response process.

[0018] In one possible implementation, when switching the camera for shooting, it is necessary to click on the flip control (also called the flip button) and then enter the preview interface of the front camera or the rear camera. The camera operation response process includes switching from front shooting to rear shooting or from rear shooting to front shooting. The operation control is a flip control. The first frame in which the flip control starts to change is the starting frame corresponding to the camera operation response process, and the first frame in which the picture appears in the preview interface is the ending frame corresponding to the camera operation response process. In the scenario of switching the camera for shooting, after detecting the starting frame in the video to be detected, the computer device then detects the picture in the preview interface that appears on the target terminal. When the picture in the preview interface appears on the target terminal, the picture in the preview interface gradually changes from clear to blurred and then becomes clear again.

[0019] In an embodiment of the present application, the computer device can realize the recognition of the picture in the preview interface by the edge detection method. The edge feature extraction is performed on the video frame located after the starting frame in the video to be detected to obtain the line feature. The line feature can indicate the number of lines. The process in which the picture in the preview interface gradually changes from clear to blurred and then becomes clear again is reflected in the gradual decrease in the number of lines and then gradually increases. In the process of detecting that the line feature indicates that the number of lines is gradually decreasing, when the number of lines is less than the second number, it means that the picture in the preview interface is blurred, and the number of lines is continued to be counted. In the process of detecting that the line feature indicates that the number of lines of multiple video frames is gradually increasing, when the number of lines is greater than the first number, it means that the picture in the preview interface is clear again. Therefore, the first frame of the video frame whose number of lines is greater than the first number is used as the first frame that appears in the preview interface, and the first frame that appears in the preview interface is used as the end frame.

[0020] In an embodiment of the present application, the computer device not only detects whether the number of lines is less than the second number to determine the process of the image becoming blurred, but also detects whether the number of lines is greater than the first number to determine the process of the image becoming clear again, and uses the cleared video frame as the end frame. The embodiment of the present application detects the process of blurring and clearing during the switching process. After detecting that the image has become blurred, the process of the image becoming clear is detected again to reduce the situation where the number of lines is greater than the first number due to unsuccessful switching, thereby ensuring successful switching and improving the accuracy of video frame detection. No manual detection is required, and the detection process of the end frame is automatically realized, thereby improving the efficiency and accuracy of video frame detection.

[0021] In one possible implementation, when launching a camera app for shooting, the user taps the camera app icon to enter the preview interface. The camera operation response process includes launching the camera app for shooting, using the camera app icon as the control, and the first frame in which the camera app icon begins to change is the start frame of the camera operation response process. The first frame in which the preview interface appears is the end frame of the camera operation response process.

[0022] In one possible implementation, the camera operation response process includes starting a camera application for shooting, the operation control is a camera application icon, and the first frame in which the camera application icon starts to change is the starting frame corresponding to the camera operation response process. When the camera application icon of the terminal is clicked, the color will change, and the color detection method can be used to identify whether the camera application icon starts to change. The pixel changes in the first preset area of ​​adjacent video frames in the video to be detected are detected to obtain the first frame in which the operation control starts to change, including: extracting color features from the first preset area of ​​the video frame in the video to be detected to obtain color features; and determining the color similarity between adjacent video frames based on the color features. When the color similarity is greater than the preset similarity, it indicates that the camera application icon starts to change. The first frame of the video frame whose color similarity is greater than the preset similarity is used as the first frame in which the camera application icon starts to change, and the first frame in which the camera application icon starts to change is the starting frame.

[0023] In an embodiment of the present application, by matching the area where the camera application icon is located in the video frame, targeted area detection can be performed when detecting whether the color of the camera application icon begins to change, thereby improving detection accuracy and reducing the amount of data when judging whether the pixel has changed, thereby improving detection efficiency.

[0024] Furthermore, when detecting whether the camera application icon starts to change, color detection or frame difference method can be used for detection, which increases the diversity and richness of detection methods.

[0025] In one possible implementation, when taking a photo using a terminal, it is necessary to trigger the photo control and then obtain a thumbnail. The thumbnail is a thumbnail of the latest captured image, which is displayed in the image echo control. The camera operation response process includes receiving a trigger operation on the photo control and generating a thumbnail corresponding to the latest captured image in the image echo control. The image echo control is used to trigger the display of the latest captured image. The operation control is a photo control. The first frame in which the photo control begins to change is the starting frame corresponding to the camera operation response process, and the last frame in which the thumbnail changes is the ending frame corresponding to the camera operation response process. The computer device can achieve complete picture recognition by using a method of picture contour detection. Detecting pixel changes in the video frame or adjacent video frames after the starting frame in the video to be detected to obtain a video frame at the end of the response includes: performing inter-frame difference on adjacent video frames after the starting frame in the video to be detected to obtain first frame difference data. The first frame difference data includes pixel differences of multiple pixels, which constitute a picture outline. The area of ​​the picture outline reflects the difference between adjacent video frames. The larger the area of ​​the picture outline, the greater the difference between adjacent video frames, indicating that the thumbnail is being scaled. The smaller the area of ​​the picture outline, the smaller the difference between adjacent video frames, indicating that the thumbnail scaling is complete (i.e., the thumbnail has stopped scaling or the thumbnail appears completely). The video frame whose picture outline area indicated by the first frame difference data is less than the first preset area is used as the last frame of the thumbnail change (i.e., the first frame where the thumbnail stops scaling), and the last frame of the thumbnail change is used as the end frame.

[0026] In an embodiment of the present application, the frame difference variation pattern of adjacent video frames is identified, and whether a large-area difference occurs in the frame difference is judged to determine whether the thumbnail appears completely. No manual detection is required, and the end frame detection process is automatically realized, thereby improving the video frame detection efficiency and detection accuracy.

[0027] In a possible implementation, the video frame detection method provided by the present application is applied in the test process of the terminal, and the terminal can be operated in a relatively static state, so the shooting interface image can be acquired in advance. The shooting interface image includes an image echo control, and the shooting interface image and the video frame in which the thumbnail is detected are images under the same perspective, and the position of the image echo control in the shooting interface image is the same as the position of the thumbnail in the video frame. Based on this, according to the position of the image echo control in the shooting interface image, regional matching is performed in the video frame in the video to be detected, and the area corresponding to the thumbnail in the video frame (i.e., the third preset area) is obtained. For the video frame located after the starting frame in the video to be detected, the video frame can be first grayscale converted to obtain a second grayscale image corresponding to the video frame, and then the area where the thumbnail of the second grayscale image corresponding to the adjacent video frames is located is subjected to inter-frame difference to obtain the first frame difference data.

[0028] In the embodiments of the present application, by selecting the area in the video frame corresponding to the thumbnail, targeted regional detection can be performed, improving detection accuracy and reducing the amount of data required to determine whether a pixel has changed, thereby improving detection efficiency. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0029] In one possible implementation, when viewing a picture just taken, you can click on the image echo control (also called the image echo button), and then the enlarged thumbnail is fully displayed in the gallery interface of the camera application (that is, the thumbnail is fully opened to become a display image). The camera operation response process includes receiving a trigger operation on the image echo control to displaying the display image corresponding to the thumbnail in the image echo control, the operation control is the image echo control, and the image echo control is used to trigger the display of the latest captured image; the first frame in which the image echo control begins to change is the starting frame corresponding to the camera operation response process, and the first frame in which the display image is fully opened is the ending frame corresponding to the camera operation response process. Detecting pixel changes in the video frame or adjacent video frames after the starting frame in the video to be detected to obtain a video frame at the end of the response includes: performing inter-frame difference on adjacent video frames after the starting frame in the video to be detected to obtain second frame difference data. The second frame difference data includes the pixel difference of multiple pixels, and the pixel difference of multiple pixels constitutes the picture outline. The area of ​​the picture outline reflects the difference between adjacent video frames. The larger the area of ​​the picture outline, the greater the difference between adjacent video frames, indicating that the display image is being scaled. The smaller the area of ​​the picture outline, the smaller the difference between adjacent video frames, indicating that the display image has been scaled (i.e., the display image has stopped scaling or the display image has completely appeared). The video frame whose area of ​​the picture outline indicated by the second frame difference data is less than the second preset area is used as the first frame of the display image that is fully opened (i.e., the first frame of the display image that stops scaling, or the last frame of the display image that changes), and the first frame of the display image that is fully opened is used as the end frame.

[0030] In an embodiment of the present application, by identifying the frame difference change pattern of adjacent video frames and judging whether there is a large-area difference in the frame difference, it is determined whether the display image is completely opened (that is, the display image stops zooming). Without manual detection, the end frame detection process is automatically realized, thereby improving the video frame detection efficiency and detection accuracy.

[0031] In one possible implementation, when recording video using a terminal, it is necessary to click on the video control and then start recording. The camera operation response process includes the process from receiving a trigger operation on the video control to starting recording, the operation control is the video control, and the first frame where the video control starts to change is the starting frame corresponding to the camera operation response process. In the video recording scene, after detecting the starting frame in the video to be detected, the video content will appear next, and the pixel changes in the first preset area of ​​the adjacent video frames in the video to be detected are detected. After obtaining the first frame where the operation control starts to change, the method also includes: determining a video frame with a preset recording duration in the video frames after the starting frame in the video to be detected; and using the first frame of the video frames with a preset recording duration as the end frame corresponding to the camera operation response process.

[0032] In an embodiment of the present application, by determining the recording duration, the first frame in the video frame with a recording duration of the preset duration is used as the end frame, without the need for manual detection, thereby realizing an automatic detection process for the end frame and improving the efficiency and accuracy of video frame detection.

[0033] In one possible implementation, the video frame detection method provided herein is applied during terminal testing to simulate the process of switching from a close-up view to a long-range view by removing a baffle. The baffle is placed in front of the terminal's rear camera. When the baffle is fully removed, a target component is displayed. The camera operation response process includes switching from a close-up view to a long-range view. The first frame in which the baffle is fully removed, representing the close-up view, serves as the starting frame for the camera operation response process. The method includes: obtaining a baffle map; performing region matching on video frames in the video to be detected based on the position of the baffle in the baffle map to obtain the baffle region in the video frame. The baffle region refers to the region within the video frame where the baffle resides. Using a frame difference method, inter-frame difference is performed on the baffle regions of adjacent video frames in the video to be detected to obtain third frame difference data. The third frame difference data includes pixel differences between multiple pixels and can indicate whether the baffle has moved. A larger pixel difference indicates a greater difference between adjacent video frames, indicating movement of the baffle. A smaller pixel difference indicates a smaller difference between adjacent video frames, indicating no movement of the baffle. If the third frame difference data indicates that the baffle has moved, inter-frame differences are performed on adjacent video frames in the video to be detected to generate fourth frame difference data. The fourth frame difference data includes pixel differences of multiple pixels, and the pixel differences of the multiple pixels constitute the image outline. When the outline of the target component is detected, it indicates that the baffle has been completely removed. The first frame in which the outline of the target component appears is regarded as the first frame in which the baffle is completely removed.

[0034] In the embodiments of the present application, by matching the baffle area in the video frame, when using the frame difference method to detect whether there are significant differences in the pixels of adjacent video frames, targeted regional detection can be performed, thereby improving detection accuracy. Furthermore, the amount of data required to determine whether a pixel change has occurred is reduced, thereby improving detection efficiency. By identifying the variation patterns of the frame differences between adjacent video frames and determining whether there are large area differences in the frame differences, it is possible to determine whether the distant view has stopped zooming, eliminating the need for manual detection and achieving automatic detection of the end frame, thereby improving the efficiency and accuracy of video frame detection.

[0035] In one possible implementation, the method further includes: pre-acquiring a component map, performing region matching on video frames in the video to be detected based on the position of the target component in the component map, and obtaining a component region of the target component in the video frame. When the third frame difference data indicates that the baffle has moved, performing inter-frame difference on adjacent video frames in the video to be detected to obtain fourth frame difference data, including: when the third frame difference data indicates that the baffle has moved, performing grayscale conversion on each video frame in the video to be detected to obtain a grayscale image corresponding to the video frame (i.e., a third grayscale image). Then, performing inter-frame difference on the component regions of the grayscale images of adjacent video frames to obtain fourth frame difference data.

[0036] In the embodiments of the present application, by matching the target component area in the video frame, when using the frame difference method to detect whether there are significant differences in the pixels of adjacent video frames, targeted regional detection can be performed, thereby improving detection accuracy and reducing the amount of data required to determine whether a pixel has changed, thereby improving detection efficiency. Using grayscale images for inter-frame differencing can reduce the amount of calculation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0037] In one possible implementation, the first frame where the distant view stops scaling is the end frame corresponding to the camera operation response process; pixel changes in a video frame or adjacent video frames located after the start frame in the video to be detected are detected to obtain a video frame where the response ends, including: performing inter-frame difference between adjacent video frames located after the start frame in the video to be detected to obtain fifth frame difference data. The fifth frame difference data includes pixel differences of multiple pixels, and the pixel differences of the multiple pixels constitute a picture outline. The area of ​​the picture outline reflects the difference between adjacent video frames; the larger the area of ​​the picture outline, the greater the difference between adjacent video frames, indicating that the distant view is scaling; the smaller the area of ​​the picture outline, the smaller the difference between adjacent video frames, indicating that the distant view has been scaled (i.e., the distant view has stopped scaling). The video frame whose picture outline area indicated by the fifth frame difference data is less than the third preset area is used as the first frame where the distant view stops scaling, and the first frame where the distant view stops scaling is used as the end frame.

[0038] In an embodiment of the present application, by identifying the change pattern of the frame difference between adjacent video frames and judging whether a large area difference appears in the frame difference, it is determined whether the distant view picture stops zooming, without the need for manual detection, thereby realizing an automatic detection process for the end frame and improving the efficiency and accuracy of video frame detection.

[0039] In one possible implementation, before performing inter-frame difference on adjacent video frames located after the starting frame in the video to be detected and obtaining fifth frame difference data, the method further includes: pre-acquiring a distant view image, wherein the distant view image and the video frame in which the second object is detected are images from the same perspective, and the position of the second object in the distant view image is the same as its position in the video frame. Based on the position characteristics of the second object in the distant view image, performing regional matching on the video frames in the video to be detected and obtaining a fourth preset area of ​​the second object in the video frame. Performing inter-frame difference on adjacent video frames located after the starting frame in the video to be detected and obtaining fifth frame difference data includes: for the video frame located after the starting frame in the video to be detected, performing grayscale conversion on the video frame to obtain a fourth grayscale image corresponding to the video frame; performing inter-frame difference on the area where the second object is located in the fourth grayscale image corresponding to the adjacent video frames to obtain fifth frame difference data.

[0040] In the embodiments of the present application, by selecting the area corresponding to the second object in the video frame, targeted regional detection can be performed, thereby improving detection accuracy and reducing the amount of data required to determine whether a pixel has changed, thereby improving detection efficiency. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Furthermore, grayscale images can eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0041] In one possible implementation, the camera photo delay is the time difference between pressing the shutter button and the camera actually capturing the image. When detecting the camera's photo delay, it is necessary to click on the photo control and then obtain the captured image. A photo delay scenario is simulated by taking a photo of the dynamic timer through the terminal, and the captured image includes the dynamic timer. In the case where the shooting scenario includes the dynamic timer, the camera operation response process includes the process from receiving a trigger operation on the photo control to completing the shooting of the dynamic timer. The operation control is the photo control, and the first frame in which the photo control begins to change is the starting frame corresponding to the camera operation response process. After detecting pixel changes in a first preset area of ​​adjacent video frames in the video to be detected and obtaining the first frame in which the operation control begins to change, the method further includes: determining a first time indicated by the dynamic timer in the first frame in which the photo control begins to change, and determining a second time indicated by the dynamic timer in the captured image. Subtracting the two to obtain the photo delay duration.

[0042] In one possible implementation, a dynamic timer includes a dynamically displayed light array, comprising a plurality of lights arranged in rows and columns. The dynamic timer controls the lights to illuminate and display in a cycle at a preset frequency, and the outlines of the illuminated lights can be detected through color recognition. Determining the first time indicated by the dynamic timer in the first frame where the photo control begins to change includes: traversing each video frame in the video to be detected, performing color recognition on each video frame, and obtaining the outlines of the illuminated lights in each video frame; deduplicating the outlines of the illuminated lights in multiple video frames of the video to be detected and superimposing them together to construct the outlines of all lights in the dynamic timer and obtain a first coordinate matrix for the dynamic timer. Then, color recognition is performed on the starting frame to obtain the first outline of the illuminated lights in the starting frame; based on the position of the first outline in the starting frame, position matching is performed in the first coordinate matrix to obtain the position of the first outline in the first coordinate matrix, and then the time of the dynamic timer corresponding to the starting frame is determined based on the position of the first outline in the first coordinate matrix.

[0043] In the embodiment of the present application, for low-resolution video frames, the entire video to be detected is traversed, and the illuminated lights in each video frame are identified to construct a coordinate matrix for the dynamic timer. The outline of the illuminated lights in the starting frame is then identified and template-matched with the coordinate matrix to obtain the row and column where the illuminated lights are located in the starting frame, thereby obtaining the time corresponding to the starting frame. By identifying the digitized video, the row and column where the lights are located in the starting frame are determined, and automated reading is achieved. This eliminates the need for manual detection and enables automatic detection of the ending frame, improving the efficiency and accuracy of video frame detection.

[0044] In one possible implementation, determining the second time indicated by a dynamic timer in a captured image includes: performing shape recognition on the captured image based on a preset shape to obtain the positions of all unlit lamps in the captured image, and constructing a second coordinate matrix based on the positions of all unlit lamps. Furthermore, performing color recognition on the captured image to obtain second outlines of lit lamps in the image; matching the second outlines of the lit lamps in the second coordinate matrix to obtain the positions of the second outlines in the second coordinate matrix; and then determining the time of the dynamic timer in the image based on the positions of the second outlines in the second coordinate matrix.

[0045] In the embodiment of the present application, the positions of unlit lamps are directly identified for a captured high-definition image, a coordinate matrix is ​​constructed, and the positions of lit lamps are then identified. Template matching is performed against the coordinate matrix to determine the row and column of the lit lamp in the image, thereby determining the time corresponding to the captured image. By recognizing the digitized video, the row and column of the lamp in the image are determined, enabling automated reading. This eliminates the need for manual detection, enabling automatic detection of the end frame, improving the efficiency and accuracy of video frame detection.

[0046] In one possible implementation, after determining the first time indicated by the dynamic timer in the first frame where the photo control begins to change and the second time indicated by the dynamic timer in the captured image, the method further includes determining the photo delay duration of the target terminal's camera based on the first time and the second time. Specifically, the photo delay duration is obtained by subtracting the first and second times. This implements a simulation process for photo delay, increasing the richness and comprehensiveness of detection scenarios.

[0047] In one possible implementation, the camera shooting rate refers to the number of frames that a camera can shoot per second. The camera shooting rate scenario can be simulated by the changes in the screen display in the preview interface during the terminal's shooting process of the dynamic timer and the changes in the display of the dynamic timer. In the case where the shooting scenario includes a dynamic timer, the dynamic timer includes a dynamically displayed light array, and the light array includes a plurality of lights arranged in rows and columns. The camera operation response process includes executing a process in response to a trigger operation of a photo control, the operation control is a photo control, and the first frame in which the photo control begins to change is the starting frame corresponding to the camera operation response process. After detecting the starting frame in the video to be detected, the computer device then detects the screen in the preview interface of the target terminal, and there are dynamically displayed lights in the preview interface of the target terminal. Detecting pixel changes in video frames after the start frame or adjacent video frames in the video to be detected to obtain a video frame where the response ends, including: detecting video frames after the start frame in the video to be detected, determining the first frame in which the screen changes from black to bright in the preview interface and the last frame in which the dynamic timer display freezes, wherein the position of the light in the video frames before and after the dynamic timer display freezes changes from unchanged to changed; and selecting the video frame that is ranked after the first frame in which the screen changes from black to bright and the last frame in which the dynamic timer display freezes as the end frame corresponding to the camera operation response process. The camera shooting rate can be determined based on the start frame and the end frame.

[0048] In one possible implementation, the screen in the preview interface brightens gradually. The transition from black to bright can be detected by the brightness change. Two methods can be used to detect transitions from black to bright. First, a black threshold (i.e., a first threshold) is pre-set. If the brightness of a video frame is less than the black threshold, the frame is judged as black. A second threshold (i.e., a second threshold) is also pre-set. If the current video frame is judged as black, and the brightness difference between the current frame and the previous frame is greater than the black-to-bright threshold, the current frame is considered the first frame to transition from black to bright. Second, the brightness of multiple frames in the video to be detected is calculated, the average of these brightnesses is taken, and the brightness threshold is determined based on the average. As the brightness of multiple frames following the starting frame gradually increases, the frame with a brightness greater than the brightness threshold is considered the first frame to transition from black to bright. This method can adapt to varying lighting conditions and does not rely on a fixed threshold, thus expanding the scope of the detection method.

[0049] In the embodiments of the present application, a black threshold is set to determine the brightness of a video frame, and a black-to-light threshold is set to determine the brightness difference between adjacent video frames. This allows the detection of video frames that transition from black to light in multiple video frames following the starting frame. This eliminates the need for manual detection, achieving automatic detection of video frames that transition from black to light, thereby improving both the efficiency and accuracy of video frame detection.

[0050] In one possible implementation, the dynamic timer controls the light to display in a cyclic manner at a preset frequency, and the outline of the lighted light can be detected by color recognition. The video frame after the starting frame in the video to be detected is detected to determine the first frame in which the screen changes from black to bright in the preview interface and the last frame where the dynamic timer display is stuck, including: in the embodiment of the present application, color recognition is performed on each video frame after the starting frame to obtain the outline of the lighted light in each video frame (i.e., the third outline). When consecutive video frames with the same outline are detected, it means that the dynamic timer display is stuck, and the detection is continued until a video frame with an outline different from that of the consecutive video frames is detected, and a plurality of video frames with the same outline are obtained. The video frame with the same outline is used as the target video frame to obtain a plurality of target video frames, and the outline of the next video frame of the last frame of the multiple target video frames is different from the outline of the multiple target video frames. The last frame of the multiple target video frames is used as the last frame where the dynamic timer display is stuck.

[0051] In this embodiment, the outline of the illuminated lamp in the video frame is identified, and then multiple target video frames with the same illuminated lamp outline are determined. The last frame in these target video frames is used as the last frame where the dynamic timer display is stuck. This automatically detects video frames where the dynamic timer display is stuck, eliminating the need for manual detection and improving the efficiency and accuracy of video frame detection.

[0052] In one possible implementation, after detecting pixel changes in a video frame following a start frame or in adjacent video frames in the video to be detected and obtaining a video frame indicating the end of the response, the method further includes: after detecting the start frame and the end frame, subtracting the frame number of the end frame from the frame number of the start frame to obtain a target frame number, and determining the camera capture rate based on the target frame number. Furthermore, the duration of the camera operation response process can be calculated based on the target frame number and the camera capture frame rate.

[0053] In a second aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the method as described in the first aspect or any one of the implementations of the first aspect is implemented.

[0054] In a third aspect, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes the method involved in the first aspect or any one of the implementations of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a chip system, which can be applied to an electronic device. The chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the method described in the first aspect or any one of the implementations of the first aspect.

[0056] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the video frame inspection method described in the first aspect.

[0057] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0059] Figure 2 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0060] Figure 3 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0061] Figure 4 This is a schematic diagram of a scenario for starting a camera application to take a photo, provided in an embodiment of the present application;

[0062] Figure 5 This is a schematic diagram of an edge detection effect provided by an embodiment of the present application;

[0063] Figure 6 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 1 ;

[0064] Figure 7 This is a process of a video frame detection method provided by an embodiment of the present application Figure 1 ;

[0065] Figure 8 This is a process of a video frame detection method provided by an embodiment of the present application Figure 2 ;

[0066] Figure 9 This is a schematic diagram of a scene from taking a photo to a thumbnail provided by an embodiment of the present application;

[0067] Figure 10 This is a schematic diagram of a differential effect of control behavior detection provided by an embodiment of the present application;

[0068] Figure 11 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 2 ;

[0069] Figure 12 This is a process of a video frame detection method provided by an embodiment of the present application Figure 3 ;

[0070] Figure 13 This is a schematic diagram of a scene from a thumbnail to a display image provided by an embodiment of the present application;

[0071] Figure 14 This is a schematic diagram of a complete screen zoom stop detection effect provided by an embodiment of the present application;

[0072] Figure 15 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 3 ;

[0073] Figure 16 This is a process of a video frame detection method provided by an embodiment of the present application Figure 4 ;

[0074] Figure 17This is a schematic diagram of a scenario of jumping from the gallery interface to the preview interface provided by an embodiment of the present application;

[0075] Figure 18 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 4 ;

[0076] Figure 19 This is a process of a video frame detection method provided by an embodiment of the present application Figure 5 ;

[0077] Figure 20 This is a schematic diagram of a scenario of switching from rear-facing shooting to front-facing shooting provided by an embodiment of the present application;

[0078] Figure 21 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 5 ;

[0079] Figure 22 This is a process of a video frame detection method provided by an embodiment of the present application Figure 6 ;

[0080] Figure 23 This is a schematic diagram of a video recording scenario provided by an embodiment of the present application;

[0081] Figure 24 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 6 ;

[0082] Figure 25 This is a process of a video frame detection method provided by an embodiment of the present application Figure 7 ;

[0083] Figure 26 This is a schematic diagram of a scene switching from a near view to a distant view provided in an embodiment of the present application;

[0084] Figure 27 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 7 ;

[0085] Figure 28 This is a process of a video frame detection method provided by an embodiment of the present application Figure 8 ;

[0086] Figure 29 is a schematic diagram of a dynamic timer provided in an embodiment of the present application;

[0087] Figure 30 This is a schematic diagram of a scene for photographing a dynamic counter provided by an embodiment of the present application;

[0088] Figure 31is a schematic diagram of a shooting auxiliary line provided in an embodiment of the present application;

[0089] Figure 32 This is a schematic diagram of the outline of a lamp range provided in an embodiment of the present application;

[0090] Figure 33 This is a schematic diagram of the position of a light-emitting lamp provided in an embodiment of the present application;

[0091] Figure 34 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 8 ;

[0092] Figure 35 This is a process of a video frame detection method provided by an embodiment of the present application Figure 9 ;

[0093] Figure 36 This is another schematic diagram of a scene for photographing a dynamic counter provided by an embodiment of the present application;

[0094] Figure 37 This is a schematic diagram of a dynamic counter display freeze provided by an embodiment of the present application;

[0095] Figure 38 This is a schematic diagram of a device interaction process provided by an embodiment of the present application. Figure 9 ;

[0096] Figure 39 This is a process of a video frame detection method provided by an embodiment of the present application Figure 10 ;

[0097] Figure 40 This is a flow chart of a video frame detection method provided by an embodiment of the present application;

[0098] Figure 41 This is a flow chart of a detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0100] It should be understood that the “multiple” mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, words such as “first” and “second” are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as “first” and “second” do not limit the quantity and execution order, and words such as “first” and “second” do not necessarily limit them to be different.

[0101] The embodiments of the present application are applied to a scenario in which a camera operation response process of a test prototype (target terminal) is detected, such as detecting specific video frames (eg, a start frame and an end frame) of the camera operation response process of the target terminal.

[0102] Figure 1 is a schematic diagram of a detection system provided in an embodiment of the present application, such as Figure 1 As shown, the detection system includes a detection device 10, a high-speed camera 20 and a target terminal 30 ( Figure 1 Take a mobile phone as an example). The target terminal 30 can shoot the shooting scene, such as Figure 1 The portrait of a person set on a bracket in the shooting scene shown is photographed so that the portrait appears in the preview interface of the target terminal 30. The high-speed camera 20 is used to shoot the screen of the target terminal 30 to obtain a video to be detected (i.e., a video frame sequence), and the video to be detected includes a plurality of continuous video frames. The high-speed camera 20 is also used to transmit the captured video to be detected to the detection device 10, and the detection device 10 executes the video frame detection method provided in the embodiment of the present application to detect the start frame and the end frame from the video to be detected. The start frame and the end frame are used to indicate the key nodes where the behavior occurs in the camera operation response process of the target terminal 30. Among them, the end frame can also be called the termination frame.

[0103] Taking the target terminal 30 as a mobile phone as an example, if the camera button of the mobile phone is triggered, the video to be detected is about the process of the mobile phone taking a photo of a person. If the video button of the mobile phone is triggered, the video to be detected is about the process of the mobile phone recording a person.

[0104] The mobile phone, acting as a test prototype, can also take a photo of the dynamic timer. The dynamic timer's cyclically glowing lights appear on the phone's preview screen. The high-speed camera 20 captures the phone's screen to produce a video to be tested. If the phone's photo button is triggered, the video captured by the high-speed camera 20 shows the phone's recording of the dynamic timer.

[0105] Of course, the mobile phone can also perform other camera operation response processes, such as launching the camera application to shoot, converting thumbnails to display images, jumping from the camera application to the preview interface, switching between front and rear cameras, and switching from close-up to long-range shots. The high-speed camera 20 is used to shoot the screen of the mobile phone to obtain videos to be tested in different scenes.

[0106] It is understandable that the high-speed camera 20 can also be integrated into the detection device 10, or the detection device 10 has the function of shooting the screen of the mobile phone. The detection device 10 shoots the screen of the target terminal 30 to obtain the video to be detected. This embodiment of the application does not limit this.

[0107] Detecting the start and end frames of the camera operation response process and obtaining the number of frames between the start and end frames to determine the duration of each node is a testing process called frame counting. Frame counting involves analyzing and identifying features in consecutive video frames to detect specific frames (also called keyframes or key nodes). This process can be applied to a variety of technical fields, including video management, traffic management, robotic navigation, and virtual reality (VR). It involves real-time processing and understanding of video frame sequences, which is crucial to the system's response speed and judgment accuracy.

[0108] Currently, the process of manually selecting specific frames from multiple video frames is labor-intensive and time-consuming. Furthermore, subjective judgment can lead to bias in the selection of specific frames, making it difficult to adhere to standardized standards. Therefore, ensuring the accuracy of the selected frames is crucial.

[0109] Currently, the manual process of selecting specific frames from multiple video frames requires manual control of video data quality. Data quality control typically involves the following stages. First, before manually selecting a specific frame, the raw video data must be manually verified to identify potential errors and promptly correct them. This verification process includes checking the raw video data for completeness, correct formatting, and corruption, ensuring its accuracy and integrity, facilitating subsequent frame selection. Second, effective frame selection requires sufficient equipment and personnel, ensuring both the quality of the equipment and the professionalism of the personnel. The equipment must be equipped with high-quality video players or editing software to accurately browse and select video frames. Furthermore, to effectively identify and select specific frames, the personnel must possess relevant expertise and extensive experience. Furthermore, standardized selection criteria are needed to ensure clarity and consistency in the steps for selecting specific frames, improving the accuracy and efficiency of the selection results. Finally, after selecting specific frames, manual data verification is required. The data inspection process includes manual inspection of the selected specific frames to ensure that they are consistent with the required video frames in the original video data; through statistical analysis and sampling inspection, to evaluate whether the selected specific frames meet the expected requirements, so as to further improve the quality and accuracy of the selected specific frames and ensure that the manually selected specific frames can meet the needs of subsequent analysis and application.

[0110] Manually selecting specific frames from large amounts of video data inevitably introduces errors, and manually checking the consistency of the selected results is also a time-consuming and labor-intensive task. During this process, factors such as the operator's experience level, emotional state, noise levels in the selection environment, and the organization of the selection process all affect the consistency of the selected results.

[0111] Traditional image processing techniques are used to analyze static images and have significant limitations when processing continuous dynamic images (i.e., continuous video frames). Because factors such as the position and shape of target objects in dynamic scenes, as well as the external environment, change over time, algorithms must be able to analyze single frames while also considering continuity and variability in the temporal dimension. Therefore, traditional image processing techniques cannot be directly applied to the selection of specific frames. There is an urgent need to provide an effective frame counting process that can detect key start and end frames by identifying and analyzing key node features in video data.

[0112] To address the above issues, this application provides a method for automatically detecting specific video frames in camera videos. This method uses template matching and edge detection to achieve automatic detection. The method pre-acquires the target terminal's control diagrams when executing different camera operation response processes, and pre-configures configuration parameters for determining the start and end frame detection criteria for each camera operation response process. Template matching is used to match the area containing the control within the video frame, and pixel changes in the area containing the control are detected. Video frames whose pixel changes meet the configuration parameters are selected. Edge detection can be used to select video frames whose edges meet the configuration parameters, i.e., to determine whether the image appears in the preview interface. Alternatively, frame difference analysis can be used to select video frames whose pixel changes meet the configuration parameters, i.e., to determine whether the image appears completely or has stopped zooming. This method automatically selects specific frames from a continuous video stream. This detection method reduces manual labor, improves detection efficiency, and reduces the subjective influence of human factors on the selection results, thereby improving detection accuracy and consistency. This detection method can quickly and accurately identify specific frames in the video to be detected and ensure that they meet standard requirements, providing a reliable data foundation for subsequent analysis and application.

[0113] Before explaining in detail the video frame detection method provided in the embodiment of the present application, the computer device and terminal involved in the embodiment of the present application are first described respectively.

[0114] Figure 2 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device can be the above Figure 1 Detection equipment 10 in, see Figure 2 The computer device includes at least one processor 201 , a communication bus 202 , a memory 203 and at least one communication interface 204 .

[0115] The processor 201 may be a microprocessor (including a central processing unit (CPU) etc.), an application-specific integrated circuit (ASIC), or may be one or more integrated circuits for controlling the execution of the program of the present application.

[0116] The communication bus 202 may include a pathway for transmitting information between the aforementioned components.

[0117] The memory 203 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compact disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 203 may exist independently and be connected to the processor 201 via the communication bus 202. The memory 203 may also be integrated with the processor 201.

[0118] The communication interface 204 uses any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0119] In a specific implementation, as an embodiment, the processor 201 may include one or more CPUs, such as Figure 2 CPU0 and CPU1 are shown in the figure.

[0120] In a specific implementation, as an embodiment, a computer device may include multiple processors, such as Figure 2 1 and 2. Each of these processors may be a single-core processor or a multi-core processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0121] In a specific implementation, as an embodiment, the computer device may further include an output device 206 and an input device 207. The output device 206 communicates with the processor 201 and can display information in a variety of ways. For example, the output device 206 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 207 communicates with the processor 201 and can receive user input in a variety of ways. For example, the input device 207 can be a mouse, a keyboard, a touch screen device, or a sensor device.

[0122] The computer device may be a general-purpose computer device or a dedicated computer device. In a specific implementation, the computer device may be a desktop computer, a portable computer, a network server, a PDA, a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of the present application do not limit the type of computer device.

[0123] The memory 203 is used to store the program code 210 for executing the solution of the present application, and the processor 201 is used to execute the program code 210 stored in the memory 203. The computer device can implement the video frame detection method provided in the embodiment of the present application through the processor 201 and the program code 210 in the memory 203.

[0124] Figure 3 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal 100 can be the above-mentioned Figure 1 The target terminal 30 in Figure 3The terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0125] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0126] The processor 110 may include one or more processing units, for example, an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0127] The controller may be the nerve center and command center of the terminal 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0128] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0129] The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0130] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0131] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the terminal 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0132] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0133] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied on the terminal 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0134] In some embodiments, antenna 1 of terminal 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that terminal 100 can communicate with the network and other devices through wireless communication technology.

[0135] Terminal 100 implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0136] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLED, or a quantum dot light-emitting diode (QLED). In some embodiments, terminal 100 may include one or N display screens 194, where N is an integer greater than one.

[0137] The terminal 100 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0138] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0139] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created by the terminal 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0140] The terminal 100 can implement audio functions, such as music playback and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D and the application processor.

[0141] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Terminal 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, terminal 100 detects the touch intensity based on pressure sensor 180A. Terminal 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0142] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor 180K can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the terminal 100, in a location different from that of the display screen 194.

[0143] In an embodiment of the present application, the camera operation response process includes at least the following scenarios: Scenario 1, starting the camera application to take a picture; Scenario 2, from receiving a trigger operation on the photo control to generating a thumbnail corresponding to the latest captured image in the image echo control (shot2see); Scenario 3, from receiving a trigger operation on the image echo control to displaying a presentation image corresponding to the thumbnail in the image echo control (see2review); Scenario 4, the camera application jumps from the gallery interface to the preview interface (Gallery2Preview); Scenario 5, switching between front-end shooting and rear-end shooting; Scenario 6, from receiving a trigger operation on the video control to starting recording; Scenario 7, switching from close-up shooting to long-range shooting; Scenario 8, from receiving a trigger operation on the photo control to completing the dynamic timer shooting (shutterlag); Scenario 9, executing the process in response to the trigger operation on the photo control (shot2Preview). The following takes the target terminal as an example of a mobile phone to illustrate the above scenarios one by one.

[0144] Below through Figure 4-Figure 8 Scenario 1 (the scenario of starting the camera application to take a picture) is described.

[0145] When the camera application is started for shooting, the mobile phone triggers the camera application icon (for example, clicks) based on the test instruction to realize the automatic triggering operation of the camera application icon. The test instruction can be initiated by the user or sent to the mobile phone by other devices, and this embodiment of the present application is not limited to this. In response to the triggering operation on the camera application icon, the mobile phone starts the camera application and displays the picture in the preview interface. The user-perceivable process duration refers to the time from the beginning of the change of the camera application icon to the appearance of the outline in the preview interface. In this process, the camera application icon begins to change and the outline appears in the preview interface as key nodes. Based on this, the operation response process of the mobile phone executing the startup camera application for shooting can be filmed by a high-speed camera to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0146] Take the example of shooting a still portrait of a person pasted on the wall with a mobile phone. Figure 4 As shown, Figure 4 The following figure shows four representative video frames in the video to be detected corresponding to the scene of starting the camera application to shoot. It can be understood that the video to be detected also includes other video frames. The process of starting the camera application on the mobile phone to shoot is as follows: Use an automated method to click the camera application icon on the display interface of the mobile phone, such as Figure 4 As shown in Figure A in the figure, the camera application icon changes color and shrinks, as shown in Figure A in the figure. Figure 4 As shown in Figure B, then Figure 4 As shown in Figure C, the camera application icon returns to normal; then the phone will display the picture in the preview interface. When the picture appears in the preview interface, it will go through the following steps: Figure 4 The slow emergence process shown in Figure C until Figure 4 Figure D in .

[0147] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (indicating that the camera application icon begins to change) and the end frame (indicating the first frame that the picture appears in the preview interface). The camera operation response process involves camera application icon recognition and picture recognition in the preview interface. The changes in the camera application icon can be identified by the frame difference method or the color detection method; when the picture appears in the preview interface, characteristic content (for example, white lines, etc.) will appear in the picture. The white lines can be identified by the edge detection operator extraction method to detect whether the picture appears in the preview interface. The edge detection operator includes but is not limited to the Canny operator, Laplacian operator, Sobel operator, Prewitt operator, Roberts operator, Marr-Hildreth operator, etc.

[0148] The video frame detection method provided by this application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the operation control diagram can be obtained in advance. The operation control diagram includes a camera application icon. The camera application icon is an operation control, such as Figure 4 As shown in Figure A in , the computer device pre-acquires an operation control diagram with a camera application icon. The operation control diagram and the video frame including the camera application icon are images from the same perspective, and the position of the camera application icon in the operation control diagram is the same as its position in the video frame. When detecting whether the camera application icon begins to change, the detection can be performed on the area where the camera application icon is located in the video frame (i.e., the first preset area), without paying attention to other areas. Based on the positional features of the area where the camera application icon is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the camera application icon in the video frame, and the area where the camera application icon is located in the video frame is treated as a specific area that needs attention.

[0149] When the phone's camera app icon is clicked, the color changes, and the icon shrinks first and then returns to normal. Figure 4 As shown in Figure B, Figure 4 Compared to Figure A in the figure, the camera application icon is smaller and grayed out, and then Figure 4 As shown in Figure C, the camera app icon returns to normal. Based on this, you can use color detection or frame difference method to detect whether the camera app icon has started to change. The following are explanations of each.

[0150] The color representation mode can adopt the RGB color space or the HSV color space, where RGB indicates red (Red), green (Green) and blue (Blue), and HSV indicates hue (Hue), saturation (Saturation) and value (Value). By extracting color features from the area where the camera application icon of the video frame in the video to be detected is located, color features are obtained. The color features can be expressed as a color histogram, a color matrix, a color set, a color aggregation vector or a color correlation graph. When judging whether a color change occurs between adjacent video frames, it can be achieved by comparing the similarity relationship between the color features of adjacent video frames. For example, the color similarity between the color features of adjacent video frames is calculated. When the color similarity is greater than the preset similarity, it indicates that the camera application icon begins to change. The first frame of the video frame whose color similarity is greater than the preset similarity is used as the first frame where the camera application icon begins to change, and the first frame where the camera application icon begins to change is used as the starting frame.

[0151] The frame difference method (i.e., the inter-frame difference method) is to subtract the pixel values ​​of the corresponding pixel points between frames to obtain the pixel difference between frames, and average the pixel differences of multiple pixel points in the frame as the frame difference. The frame difference method can also be to first average the pixel values ​​of multiple pixel points included in the frame to obtain the brightness of the frame, and then subtract the brightness of the frame from the frame as the frame difference. In the present application, the computer device uses the frame difference method to perform inter-frame difference on the area where the camera application icon of adjacent video frames in the video to be detected to obtain the frame difference. When the frame difference is greater than the preset difference value, it means that there is a more obvious difference between the adjacent video frames, that is, the camera application icon begins to change. The first frame with a frame difference value greater than the preset difference value is used as the first frame in which the camera application icon begins to change, and the first frame in which the camera application icon begins to change is used as the starting frame.

[0152] When using the frame difference method to perform inter-frame differences between adjacent video frames, the inter-frame differences can be performed on the grayscale images of adjacent video frames. Grayscale images can better reflect the differences in pixel changes. Based on this, for each video frame in the video to be detected, the grayscale of the video frame can be first transformed to obtain the corresponding grayscale image of the video frame. Then, the inter-frame differences of the grayscale images of adjacent video frames, where the camera app icon is located, can be performed to obtain the frame difference value.

[0153] Using grayscale images for inter-frame differencing can reduce computational complexity and improve data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the impact of video flicker on detection, improving the accuracy of data processing results. Focusing on the brightness changes of the camera app icon and enhancing the contrast of the area where the camera app icon is located makes it easier to detect when the camera app icon changes.

[0154] Among them, the preset similarity and preset difference values ​​are configuration parameters set by those skilled in the art based on the similarity threshold and difference threshold adopted in a large number of experiments. As long as it can be determined whether the camera application icon starts to change, the embodiment of the present application does not limit this.

[0155] In addition, the preset difference value can also be positively correlated with the degree of flicker of the camera in the mobile phone. The degree of flicker of the camera refers to the change in the brightness of the flash, that is, the intensity of the light emitted by the flash when taking a photo. The brighter the brightness of the flash (that is, the camera is in high-brightness mode), the brighter the captured image, and the preset difference value for judging whether the camera application icon has started to change should be set to a larger value. The smaller the brightness of the flash (that is, the camera is in low-brightness mode), the softer or dimmer the captured image, and the preset difference value for judging whether the camera application icon has started to change should be set to a smaller value. Setting the preset difference value according to the brightness of the camera increases the flexibility of the judgment conditions and further improves the accuracy of the video frame detection results.

[0156] In the embodiments of the present application, by matching the area where the camera application icon is located in the video frame, when detecting whether the color of the camera application icon begins to change, or using the frame difference method to detect whether there is a significant difference in the pixels of adjacent video frames, targeted area detection can be performed, thereby improving detection accuracy, reducing the amount of data required to determine whether pixels have changed, and improving detection efficiency. Furthermore, when detecting whether the camera application icon begins to change, color detection or frame difference method can be used for detection, increasing the diversity and richness of detection methods.

[0157] In the scenario where the camera application is started to shoot, after detecting the starting frame of the video to be detected, the computer device then detects the picture in the preview interface of the mobile phone. When the picture in the preview interface appears, the mobile phone will go through the following steps: Figure 4 The slow emergence process shown in Figure C until Figure 4 Figure D in the figure. In the present application, the computer device can realize the recognition of the picture in the preview interface by the edge detection method. The edge feature extraction is performed on the video frame located after the start frame in the video to be detected to obtain the line feature. The line feature can indicate the number of lines. When the number of lines is greater than the first number, it means that the picture appears in the preview interface and the camera operation response ends. The first frame of the video frame whose line feature indicates that the number of lines is greater than the first number is used as the first frame of the picture appearing in the preview interface, and the first frame of the picture appearing in the preview interface is used as the end frame. By comparing the number of lines that constitute the contour, it is determined whether the picture appears in the preview interface, without the need for manual detection, and the automatic detection process of the end frame is realized, thereby improving the efficiency and accuracy of video frame detection.

[0158] The image recognition detection effect in the preview interface is as follows Figure 5As shown, Figure 5 This is merely an example of edge detection. Figure 5 Figure A and Figure 5 The B picture in is the adjacent video frame. Figure 5 The preview interface in Figure A shows no lines detected, that is, no picture outline appears. Figure 5 The preview interface shown in Figure B detects lines, that is, the outline of the screen appears. The outline of the screen is a white line. Therefore, Figure 5 Figure B is the key node, and in practical applications, it is used as the end frame.

[0159] Among them, the first number is a configuration parameter set by technical personnel in this field based on a number of experimental thresholds adopted. As long as it can be determined whether the picture appears in the preview interface, this embodiment of the present application does not limit this.

[0160] The video frame detection method provided by the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, an operation response diagram can be obtained in advance. The operation response diagram is an interface diagram displayed by the mobile phone in response to a trigger operation (for example, a click operation) on an operation control. Figure 4 The D graph in the image. The operation response graph includes a still portrait (i.e., the first object) and the surrounding background. The background can be black or white, etc. The operation response graph and the video frame in which the still portrait is detected are images under the same viewing angle. The position of the still portrait in the operation response graph is the same as its position in the video frame. When detecting whether a still portrait appears in the preview interface, the detection can be performed on the area where the still portrait is located in the video frame (i.e., the second preset area), without paying attention to other areas. Based on the positional features of the area where the still portrait is located in the operation response graph, template matching is performed in the video frame after the starting frame, the position coordinates of the still portrait are determined in the video frame, and the area where the still portrait is located in the video frame is used as a specific area that needs attention. Then, edge feature extraction is performed on the second preset area of ​​the video frame after the starting frame to obtain the line features of the video frame after the starting frame.

[0161] In the embodiment of the present application, under normal circumstances, the appearance of the picture in the preview interface is the process of the appearance of the still portrait, and there is no need to pay attention to the background. By selecting the area corresponding to the still portrait in the video frame, targeted area detection can be performed, thereby improving the detection accuracy, reducing the edge feature extraction range, and improving the detection efficiency.

[0162] In this embodiment of the present application, the duration of the camera operation response process includes the camera application startup duration. After detecting the start frame and the end frame, the two are subtracted to obtain the number of captured frames, and the camera application startup duration is calculated based on the number of captured frames and the camera shooting frame rate.

[0163] The above scenario 1 involves the interaction between mobile phones, high-speed cameras and computer equipment, such as Figure 6 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Figure 6 ,The interaction process includes the following steps.

[0164] S11. A high-speed camera shoots the screen of the mobile phone.

[0165] Aim the high-speed camera at the screen of the mobile phone, then start the high-speed camera. In response to the start-up operation, the high-speed camera starts to shoot the screen of the mobile phone.

[0166] S12. Trigger the phone’s camera app icon.

[0167] S13. In response to the triggering operation, the mobile phone starts the camera application and displays the picture in the preview interface.

[0168] S14. The high-speed camera sends the captured video to be detected to the computer device.

[0169] S15. The computer device identifies whether the camera application icon starts to change in the video to be detected through a frame difference method or a color detection method, takes the first frame where the camera application icon starts to change as the starting frame, and identifies the picture in the preview interface, and takes the first frame where the picture appears in the preview interface as the ending frame.

[0170] As an example, the video frame detection method performed by the computer device in scenario 1 is as follows: Figure 7 Please refer to Figure 7 , the method comprises the following steps:

[0171] S151. Obtain the video to be detected and the corresponding operation control diagram and operation response diagram.

[0172] The operation control diagram includes the camera application icon, such as Figure 4 The operation response diagram is an interface diagram displayed by the mobile phone in response to the triggering operation of the operation control. The operation response diagram includes a still portrait (ie, the first object), such as Figure 4 As shown in Figure D.

[0173] S152. Perform region matching on the video frame in the video to be detected according to the position of the camera application icon in the operation control diagram to obtain a first preset region of the camera application icon in the video frame.

[0174] Based on the positional features of the area where the camera application icon is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the camera application icon in the video frame, and the area where the camera application icon is located in the video frame (i.e., the first preset area) is treated as a specific area that needs attention.

[0175] S153. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0176] S154. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0177] S155. Determine whether the frame difference is greater than a preset difference value?

[0178] If yes, execute S156; if no, execute S152.

[0179] S156. The first frame whose frame difference is greater than the preset difference value is used as the first frame in which the camera application icon starts to change.

[0180] S157. Based on the position of the first object in the operation response graph, perform region matching on the video frames after the start frame in the video to be detected to obtain a second preset region of the first object in the video frame.

[0181] Based on the positional features of the area where the first object is located in the operation response graph, template matching is performed in the video frame located after the start frame to determine the position coordinates of the first object in the video frame, and the area where the first object is located in the video frame (i.e., the second preset area) is taken as a specific area that needs attention.

[0182] S158. Extract edge features from the second preset areas of the video frames after the start frame in the video to be detected, and obtain line features of each video frame after the start frame.

[0183] S159. Determine whether the number of lines indicated by the line feature is greater than the first number?

[0184] If yes, execute S161; if no, execute S157.

[0185] S161. The first frame of the video frame in which the number of line feature indication lines is greater than the first number is used as the first frame of the picture appearing in the preview interface.

[0186] As an example, the video frame detection method executed by the computer device in scenario 1 can also be as follows: Figure 8 Please refer to Figure 8 , the method comprises the following steps:

[0187] S151. Obtain the video to be detected and the corresponding operation control diagram and operation response diagram.

[0188] S152. Perform region matching on the video frame in the video to be detected according to the position of the camera application icon in the operation control diagram to obtain a first preset region of the camera application icon in the video frame.

[0189] S162. Perform color extraction on a first preset area of ​​a video frame in the video to be detected to obtain a color feature.

[0190] S163. Determine the color similarity between adjacent video frames based on the color features.

[0191] S164. Determine whether the color similarity is greater than a preset similarity?

[0192] If yes, execute S164; if no, execute S152.

[0193] S165. The first frame of the video frame whose color similarity is greater than the preset similarity is used as the first frame in which the camera application icon starts to change.

[0194] S157. Based on the position of the first object in the operation response graph, perform region matching on the video frames after the start frame in the video to be detected to obtain a second preset region of the first object in the video frame.

[0195] S158. Extract edge features from the second preset areas of the video frames after the start frame in the video to be detected, and obtain line features of each video frame after the start frame.

[0196] S159. Determine whether the number of lines indicated by the line feature is greater than the first number?

[0197] If yes, execute S161; if no, execute S157.

[0198] S161. The first frame of the video frame in which the number of line feature indication lines is greater than the first number is used as the first frame of the picture appearing in the preview interface.

[0199] In this application Figure 6-Figure 8 This section introduces the scenario of launching the camera application to take pictures. The implementation process and the beneficial effects achieved can be found in the above Figure 4 and Figure 5 The description is not repeated here.

[0200] Below through Figures 9-12 Scenario 2 (the scenario from receiving a trigger operation on the photo control to generating a thumbnail corresponding to the latest captured image in the image echo control) is described.

[0201] When taking pictures with a mobile phone, the mobile phone triggers the photo control (also known as the photo button) in the camera application interface based on the test instruction (for example, a click operation) to realize the automatic triggering operation of the photo control. In response to the triggering operation of the photo control, the mobile phone performs the shooting operation and displays a thumbnail of the latest captured image in the image echo control of the camera application interface. The user-perceivable process duration refers to the time from the start of the change of the photo control to the complete appearance of the thumbnail in the image echo control. In this process, the start of the change of the photo control and the complete appearance of the thumbnail in the image echo control are key nodes. Based on this, the operation response process of the mobile phone performing the photo shooting can be filmed by a high-speed camera to obtain a video to be tested, and the video to be tested includes multiple continuous video frames.

[0202] like Figure 9 As shown, Figure 9 Four video frames in the photo shooting scene are shown. It is understandable that the video to be detected in this scene also includes other video frames. The process of triggering the photo shooting control to shoot is as follows: Use an automated method to click the photo shooting control on the shooting interface of the camera application, such as Figure 9 As shown in Figure A; the photo control is reduced, as shown in Figure 9 As shown in Figure B, then Figure 9 As shown in Figure C, the camera controls return to normal; then the phone will display thumbnails, which will go through the following steps: Figure 9 The zooming process shown in Figure C in the figure is repeated until the thumbnail is zoomed in. Figure 9 Figure D in .

[0203] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the photo control starts to change) and the end frame (the last frame indicating that the thumbnail appears completely). The camera operation response process involves photo control recognition and complete picture recognition. The changes in the photo control can be identified by the frame difference method; complete picture recognition can be achieved by calculating the frame difference. When the picture is zoomed, the frame difference is large, and the frame difference is small after the zoom is completed. Through the change pattern of the frame difference amplitude, it is recognized that the picture display is complete and the zooming has stopped. That is to say, when a large area difference appears in the frame difference, it means that the picture is zooming. When a large area difference stops appearing in the frame difference, it means that the picture zooming is complete. The last frame that appears to be zoomed is the frame where the picture stops zooming. The following explains the photo control recognition and complete picture recognition.

[0204] The video frame detection method provided by this application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the operation control diagram can be obtained in advance, and the operation control diagram includes a photo taking control. Figure 9As shown in Figure A in the figure, the computer device pre-acquires an operation control diagram with a photo control. The operation control diagram and the video frame including the photo control are images from the same perspective, and the position of the photo control in the operation control diagram is the same as its position in the video frame. When detecting whether the photo control begins to change, the detection can be performed on the area where the photo control is located in the video frame (i.e., the first preset area), without paying attention to other areas. Based on the positional features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the photo control are determined in the video frame, and the area where the photo control is located in the video frame is treated as a specific area that needs to be paid attention to.

[0205] When the camera control of the phone is triggered, the control will shrink first and then return to normal. Figure 9 As shown in Figure B, Figure 9 Compared with Figure A in the figure, the camera control is reduced, and then Figure 9 As shown in Figure C, the camera control has returned to normal. Based on this, the present application uses a frame difference method to perform inter-frame differences on the area containing the camera control in adjacent video frames in the video to be tested, obtaining a frame difference value. When the frame difference value is greater than a preset difference value, it indicates that a significant difference has occurred between the adjacent video frames, indicating that the camera control has begun to change. The first frame with a frame difference greater than the preset difference value is regarded as the first frame where the camera control begins to change, and the first frame where the camera control begins to change is regarded as the starting frame.

[0206] When using the frame difference method to perform inter-frame differencing between adjacent video frames, you can first convert the video frames to grayscale before performing inter-frame differencing. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0207] Among them, the recognition process of the photo control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0208] The difference effect of control behavior detection is as follows Figure 10 As shown, Figure 10 This is just an example of the camera controls on a particular phone. Figure 10 Indicates the frame difference between the current video frame and the previous video frame after the inter-frame difference, which is represented by Figure 10 It can be seen that the camera control starts to change in the current video frame. In this application, the computer device determines whether the camera control starts to change by determining whether the frame difference value is greater than a preset difference value. Figure 10 It is a key node and is used as the starting frame in practical applications.

[0209] In an embodiment of the present application, by matching the area where the photo control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy and reducing the amount of data used to determine whether the pixels have changed, thereby improving the detection efficiency.

[0210] In the scene from taking a photo to thumbnail, after detecting the starting frame of the video to be detected, the phone will display a thumbnail. When the thumbnail appears, it can be generated through a sliding animation or a zooming animation. The video frame detection method provided by this application can be applied to a variety of thumbnail generation schemes. This example uses the zooming animation to generate thumbnails as an example. When an incorrect thumbnail appears, it can be generated through the following methods: Figure 9 The zooming process shown in Figure C in the figure is repeated until the thumbnail is zoomed in. Figure 9 Figure D in the figure. In the present application, for the process of displaying thumbnails in the form of sliding animation and zooming animation, the computer equipment can realize complete picture recognition by the method of picture contour detection. The adjacent video frames located after the start frame in the video to be detected are subjected to inter-frame difference to obtain the first frame difference data. The first frame difference data includes the pixel difference of multiple pixel points, and the pixel difference of multiple pixel points constitutes the picture contour. The area of ​​the picture contour reflects the difference between adjacent video frames; the larger the area of ​​the picture contour, the greater the difference between adjacent video frames, indicating that the thumbnail is being scaled; the smaller the area of ​​the picture contour, the smaller the difference between adjacent video frames, indicating that the thumbnail scaling is completed (that is, the thumbnail stops scaling or the thumbnail appears completely). The video frame whose area of ​​the picture contour indicated by the first frame difference data is less than the first preset area is used as the last frame of the thumbnail change (that is, the first frame where the thumbnail stops scaling), and the last frame of the thumbnail change is used as the end frame.

[0211] In an embodiment of the present application, the frame difference variation pattern of adjacent video frames is identified, and whether a large-area difference appears in the frame difference is determined to determine whether the thumbnail appears completely (i.e., the zoom is stopped). No manual detection is required, and the end frame detection process is automatically realized, thereby improving the video frame detection efficiency and detection accuracy.

[0212] Among them, the first preset area is a configuration parameter set by those skilled in the art based on the area threshold adopted in a large number of experiments. As long as it can be determined whether the thumbnail appears completely or stops zooming, the embodiment of the present application does not limit this.

[0213] The video frame detection method provided by this application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the shooting interface image can be obtained in advance, such as Figure 9 Figure C and Figure 9As shown in Figure D in . The shooting interface diagram includes an image echo control. The shooting interface diagram and the video frame where the thumbnail is detected are images from the same perspective, and the position of the image echo control in the shooting interface diagram is the same as the position of the thumbnail in the video frame. When detecting whether the thumbnail appears completely, the area where the thumbnail is located in the video frame (i.e., the third preset area) can be detected without paying attention to other areas. According to the positional characteristics of the area where the image echo control is located in the shooting interface diagram, template matching is performed in the video frame, the position coordinates of the thumbnail are determined in the video frame, and the area where the thumbnail is located in the video frame is used as a specific area that needs attention. Then, inter-frame difference is performed on the area where the thumbnail of the adjacent video frames after the starting frame in the video to be detected is located to obtain the first frame difference data, and the video frame whose area of ​​the picture outline indicated by the first frame difference data is smaller than the first preset area is used as the last frame of the thumbnail change.

[0214] In an embodiment of the present application, targeted area detection can be performed by selecting the area corresponding to the thumbnail in the video frame, thereby improving detection accuracy, reducing the amount of data required to determine whether a pixel has changed, and improving detection efficiency.

[0215] When using the frame difference method to perform inter-frame differencing on adjacent video frames, for each video frame following the starting frame in the video to be detected, the video frame is first converted to grayscale to obtain a corresponding grayscale image (i.e., a second grayscale image). Inter-frame differencing is then performed on the region containing the thumbnails of the grayscale images of the adjacent video frames to obtain the first frame difference data. Using grayscale images for inter-frame differencing reduces computational complexity and increases data processing speed. Furthermore, grayscale images can eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0216] In this embodiment of the present application, the duration of the camera operation response process includes the photo response duration. After detecting the start frame and the end frame, the two are subtracted to obtain the number of shooting frames, and the photo response duration is calculated based on the number of shooting frames and the camera shooting frame rate.

[0217] The above scenario 2 involves the interaction between mobile phones, high-speed cameras and computer equipment, such as Figure 11 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Figure 11 ,The interaction process includes the following steps.

[0218] S21. A high-speed camera films the screen of the mobile phone.

[0219] S22. Trigger the photo control in the mobile phone shooting interface.

[0220] S23. In response to the triggering operation, the mobile phone displays a thumbnail corresponding to the latest captured image in the shooting interface.

[0221] S24. The high-speed camera sends the captured video to be detected to the computer device.

[0222] S25. The computer device uses a frame difference method to identify whether the photo control begins to change in the video to be detected, and uses the first frame in which the photo control begins to change as the starting frame. The computer device also uses a frame difference method to identify whether the thumbnail in the image echo control appears completely, and uses the last frame in which the thumbnail changes as the ending frame.

[0223] As an example, the video frame detection method performed by the computer device in scenario 2 is as follows: Figure 12 Please refer to Figure 12 , the method comprises the following steps:

[0224] S251. Obtain the video to be detected and the corresponding operation control diagram and shooting interface diagram.

[0225] The operation control diagram includes a photo taking control, such as Figure 9 As shown in Figure A. The shooting interface includes image echo controls, such as Figure 9 As shown in Figure C.

[0226] S252. Based on the position of the photo control in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the photo control in the video frame.

[0227] Based on the positional features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the photo control in the video frame, and the area where the photo control is located in the video frame (i.e., the first preset area) is treated as a specific area that needs attention.

[0228] S253. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0229] S254. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0230] S255. Determine whether the frame difference is greater than a preset difference value?

[0231] If yes, execute S256; if no, execute S252.

[0232] S256. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the camera control to start changing.

[0233] S257. Based on the position of the image echo control in the shooting interface diagram, perform region matching on the video frames after the start frame in the video to be detected to obtain a third preset region of the thumbnail in the video frame.

[0234] According to the positional features of the area where the image echo control is located in the shooting interface image, template matching is performed in the video frame after the start frame to determine the position coordinates of the thumbnail in the video frame, and the area where the thumbnail is located in the video frame (i.e., the third preset area) is regarded as a specific area that needs attention.

[0235] S258. Perform grayscale conversion on the video frames after the start frame in the video to be detected to obtain a second grayscale image corresponding to the video frame.

[0236] S259. Perform inter-frame difference on a third preset area of ​​the second grayscale image corresponding to adjacent video frames to obtain first frame difference data.

[0237] S260: Determine whether the area of ​​the picture outline indicated by the first frame difference data is smaller than a first preset area.

[0238] S261. The video frame whose first frame difference data indicates that the area of ​​the picture outline is smaller than the first preset area is used as the last frame of the thumbnail change.

[0239] In this application Figure 11 and Figure 12 This article introduces the process from receiving a trigger operation on the photo control to generating a thumbnail (shot2see) corresponding to the latest captured image in the image echo control. This scenario can include front highlight, front low light, rear highlight, and rear low light. The implementation process and the beneficial effects achieved can be found in the above. Figure 9 and Figure 10 The description is not repeated here.

[0240] Below through Figure 13-16 The scenario 3 (the scenario from receiving a trigger operation on the image echo control to displaying a presentation image corresponding to the thumbnail in the image echo control) is described.

[0241] When viewing the picture just taken, the mobile phone triggers the image echo control (also known as the image echo button) based on the test instruction (for example, a click operation) to realize the automatic triggering operation process of the image echo control. In response to the triggering operation of the image echo control, the mobile phone fully displays the enlarged thumbnail in the gallery interface of the camera application (that is, the thumbnail is fully opened to become a display image). The user-perceivable process duration refers to the time from the beginning of the image echo control to the complete opening of the display image. In this process, the beginning of the change of the image echo control and the complete appearance of the display image are key nodes. The display image corresponding to the thumbnail is the latest image taken, and triggering the image echo control can display the latest image taken. Based on this, the high-speed camera can be used to shoot the mobile phone's operation response process of viewing thumbnails to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0242] Take the example of shooting a still portrait of a person pasted on the wall with a mobile phone. Figure 13 As shown, Figure 13 The following figure shows four representative video frames in the video to be detected corresponding to the scene from thumbnail to display image. It is understandable that the video to be detected in this scene also includes other video frames. The process from thumbnail to display image is as follows: Use an automated method to click the image echo control on the shooting interface of the camera application, such as Figure 13 As shown in Figure A in the figure; the image echo control is reduced, that is, the thumbnail displayed in the image echo control will be as follows Figure 13 The zooming process is shown in Figure B. Next, the phone will display a display image, which will go through the following steps: Figure 13 The zooming process shown in Figure C in the figure is repeated until the display stops zooming, as shown in Figure C. Figure 13 Figure D in .

[0243] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the image echo control starts to change) and the end frame (indicating that the display image is fully opened or appears). The camera operation response process involves image echo control recognition and complete screen recognition. The changes in the image echo control can be identified by the frame difference method; complete screen recognition can be achieved by calculating the frame difference. When the picture is zoomed, the frame difference is large, and the frame difference is small after the zoom is completed. Through the change pattern of the frame difference amplitude, it is recognized that the picture display is complete and the zooming has stopped. The image echo control recognition and complete screen recognition are explained below.

[0244] The video frame detection method provided by this application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the operation control diagram can be obtained in advance. The operation control diagram includes the image echo control. Figure 13As shown in Figure A in , the computer device pre-acquires an operation control diagram with an image echo control. The operation control diagram and the video frame including the image echo control are images from the same perspective, and the position of the image echo control in the operation control diagram is the same as its position in the video frame. When detecting whether the image echo control begins to change, the detection can be performed on the area where the image echo control is located in the video frame (i.e., the first preset area), without paying attention to other areas. Based on the positional features of the area where the image echo control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the image echo control are determined in the video frame, and the area where the image echo control is located in the video frame is treated as a specific area that needs attention.

[0245] When the image echo control of the mobile phone is clicked, the control will shrink, such as Figure 13 As shown in Figure B, Figure 13 Compared with Figure A in the video, the image echo control is reduced. That is, it is necessary to detect the zooming behavior of the thumbnail to detect the first frame of clicking the image echo control. Based on this, the present application adopts the frame difference method to perform inter-frame difference on the area where the image echo control of adjacent video frames in the video to be detected, and obtains the frame difference value. When the frame difference value is greater than the preset difference value, it means that there is a more obvious difference between the adjacent video frames, that is, the image echo control begins to change. The first frame with a frame difference value greater than the preset difference value is used as the first frame where the image echo control starts to change, and the first frame where the image echo control starts to change is used as the starting frame.

[0246] When using the frame difference method to perform inter-frame differencing between adjacent video frames, you can first convert the video frames to grayscale before performing inter-frame differencing. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0247] Among them, the recognition process of the image echo control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0248] In an embodiment of the present application, by matching the area where the image echo control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy, reducing the amount of data used to determine whether the pixels have changed, and improving the detection efficiency.

[0249] In the scenario from thumbnail to display image, after the start frame of the video to be detected is detected, the enlarged thumbnail (ie, display image) will appear next. When the display image appears, it will go through the following steps: Figure 13The scaling process shown in Figure C until Figure 13 Figure D in the image. In the present application, the computer device can realize complete picture recognition by the method of picture contour detection. Inter-frame difference is performed on the adjacent video frames located after the starting frame in the video to be detected to obtain second frame difference data. The second frame difference data includes the pixel difference of multiple pixel points, and the pixel difference of multiple pixel points constitutes the picture contour. The area of ​​the picture contour reflects the difference between adjacent video frames; the larger the area of ​​the picture contour, the larger the difference between adjacent video frames, indicating that the display image is being scaled; the smaller the area of ​​the picture contour, the smaller the difference between adjacent video frames, indicating that the display image is scaled (that is, the display image stops scaling or the display image appears completely). The video frame whose area of ​​the picture contour indicated by the second frame difference data is less than the second preset area is used as the first frame of the display image fully opened (that is, the first frame of the display image stops scaling, or the last frame of the display image change), and the first frame of the display image fully opened is used as the end frame.

[0250] The full screen stops zooming and the detection effect is as follows Figure 14 As shown, Figure 14 It is merely an example of the appearance of a complete picture. Figure 14 The frame difference binary image after the current video frame and the previous video frame of the current video frame are shown in FIG. Figure 14 It can be seen that the difference between adjacent video frames is small, indicating that the complete picture is scaled. In this application, the computer device determines whether the complete picture is scaled by determining whether the area of ​​the frame difference binary image is smaller than the preset area. Figure 14 It is a key node and is used as the end frame in practical applications.

[0251] By identifying the changing patterns of frame differences between adjacent video frames and judging whether there are large-area differences in the frame differences, we can determine whether the display image is fully opened (i.e., the display image stops zooming). Without manual detection, the end frame detection process is automatically realized, thereby improving the efficiency and accuracy of video frame detection.

[0252] Among them, the second preset area is a configuration parameter set by technical personnel in this field based on the area threshold adopted in a large number of experimental processes. As long as it can be determined whether the display image is fully opened or the display image stops zooming, the embodiment of this application does not limit this.

[0253] When using the frame difference method to perform inter-frame difference analysis on adjacent video frames, for each video frame following the starting frame in the video to be detected, the video frame can first be grayscale converted to obtain the corresponding grayscale image of the video frame. Then, inter-frame difference analysis is performed on the grayscale images of adjacent video frames to obtain the second frame difference data. Using grayscale images for inter-frame difference analysis can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0254] In this embodiment of the present application, the duration of the camera operation response process includes the echo duration. After detecting the start frame and the end frame, the two are subtracted to obtain the number of captured frames, and the echo duration is calculated based on the number of captured frames and the camera shooting frame rate.

[0255] The above scenario 3 involves the interaction between mobile phones, high-speed cameras and computer equipment, such as Figure 15 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Figure 15 ,The interaction process includes the following steps.

[0256] S31. A high-speed camera shoots the screen of a mobile phone.

[0257] S32. Trigger the image echo control on the mobile phone shooting interface.

[0258] S33. In response to the triggering operation, the mobile phone displays the full image corresponding to the thumbnail in the shooting interface.

[0259] S34. The high-speed camera sends the captured video to be detected to the computer device.

[0260] S35. The computer device uses a frame difference method to identify whether the image echo control has begun to change in the video frame to be detected, and uses the first frame in which the image echo control begins to change as the starting frame. The computer device also uses a frame difference method to identify whether the display image corresponding to the thumbnail is fully opened, and uses the first frame in which the display image is fully opened as the ending frame.

[0261] As an example, the video frame detection method performed by the computer device in scenario 3 is as follows: Figure 16 Please refer to Figure 16 , the method comprises the following steps:

[0262] S351. Obtain the video to be detected and the corresponding operation control diagram.

[0263] The operation control diagram includes image echo controls, such as Figure 13As shown in Figure A.

[0264] S352. Based on the position of the image echo control in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the image echo control in the video frame.

[0265] Based on the positional features of the area where the image echo control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the image echo control in the video frame, and the area where the image echo control is located in the video frame (i.e., the first preset area) is treated as a specific area that needs attention.

[0266] S353. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0267] S354. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0268] S355. Determine whether the frame difference is greater than a preset difference value?

[0269] If yes, execute S356; if no, execute S352.

[0270] S356. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the image echo control to start changing.

[0271] S357. Perform grayscale conversion on the video frames after the start frame in the video to be detected to obtain a grayscale image corresponding to the video frame.

[0272] S358. Perform inter-frame difference on the grayscale images of adjacent video frames located after the start frame in the video to be detected to obtain second frame difference data.

[0273] S359. Determine whether the area of ​​the picture outline indicated by the second frame difference data is smaller than a second preset area?

[0274] If yes, execute S360; if no, execute S357.

[0275] S360. The video frame whose second frame difference data indicates that the area of ​​the picture outline is smaller than the second preset area is used as the first frame of the fully opened display image.

[0276] In this application Figure 15 and Figure 16 This article describes the process from receiving a trigger operation on the image echo control to displaying the corresponding presentation image (see2review) of the thumbnail in the image echo control. The implementation process and the beneficial effects achieved can be found in the above Figure 13 and Figure 14 The description is not repeated here.

[0277] Below through Figure 17-Figure 19 Scenario 4 (the scene in which the camera application jumps from the gallery interface to the preview interface) is described.

[0278] When you need to continue shooting after viewing the taken pictures, the mobile phone triggers the return control (also known as the return button) based on the test instruction (for example, clicks on it) to realize the automatic triggering operation process of the return control. In response to the triggering operation of the return control, the mobile phone returns to the preview interface of the camera application. The user-perceivable process duration refers to the time from the return control starting to change to the appearance of the outline in the preview interface. In this process, the return control starts to change and the outline appears in the preview interface as key nodes. Based on this, the operation response process of the mobile phone returning to continue shooting after viewing the taken pictures can be captured by a high-speed camera to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0279] Take the example of shooting a still portrait of a person pasted on the wall with a mobile phone. Figure 17 As shown, Figure 17 The following figure shows four representative video frames in the video to be detected corresponding to the scene of jumping from the gallery interface of the camera application to the preview interface. It is understandable that the video to be detected in this scene also includes other video frames. The process of jumping from the gallery interface of the camera application to the preview interface is as follows: Use an automated method to click the return control on the gallery interface of the camera application, such as Figure 17 As shown in Figure A; return to the control to zoom out, as ... Figure 17 As shown in Figure B; the phone will then display the preview screen. When the preview screen appears, it will go through the following steps: Figure 17 The fuzzy process shown in Figure C until Figure 17 Figure D in .

[0280] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the return control starts to change) and the end frame (the first frame indicating that the picture appears in the preview interface). The camera operation response process involves return control identification and picture identification in the preview interface. The change of the return control can be identified by the frame difference method; the white lines are identified by the edge detection operator extraction method to detect whether the picture appears in the preview interface. The return control identification and picture identification in the preview interface are explained below.

[0281] The video frame detection method provided by this application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the operation control diagram can be obtained in advance. The operation control diagram includes a return control, such as Figure 17As shown in Figure A in , the computer device pre-acquires an operation control diagram with a return control. The operation control diagram and the video frame including the return control are images from the same perspective, and the position of the return control in the operation control diagram is the same as its position in the video frame. When detecting whether the return control begins to change, the detection can be performed on the area where the return control is located in the video frame (i.e., the first preset area), without paying attention to other areas. Based on the positional features of the area where the return control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the return control are determined in the video frame, and the area where the return control is located in the video frame is treated as a specific area that needs attention.

[0282] When the phone's back control is clicked, the control will shrink, such as Figure 17 As shown in Figure B, Figure 17 Compared with Figure A in the video, the return control is smaller. Based on this, the present application uses the frame difference method to perform inter-frame difference on the area where the return control of adjacent video frames in the video to be detected to obtain a frame difference value. When the frame difference value is greater than the preset difference value, it means that there is a more obvious difference between the adjacent video frames, that is, the return control begins to change. The first frame whose frame difference value is greater than the preset difference value is used as the first frame where the return control begins to change, and the first frame where the return control begins to change is used as the starting frame.

[0283] When using the frame difference method to perform inter-frame differencing between adjacent video frames, you can first convert the video frames to grayscale before performing inter-frame differencing. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0284] Among them, the recognition process of the return control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0285] In an embodiment of the present application, by matching the area where the return control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy and reducing the amount of data used to determine whether the pixels have changed, thereby improving the detection efficiency.

[0286] In the scene from the gallery interface of the camera application to the preview interface, after detecting the starting frame of the video to be detected, the computer device then detects the screen of the preview interface of the mobile phone. When the screen in the preview interface appears, the mobile phone will go through the following steps: Figure 17 The fuzzy process shown in Figure C until Figure 17Figure D in the figure. In this application, the computer device can recognize the image in the preview interface by using an edge detection method. The recognition process of the image in the preview interface is consistent with the recognition process of the image in the preview interface in the scene of starting the camera application for shooting. Its implementation process and the technical effects achieved can be referred to the recognition process of the image in the preview interface above, and will not be repeated here.

[0287] By comparing the number of lines that make up the outline, it is determined whether the picture appears in the preview interface without manual detection, and the end frame detection process is automatically realized, which improves the efficiency and accuracy of video frame detection.

[0288] In this embodiment of the present application, the duration of the camera operation response process includes the interface jump duration. After detecting the start frame and the end frame, the two are subtracted to obtain the number of shooting frames, and the interface jump duration is calculated based on the number of shooting frames and the camera shooting frame rate.

[0289] Scenario 4 above involves the interaction between mobile phones, high-speed cameras and computer equipment, such as Figure 18 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Figure 18 ,The interaction process includes the following steps.

[0290] S41. A high-speed camera films the screen of a mobile phone.

[0291] S42. Trigger the return control in the gallery interface.

[0292] S43. In response to the triggering operation, the mobile phone displays the picture in the preview interface.

[0293] S44. The high-speed camera sends the captured video to be detected to the computer device.

[0294] S45. The computer device uses a frame difference method to identify whether the return control begins to change in the video to be detected, and uses the first frame where the return control begins to change as the starting frame. The computer device also identifies whether a picture appears in the preview interface, and uses the first frame where the picture appears in the preview interface as the ending frame.

[0295] As an example, the video frame detection method performed by the computer device in scenario 4 is as follows: Figure 19 Please refer to Figure 19 , the method comprises the following steps:

[0296] S451. Obtain the video to be detected and the corresponding operation control diagram and operation response diagram.

[0297] The operation control diagram includes return controls, such as Figure 17 The operation response diagram is an interface diagram displayed by the mobile phone in response to the triggering operation of the operation control. The operation response diagram includes a still portrait (ie, the first object), such as Figure 17 As shown in Figure D.

[0298] S452. Based on the position of the return control in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the return control in the video frame.

[0299] Based on the positional features of the area where the return control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the return control in the video frame, and the area where the return control is located in the video frame (i.e., the first preset area) is treated as a specific area that needs attention.

[0300] S453. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0301] S454. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0302] S455. Determine whether the frame difference is greater than a preset difference value?

[0303] If yes, execute S456; if no, execute S452.

[0304] S456. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the return control to start changing.

[0305] S457. Based on the position of the first object in the operation response graph, perform region matching on the video frames after the start frame in the video to be detected to obtain a second preset region of the first object in the video frame.

[0306] Based on the positional features of the area where the first object is located in the operation response graph, template matching is performed in the video frame located after the start frame to determine the position coordinates of the first object in the video frame, and the area where the first object is located in the video frame (i.e., the second preset area) is taken as a specific area that needs attention.

[0307] S458. Extract edge features from the second preset areas of the video frames after the start frame in the video to be detected, and obtain line features of each video frame after the start frame.

[0308] S459. Determine whether the number of lines indicated by the line feature is greater than the first number?

[0309] If yes, execute S460; if no, execute S457.

[0310] S460. The first frame of the video frame in which the number of line feature indication lines is greater than the first number is used as the first frame of the picture appearing in the preview interface.

[0311] In this application Figure 18 and Figure 19 This section introduces the scenario of the camera application jumping from the gallery interface to the preview interface (Gallery2Preview). The implementation process and the beneficial effects achieved can be found in the above Figure 17 The description is not repeated here.

[0312] Below through Figure 20-22 Scene 5 (the scene of switching between front-camera shooting and rear-camera shooting) will be described.

[0313] This scenario includes switching from front-facing shooting to rear-facing shooting and from rear-facing shooting to front-facing shooting. When switching the camera for shooting, the mobile phone triggers the flip control (also called the flip button) based on the test instruction (for example, a click operation) to realize the automatic triggering operation of the flip control. In response to the triggering operation of the flip control, the mobile phone displays the preview interface of the front camera or the rear camera. The user-perceivable process duration refers to the time from the beginning of the change of the flip control to the appearance of the outline in the preview interface. In this process, the beginning of the change of the flip control and the appearance of the outline in the preview interface are key nodes. Based on this, the process of switching the front camera or the rear camera of the mobile phone can be filmed by a high-speed camera to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0314] For example, the rear camera of a mobile phone takes a picture of a still person pasted on a wall, and the front camera of the mobile phone takes a picture of a still picture. Figure 20 As shown, Figure 20 The following figure shows five representative frames of the video to be detected corresponding to the scene of switching from rear-facing shooting to front-facing shooting. It is understandable that the video to be detected in this scene also includes other video frames. The process of switching from the rear camera to the front camera is as follows: Use an automated method to click the flip control on the shooting interface of the camera application, such as Figure 20 As shown in Figure A; flip the control to zoom out, as shown in Figure Figure 20 As shown in Figure B, then Figure 20 As shown in Figure C, the flip control returns to normal; then the phone will display the screen in the preview interface, and when the screen in the preview interface appears, it will go through the following steps: Figure 20 Figure C and Figure 20 The fuzzy process shown in Figure D until Figure 20 Figure E in the figure means that the image in the preview interface gradually changes from clear to blurred and then becomes clear again.

[0315] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the flip control starts to change) and the end frame (the first frame indicating that the picture appears in the preview interface). The camera operation response process involves flip control recognition and picture recognition in the preview interface. The changes in the flip control can be identified by the frame difference method; the white lines are identified by the edge detection operator extraction method to detect whether the picture appears in the preview interface. The following describes the flip control recognition and picture recognition in the preview interface.

[0316] The video frame detection method provided by this application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the operation control diagram can be obtained in advance, and the operation control diagram includes a flip control. Figure 20 As shown in Figure A in , the computer device pre-acquires an operation control diagram with a flip control. The operation control diagram and the video frame including the flip control are images from the same perspective, and the position of the flip control in the operation control diagram is the same as its position in the video frame. When detecting whether the flip control begins to change, the detection can be performed on the area where the flip control is located in the video frame (i.e., the first preset area), without paying attention to other areas. Based on the positional features of the area where the flip control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the flip control are determined in the video frame, and the area where the flip control is located in the video frame is treated as a specific area that needs attention.

[0317] When the flip control of the mobile phone is clicked, the control will shrink first and then return to normal, such as Figure 20 As shown in Figure B, Figure 20 Compared to Figure A in the figure, flip the control to zoom out, and then Figure 20 As shown in Figure C in the figure, the flip control returns to normal. Based on this, the present application uses the frame difference method to perform inter-frame difference on the area where the flip control is located in adjacent video frames in the video to be detected to obtain a frame difference value. When the frame difference value is greater than the preset difference value, it indicates that there is a relatively obvious difference between the adjacent video frames, that is, the flip control begins to change. The first frame with a frame difference value greater than the preset difference value is regarded as the first frame where the flip control begins to change, and the first frame where the flip control begins to change is regarded as the starting frame.

[0318] When using the frame difference method to perform inter-frame differencing between adjacent video frames, you can first convert the video frames to grayscale before performing inter-frame differencing. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0319] Among them, the recognition process of the flip control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0320] In an embodiment of the present application, by matching the area where the flip control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy and reducing the amount of data used to determine whether the pixels have changed, thereby improving the detection efficiency.

[0321] In the scenario of switching from rear-facing shooting to front-facing shooting, after detecting the starting frame of the video to be detected, the computer device then detects the picture in the preview interface of the mobile phone. When the picture in the preview interface appears, the mobile phone will go through the following steps: Figure 20 Figure C and Figure 20 The fuzzy process shown in Figure D until Figure 20 Figure E in the figure, that is, the picture in the preview interface gradually changes from clear to blurred and then becomes clear again. In the present application, the computer device can realize the recognition of the picture in the preview interface by the edge detection method. The edge feature extraction is performed on the video frame located after the starting frame in the video to be detected to obtain the line feature. The line feature can indicate the number of lines. The process in which the picture in the preview interface gradually changes from clear to blurred and then becomes clear again is reflected in the gradual decrease and then gradual increase of the number of lines. In the process of detecting that the line feature indicates that the number of lines gradually becomes smaller, when the number of lines is less than the second number, it means that the picture in the preview interface becomes blurred, and the number of lines is continued to be counted. In the process of detecting that the line feature indicates that the number of lines of multiple video frames gradually becomes larger, when the number of lines is greater than the first number, it means that the picture in the preview interface becomes clear again. Therefore, the first frame of the video frame whose number of lines is greater than the first number is used as the first frame that appears in the preview interface, and the first frame that appears in the preview interface is used as the end frame. By comparing the number of lines that make up the contour, and judging whether the corresponding picture becomes blurred or becomes clear again based on whether the number of lines is lower than the second number or higher than the first number, the cleared video frame is used as the end frame without manual detection, thus realizing the automatic detection process of the end frame and improving the efficiency and accuracy of video frame detection.

[0322] Among them, the second number is a configuration parameter set by those skilled in the art based on a number threshold adopted in a large number of experiments. As long as it can be determined that the picture becomes blurred, this embodiment of the present application does not limit this.

[0323] When switching from front-facing camera to rear-facing camera or vice versa, the preview interface will switch twice. Figure 20 Taking the switching from the rear camera to the front camera as an example, the picture in the preview interface changes from clear to blurry as the rear camera is switched, and then changes from blurry to clear as the front camera is switched. Therefore, it is necessary not only to detect whether the number of lines is less than the second number to determine the process of the picture becoming blurry, but also to detect whether the number of lines is greater than the first number to determine the process of the picture becoming clear again. The embodiment of the present application detects the process of becoming blurry and clear during the switching process. After detecting that the picture has become blurry, it then detects the process of the picture becoming clear, so as to reduce the situation where the number of lines is greater than the first number due to unsuccessful switching, ensure successful switching, and improve the accuracy of video frame detection.

[0324] The video frame detection method provided in this application is applied during the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, for switching to front-facing shooting or switching to rear-facing shooting, an operation response diagram can be obtained in advance. For ease of distinction, they can be called front-facing operation response diagrams and rear-facing operation response diagrams.

[0325] For the scenario of switching from rear-facing shooting to front-facing shooting, the front-facing operation response image may include a still image and a surrounding background, which may be black or white, for example. Figure 20 As shown in Figure E in FIG. The pre-operation response image and the video frame in which the still image is detected are images from the same viewing angle, and the position of the still image in the pre-operation response image is the same as its position in the video frame. When detecting whether a still image is present, detection can be performed on the area in the video frame where the still image is located, without paying attention to other areas. Based on the positional features of the area where the still image is located in the pre-operation response image, template matching is performed in the video frame after the number of lines detected is less than the second number, the position coordinates of the still image are determined in the video frame, and the area where the still image is located in the video frame is designated as a specific area requiring attention. Then, edge feature extraction is performed on the area where the still image is located in the video frame after the number of lines detected is less than the second number to obtain line features. Based on the line features, it is determined whether the still image has become clear.

[0326] For the scene of switching from front-facing shooting to rear-facing shooting, that is, Figure 20 The post-operation response image can include a still portrait and the surrounding background, which can be black or white, such as Figure 20As shown in Figure A, the post-operation response image and the video frame in which the still portrait is detected are images from the same perspective, and the position of the still portrait in the post-operation response image is the same as its position in the video frame. When detecting whether the still portrait has become blurred, the detection can be performed on the area in the video frame where the still portrait is located, without paying attention to other areas. Based on the positional features of the area where the still portrait is located in the post-operation response image, template matching is performed in the video frame to determine the position coordinates of the still portrait in the video frame, and the area where the still portrait is located in the video frame is designated as a specific area of ​​interest. Then, edge features are extracted from the area where the still portrait is located in the video frame after the starting frame to obtain line features. Based on the line features, it is determined whether the still portrait has become blurred.

[0327] In an embodiment of the present application, by selecting the area corresponding to the still portrait in the video frame after the starting frame, and selecting the area corresponding to the still picture in the video frame after the number of lines is detected to be less than the second number, targeted area detection can be performed, thereby improving detection accuracy, reducing the edge feature extraction range, and improving detection efficiency.

[0328] In the embodiment of the present application, the duration of the camera operation response process includes the camera switching duration. After detecting the start frame and the end frame, the two are subtracted to obtain the number of captured frames, and the camera switching duration is calculated based on the number of captured frames and the camera shooting frame rate.

[0329] Scenario 5 mentioned above involves the interaction between mobile phones, high-speed cameras and computer equipment, e.g. Figure 21 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Figure 21 ,The interaction process includes the following steps.

[0330] S51. A high-speed camera shoots the screen of a mobile phone.

[0331] S52. Trigger the flip control in the mobile phone shooting interface.

[0332] S53. In response to the triggering operation, the mobile phone displays the picture in the preview interface.

[0333] S54. The high-speed camera sends the captured video to be detected to the computer device.

[0334] S55. The computer device uses a frame difference method to identify whether the flip control has started to change in the video to be detected, and uses the first frame in which the flip control starts to change as the starting frame. The computer device also identifies whether the image reappears in the preview interface, and uses the first frame in which the image reappears in the preview interface as the ending frame.

[0335] As an example, the video frame detection method performed by the computer device in scenario 5 is as follows: Figure 22 Please refer to Figure 22 , the method comprises the following steps:

[0336] S551. Obtain the video to be detected and the corresponding operation control diagram and operation response diagram.

[0337] The operation control diagram includes a flip control, such as Figure 20 The operation response diagram is an interface diagram displayed by the mobile phone in response to a trigger operation on the operation control. The operation response diagram may include a still picture (ie, a first object), such as Figure 20 As shown in Figure E.

[0338] S552. Perform region matching on the video frame in the video to be detected according to the position of the flip control in the operation control diagram to obtain a first preset region of the flip control in the video frame.

[0339] Based on the positional features of the area where the flip control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the flip control in the video frame, and the area where the flip control is located in the video frame (i.e., the first preset area) is treated as a specific area that needs attention.

[0340] S553. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0341] S554. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0342] S555. Determine whether the frame difference is greater than a preset difference value?

[0343] S556. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the flip control to start changing.

[0344] S557. Based on the position of the first object in the operation response graph, perform region matching on the video frames after the start frame in the video to be detected to obtain a second preset region of the first object in the video frame.

[0345] Based on the positional features of the area where the first object is located in the operation response graph, template matching is performed in the video frame located after the start frame to determine the position coordinates of the first object in the video frame, and the area where the first object is located in the video frame (i.e., the second preset area) is taken as a specific area that needs attention.

[0346] S558. Extract edge features from the second preset areas of the video frames after the start frame in the video to be detected, and obtain line features of each video frame after the start frame.

[0347] S559. In the process in which the number of lines indicated by the line feature of the plurality of video frames after the start frame gradually decreases, determine whether the number of lines indicated by the line feature is less than a second number?

[0348] If yes, execute S560; if no, execute S557.

[0349] S560. In the process in which the number of lines indicated by the line feature of the plurality of video frames after the start frame gradually increases, determine whether the number of lines indicated by the line feature is greater than a first number.

[0350] If yes, execute S561; if no, continue to execute S560.

[0351] S561. The first frame of the video frame in which the number of line feature indication lines is greater than the first number is used as the first frame of the picture appearing in the preview interface.

[0352] In this application Figure 21 and Figure 22 This section introduces the scenarios of switching from front-facing to rear-facing or vice versa. The implementation process and the beneficial effects achieved can be found in the above Figure 20 The description is not repeated here.

[0353] Below through Figure 23-Figure 25 Scenario 6 (from receiving a trigger operation on a video recording control to starting video recording) is described.

[0354] When using a mobile phone to record a video, the mobile phone triggers the video control (also called the video button) based on the test instruction (for example, clicks on it) to realize the automatic triggering operation of the video control. In response to the triggering operation of the video control, the mobile phone starts recording. The user-perceivable process duration refers to the duration from the change of the video control to the start of the video recording (for example, the video recording duration is a preset duration). In this process, the start of the change of the video control and the start of the video recording are key nodes. Based on this, the operation response process of the mobile phone performing the video recording can be captured by a high-speed camera to obtain the video to be tested, and the video to be tested includes multiple continuous video frames.

[0355] Take the example of using a mobile phone to record a still portrait of a person pasted on the wall. Figure 23 As shown, Figure 23 The following figure shows three representative video frames in the video to be detected corresponding to the recording scene. It is understandable that the video to be detected in this scene also includes other video frames. The recording process is as follows: Use an automated method to click the recording control on the shooting interface of the camera application, such as Figure 23 As shown in Figure A; the video control is reduced, as shown in Figure 23 As shown in Figure B; then the phone will display the video interface, such as Figure 23 As shown in Figure C.

[0356] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the recording control begins to change) and the end frame (indicating that the recording duration is the preset duration). The camera operation response process involves the identification of the recording control and the determination of the recording duration. The change of the recording control can be identified by the frame difference method. The following describes the identification of the recording control and the determination of the recording duration.

[0357] The video frame detection method provided by this application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the operation control diagram can be obtained in advance. The operation control diagram includes a video control, such as Figure 23 As shown in Figure A in the figure, the computer device pre-acquires an operation control diagram with a video control. The operation control diagram and the video frame including the video control are images from the same perspective, and the position of the video control in the operation control diagram is the same as its position in the video frame. When detecting whether the video control begins to change, the detection can be performed on the area where the video control is located in the video frame (i.e., the first preset area), without paying attention to other areas. Based on the positional features of the area where the video control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the video control are determined in the video frame, and the area where the video control is located in the video frame is treated as a specific area that needs to be paid attention to.

[0358] When the recording control of the mobile phone is clicked, the control will shrink, such as Figure 23 As shown in Figure B, Figure 23 Compared with Figure A in the image, the video control is smaller. Based on this, the present application uses the frame difference method to perform inter-frame difference on the area where the video control is located in adjacent video frames in the video to be detected, and obtains a frame difference value. When the frame difference value is greater than the preset difference value, it indicates that there is a relatively obvious difference between the adjacent video frames, that is, the video control begins to change. The first frame whose frame difference value is greater than the preset difference value is regarded as the first frame where the video control begins to change, and the first frame where the video control begins to change is regarded as the starting frame.

[0359] When using the frame difference method to perform inter-frame differencing between adjacent video frames, you can first convert the video frames to grayscale before performing inter-frame differencing. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0360] Among them, the recognition process of the video control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0361] In an embodiment of the present application, by matching the area where the recording control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy and reducing the amount of data used to determine whether the pixels have changed, thereby improving the detection efficiency.

[0362] A video frame having a recording time of a preset duration can be determined from the video frames located after the start frame in the video to be detected, and the first frame of the video frames having a recording time of the preset duration is used as the end frame.

[0363] The preset duration is a configuration parameter appropriately set by those skilled in the art according to actual conditions, for example, 00:01, 00:02, etc. Figure 23 As shown in Figure C, Figure 23 Figure C shows the interface when the recording duration is 00:01.

[0364] By determining the recording duration, the first frame of the video frame with a preset recording duration is used as the end frame. No manual detection is required, and the end frame detection process is automatically realized, thereby improving the efficiency and accuracy of video frame detection.

[0365] In this embodiment of the present application, the duration of the camera operation response process includes the video recording response duration. After detecting the start frame and the end frame, the two are subtracted to obtain the number of captured frames, and the video recording response duration is calculated based on the number of captured frames and the camera shooting frame rate.

[0366] Scenario 6 mentioned above involves the interaction between mobile phones, high-speed cameras and computer equipment, e.g. Figure 24 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Figure 24 ,The interaction process includes the following steps.

[0367] S61. A high-speed camera films the screen of a mobile phone.

[0368] S62. Trigger the video recording control in the mobile phone shooting interface.

[0369] S63. In response to the triggering operation, the mobile phone displays a video recording interface.

[0370] S64. The high-speed camera sends the captured video to be detected to the computer device.

[0371] S65. The computer device uses a frame difference method to identify whether the recording control starts to change in the video to be detected, and uses the first frame where the recording control starts to change as the start frame. The computer device uses the first frame of the video frame with a recording duration of a preset duration as the end frame.

[0372] As an example, the video frame detection method performed by the computer device in scenario 6 is as follows: Figure 25 Please refer to Figure 25 , the method comprises the following steps:

[0373] S651. Obtain the video to be detected and the corresponding operation control diagram.

[0374] The operation control diagram includes the video control, such as Figure 23 As shown in Figure A.

[0375] S652. Perform region matching on the video frame in the video to be detected based on the position of the video recording control in the operation control diagram to obtain a first preset region of the video recording control in the video frame.

[0376] Based on the positional features of the area where the video control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the video control in the video frame, and the area where the video control is located in the video frame (i.e., the first preset area) is treated as a specific area that needs attention.

[0377] S653. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0378] S654. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0379] S655. Determine whether the frame difference is greater than a preset difference value?

[0380] If yes, execute S656; if no, execute S652.

[0381] S656. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the video recording control to start changing.

[0382] S657. Determine, among the video frames after the start frame in the video to be detected, the video frames whose recording duration is the preset duration.

[0383] S658. The first frame of the video frames whose recording duration is the preset duration is used as the end frame corresponding to the camera operation response process.

[0384] In this application Figure 24 and Figure 25 This section describes the process from receiving a trigger operation on the recording control to starting recording. The implementation process and the beneficial effects achieved can be found in the above Figure 23 The description is not repeated here.

[0385] Below through Figure 26-Figure 28 Scene 7 (the scene switching from close-up shooting to long-range shooting) will be described.

[0386] The video frame detection method provided in this application is applied in the testing process of a mobile phone, and the process of switching from a close-up view to a distant view can be simulated by removing the baffle. Place the baffle in front of the rear camera of the mobile phone, and the distant view will appear zoomed after the baffle is completely removed. After the baffle is completely removed, a component will be displayed, which may be a trapezoidal component, a triangular component, a square component or a button, etc., and this embodiment of the application is not limited to this. Whether the baffle is completely removed can be detected by detecting whether the outline of the component appears. When the outline of the component is detected, it means that the baffle is completely removed. The user-perceivable process duration refers to the time from when the baffle is completely removed to when the distant view stops zooming. In this process, the outline that appears after the baffle is completely removed and the distant view stops zooming are key nodes. Based on this, the operation response process of the mobile phone performing a photo-taking operation can be filmed by a high-speed camera to obtain a video to be detected, and the video to be detected includes multiple continuous video frames.

[0387] Take the example of shooting a still portrait of a person stuck on a wall in the distance with a mobile phone. Figure 26 As shown, Figure 26 The following figure shows four representative video frames of the video to be detected corresponding to the scene of switching from close view to distant view. It is understandable that the video to be detected in this scene also includes other video frames. The process of switching from close view to distant view is as follows: a baffle is placed in front of the camera as the close view, and the image in the preview interface of the mobile phone is the baffle, such as Figure 26 As shown in Figure A; remove the baffle, as shown in Figure Figure 26 As shown in Figure B; when the baffle is completely removed, parts will appear, such as Figure 26 As shown in Figure C, the phone focuses on the distant view, and the distant view will be blurred and scaled. Figure 26 The zooming process shown in Figure C in the figure is continued until the distant view stops. Figure 26Figure D in the figure. The video frame detection method provided in this application is used to detect the video to be detected, and detect whether the baffle moves. When the movement of the baffle is detected, the key nodes of the behavior in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the baffle is completely removed) and the end frame (indicating that the picture in the distance stops zooming). The camera operation response process involves the recognition that the baffle is completely removed and the picture stops zooming. The frame difference method is used to identify whether the baffle moves. After it is identified that the baffle moves, the change in the component outline is identified to determine whether the baffle is completely removed. The recognition that the picture stops zooming can be achieved through frame difference. When the picture is zoomed, the frame difference is large, and the frame difference is small after the zoom is completed. The change pattern of the frame difference amplitude is used to identify whether the picture stops zooming. The following explains component recognition and picture stop zooming recognition.

[0388] The video frame detection method provided by this application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the baffle image can be obtained in advance, such as Figure 26 As shown in Figure A in the figure, the baffle is placed on the rear camera of the camera, and the baffle appears in the preview interface. The baffle image includes the baffle, and the baffle image and the video frame including the baffle are images from the same perspective. The position of the baffle in the baffle image is the same as its position in the video frame. When detecting whether the baffle has moved, the area where the baffle is located in the video frame (i.e., the baffle area) can be detected without paying attention to other areas. Based on the positional features of the area where the baffle is located in the baffle image, template matching is performed in the video frame, the position coordinates of the baffle are determined in the video frame, and the baffle area in the video frame is used as a specific area that needs attention.

[0389] This application uses a frame difference method to perform inter-frame differences on the baffle area of ​​adjacent video frames in the video to be detected, generating third frame difference data. The third frame difference data includes the pixel differences of multiple pixels and can reflect whether the baffle has moved. The larger the pixel difference, the greater the difference between adjacent video frames, indicating that the baffle has moved. The smaller the pixel difference, the smaller the difference between adjacent video frames, indicating that the baffle has not moved. The third frame difference data is used to determine whether the baffle has moved. If the baffle has moved, it is then determined whether it has completely moved.

[0390] When detecting whether the bezel has moved, the detection can also be performed on the area where the mobile control is located. The mobile control is used to move the bezel, and when the mobile control is clicked, the bezel begins to move. The recognition process of the mobile control is consistent with the recognition process of the camera application icon using the frame difference method described above. The implementation process and technical effects achieved can be referred to the recognition process of the camera application icon using the frame difference method described above, and will not be repeated here.

[0391] In the embodiment of the present application, the mobile phone can be operated in a relatively static state, so the component diagram can be obtained in advance. Figure 26 As shown in Figure B, the baffle moves to the left. After the baffle is completely moved away, a component appears at the back, such as Figure 26 As shown in Figure C in the figure, the component is shown as a trapezoidal component. The component diagram includes a trapezoidal component, and the component diagram and the video frame including the trapezoidal component are images under the same viewing angle, and the position of the trapezoidal component in the component diagram is the same as its position in the video frame. When detecting whether the baffle is completely removed, the area where the trapezoidal component is located in the video frame (i.e., the component area) can be detected without paying attention to other areas. Based on the positional features of the area where the trapezoidal component is located in the component diagram, template matching is performed in the video frame, the position coordinates of the trapezoidal component are determined in the video frame, and the area where the trapezoidal component is located in the video frame is treated as a specific area that needs attention.

[0392] Inter-frame difference is performed on the component area of ​​adjacent video frames in the video to be detected to obtain fourth frame difference data. The fourth frame difference data includes pixel differences of multiple pixels, and the pixel differences of multiple pixels constitute the image outline. When the outline of the trapezoidal component is detected, it indicates that the baffle is completely removed. The first frame in which the outline of the trapezoidal component appears is used as the first frame in which the baffle is completely removed, and the first frame in which the baffle is completely removed is used as the starting frame.

[0393] In an embodiment of the present application, by matching the baffle area and the component area in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy and reducing the amount of data used to determine whether the pixels have changed, thereby improving the detection efficiency.

[0394] When using the frame difference method to perform inter-frame differencing on component regions of adjacent video frames, each video frame in the video to be detected can first be grayscale converted to obtain a grayscale image corresponding to the video frame (i.e., a third grayscale image). Inter-frame differencing is then performed on the component regions of the grayscale images of the adjacent video frames to obtain fourth frame difference data. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Furthermore, grayscale images can eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0395] In the scene switching from close-up to distant view, after detecting the starting frame in the video to be detected, that is, after the baffle is completely removed, the phone focuses on the distant view, and the distant view will be blurred and scaled. The distant view will go through the following steps: Figure 26 The scaling process shown in Figure C until Figure 26Figure D in the figure. Based on this, inter-frame difference is performed on the adjacent video frames located after the starting frame in the video to be detected to obtain fifth frame difference data. The fifth frame difference data includes pixel differences of multiple pixel points, and the pixel differences of multiple pixel points constitute the picture outline. The area of ​​the picture outline reflects the difference between adjacent video frames; the larger the area of ​​the picture outline, the greater the difference between adjacent video frames, indicating that the distant view picture is being scaled; the smaller the area of ​​the picture outline, the smaller the difference between adjacent video frames, indicating that the distant view picture is scaled (that is, the distant view picture stops scaling). The video frame whose area of ​​the picture outline indicated by the fifth frame difference data is less than the third preset area is used as the first frame for which the distant view picture stops scaling, and the first frame for which the distant view picture stops scaling is used as the end frame.

[0396] Among them, the third preset area is a configuration parameter set by those skilled in the art based on an area threshold adopted in a large number of experiments. As long as it can determine whether the distant view stops zooming, the embodiment of the present application does not limit this.

[0397] By identifying the changing patterns of the frame differences between adjacent video frames and judging whether there are large-area differences in the frame differences, it is possible to determine whether the distant view has stopped zooming. Without manual detection, the end frame detection process is automatically realized, thereby improving the efficiency and accuracy of video frame detection.

[0398] The video frame detection method provided by the present application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, a distant view image can be acquired in advance. The distant view image includes a static portrait (i.e., the second object) and the surrounding background. The background can be black or white, etc. Figure 26 As shown in Figure D in . The distant view image and the video frame in which the still portrait is detected are images from the same perspective, and the position of the still portrait in the distant view image is the same as its position in the video frame. When detecting whether the still portrait stops zooming, the detection can be performed on the area where the still portrait is located in the video frame (i.e., the fourth preset area), without paying attention to other areas. According to the positional features of the area where the still portrait is located in the distant view image, template matching is performed in the video frame, the position coordinates of the still portrait are determined in the video frame, and the area where the still portrait is located in the video frame is treated as a specific area that needs attention. Then, inter-frame difference is performed on the area where the still portrait is located in the adjacent video frames after the starting frame in the video to be detected, and the fifth frame difference data is obtained. The video frame whose area of ​​the picture outline indicated by the fifth frame difference data is smaller than the third preset area is used as the first frame where the distant view picture stops zooming.

[0399] In an embodiment of the present application, by selecting the area corresponding to the still portrait in the video frame, targeted area detection can be performed, thereby improving detection accuracy, reducing the amount of data required to determine whether a pixel has changed, and improving detection efficiency.

[0400] When using the frame difference method to perform inter-frame differencing on adjacent video frames, for each video frame after the starting frame in the video to be detected, the video frame can first be grayscale converted to obtain the corresponding grayscale image of the video frame (i.e., the fourth grayscale image). Inter-frame differencing is then performed on the region containing the still person in the grayscale images of the adjacent video frames to obtain the fifth frame difference data. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0401] In this embodiment of the present application, the duration of the camera operation response process includes the near-to-far response duration. After detecting the start frame and the end frame, the two are subtracted to obtain the number of captured frames, and the near-to-far response duration is calculated based on the number of captured frames and the camera shooting frame rate.

[0402] The above scenario 7 involves the interaction between mobile phones, high-speed cameras and computer equipment, such as Figure 27 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Figure 27 ,The interaction process includes the following steps.

[0403] S71. Use a high-speed camera to shoot the screen of a mobile phone.

[0404] S72. Remove the barrier for the close-up view.

[0405] S73. Display the image on the preview interface of the mobile phone.

[0406] S74. The high-speed camera sends the captured video to be detected to the computer device.

[0407] S75. The computer device uses a frame difference method to identify whether the baffle is completely removed in the video to be detected, and uses the first frame in which the baffle is completely removed as the starting frame. The computer device also uses the frame difference method to identify whether the image in the preview interface is zoomed, and uses the first frame in which the image in the preview interface stops zooming as the ending frame.

[0408] As an example, the video frame detection method performed by the computer device in scene 7 is as follows: Figure 28 Please refer to Figure 28 , the method comprises the following steps:

[0409] S751. Obtain the video to be detected and the corresponding baffle image, component image, and perspective image.

[0410] The baffle diagram includes baffles, e.g. Figure 26As shown in Figure A in . The component diagram includes a trapezoidal component (i.e., the target component), as shown in Figure 26 As shown in Figure C in FIG. The distant view includes a still portrait (ie, the second object), such as Figure 26 As shown in Figure D.

[0411] S752. Perform region matching on the video frame in the video to be detected according to the position of the baffle in the baffle map to obtain the baffle region in the video frame.

[0412] Based on the position features of the baffle area in the baffle map, template matching is performed in the video frame to determine the position coordinates of the baffle in the video frame, and the area where the baffle is located in the video frame (i.e., the baffle area) is regarded as a specific area that needs attention.

[0413] S753. Perform inter-frame difference on the baffle area of ​​adjacent video frames in the video to be detected to obtain third frame difference data.

[0414] S754. When the third frame difference data indicates that the baffle has moved, perform area matching on the video frame in the video to be detected according to the position of the target component in the component map to obtain the component area of ​​the target component in the video frame, and perform grayscale conversion on the video frame to obtain the third grayscale map corresponding to the video frame.

[0415] Based on the positional features of the area where the target component is located in the component map, template matching is performed in the video frame to determine the position coordinates of the target component in the video frame, and the area where the target component is located in the video frame (i.e., the component area) is regarded as a specific area that needs attention.

[0416] S755. Perform inter-frame difference on the component areas of the third grayscale image corresponding to adjacent video frames to obtain fourth frame difference data.

[0417] S756. The first frame in which the fourth frame difference data indicates the appearance of the outline of the target component is used as the first frame in which the baffle is completely removed. The target component is the component displayed after the baffle is completely removed.

[0418] S757. Based on the position of the second object in the distant view, perform region matching on the video frames after the start frame in the video to be detected to obtain a fourth preset region of the second object in the video frames.

[0419] Based on the position features of the area where the second object is located in the distant view, template matching is performed in the video frame to determine the position coordinates of the second object in the video frame, and the area where the second object is located in the video frame (i.e., the fourth preset area) is regarded as a specific area that needs attention.

[0420] S758. Perform grayscale conversion on the video frames after the start frame in the video to be detected to obtain a fourth grayscale image corresponding to the video frame.

[0421] S759. Perform inter-frame difference on the fourth preset area of ​​the fourth grayscale image corresponding to adjacent video frames to obtain fifth frame difference data.

[0422] S760. Determine whether the area of ​​the picture outline indicated by the fifth frame difference data is smaller than a third preset area.

[0423] If yes, execute S761; if no, execute S757.

[0424] S761. Use the video frame whose fifth frame difference data indicates that the area of ​​the picture outline is smaller than the third preset area as the first frame for stopping zooming of the distant picture.

[0425] In this application Figure 27 and Figure 28 Introducing the scene of switching from close-up shooting to long-range shooting. The implementation process and the beneficial effects achieved can be found in the above Figure 26 The description is not repeated here.

[0426] Below through Figures 29-35 Scene 8 (the scene from receiving a trigger operation on the photo control to completing the shooting of the dynamic timer) is described.

[0427] By testing the video to be tested from the time the phone receives the trigger operation of the photo control to the time the dynamic timer completes the shooting of the scene, the phone's photo delay can be detected. In the actual test process, the test of the photo delay scene is achieved by detecting the time corresponding to the starting frame and the time corresponding to the captured picture. The dynamic timer is a timing tool that turns on and off lights regularly according to a time period. The dynamic timer includes a dynamically displayed light array, which includes multiple lights arranged in rows and columns. The dynamic timer controls different lights to emit light in a cycle according to a preset frequency. The dynamic timer can also be called a carousel. The display time corresponding to the dynamic timer can be obtained by identifying the carousel time.

[0428] Each light in the dynamic display light array can be a light emitting diode (LED), etc. The light array can be timed by the position of the dynamically displayed light. The light array can have different refresh frequencies, and the display area of ​​the light array can have different light number specifications. The embodiment of the application does not limit the specifications and operating frequency of the light array. Figure 29 Introduce one of the light arrays. Figure 29A schematic diagram of the dynamic display of the light array in the dynamic timer is shown. The dynamic timer includes a plurality of lights displayed in rows and columns. The dynamic timer can control the lights in different positions to dynamically illuminate according to a preset frequency. The lighting of lights in different rows and columns can correspond to different times. The dynamic timer is divided into two display areas, the upper display area includes 100 lights, and the lower display area includes 10 lights. In the upper display area, the lighting display is cyclically displayed in units of 5 consecutive lights; in the lower display area, the lighting display is cyclically displayed in units of 1 light. The 5 illuminated lights are cyclically displayed in the upper display area, corresponding to the 1 illuminated light in the lower display area moving one position to the right. At the same moment, there are 5 illuminated lights in the upper display area and 1 illuminated light in the lower display area. The current moment can be determined based on the row and column positions of the 5 illuminated lights in the upper display area and the position of the 1 illuminated light in the lower display area. Figure 29 Take the dynamic display in Figure A as an example, the first display moment is as follows Figure 29 As shown in Figure A, the lights in row number 90, column number 3 to row number 90, column number 7 in the upper display area light up, and the light in the lower display area light up, and the other lights do not light up. The time corresponding to the lights that light up is 2×100+90+3; the second display time cycle is to Figure 29 In Figure B, the lights in row number 90, column number 8 to row number 90, column number 9 in the upper display area are lit, as well as the lights in row number 0, column number 0 to row number 0, column number 2 are lit. The light in the lower display area is number 2 and the other lights are not lit. The time corresponding to the lights that are lit is 2×100+90+8. The third display time cycle is to Figure 29 In Figure C, the lights in the upper display area with row number 0, column number 3 to row number 0, column number 7 are lit, and the light with number 3 in the lower display area is lit, and the other lights are not lit. The time corresponding to the lights that are lit is 3×100+0+3; the fourth display time cycle is to Figure 29 In Figure D, the lights in row number 0, column number 8 to row number 0, column number 9 in the upper display area are lit, as are the lights in row number 1, column number 0 to row number 1, column number 2. The light in the lower display area number 3 is lit, and the other lights are not lit. The time corresponding to the lit lights is 3×100+0+8.

[0429] Camera photo delay is the time difference between pressing the shutter button and the camera actually capturing the image. To test the camera's photo delay, the phone triggers the photo control based on a test command, automatically triggering the photo control. In response to the triggering of the photo control, the phone captures the captured image. The time corresponding to the start of the photo control change and the corresponding time in the captured image are key points in this process. Based on this, a dynamic timer can be set in the shooting scene. The phone can then take a photo of the dynamic timer to simulate a photo delay scenario, with the image captured by the phone including the dynamic timer.

[0430] Obtain the time corresponding to the dynamic timer when the photo control begins to change and the time corresponding to the dynamic timer in the captured image, and subtract the two to obtain the photo delay duration. A high-speed camera can be used to capture the mobile phone's response to the dynamic timer to obtain a video to be tested, which includes multiple consecutive video frames.

[0431] For example, Figure 30 As shown, Figure 30 The following figure shows two representative video frames in the video to be detected corresponding to the scene of taking a photo of the dynamic timer. It is understandable that the video to be detected in this scene also includes other video frames. The process of triggering the photo control to take a photo of the dynamic timer is as follows: Use an automated method to click the photo control on the shooting interface of the camera application, such as Figure 30 As shown in Figure A; the photo control is reduced, as shown in Figure 30 As shown in Figure B; the phone will then take a picture, such as Figure 30 Figure C in .

[0432] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the photo control starts to change). Then the time of the dynamic timer corresponding to the start frame is determined, and the light of the captured picture is identified to determine the time of the dynamic timer in the captured picture. The camera operation response process involves the recognition of the photo control and the time recognition of the light in the video frame and the captured picture. The change of the photo control can be identified by the frame difference method. The following describes the recognition of the photo control and the recognition of the revolving lantern time.

[0433] The video frame detection method provided by this application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the operation control diagram can be obtained in advance. The operation control diagram includes a photo taking control, such as Figure 30As shown in Figure A in . The operation control diagram and the video frame including the photo control are images from the same perspective, and the position of the photo control in the operation control diagram is the same as its position in the video frame. When detecting whether the photo control starts to change, the detection can be performed on the area where the photo control is located in the video frame (i.e., the first preset area), without paying attention to other areas. Based on the positional features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the photo control are determined in the video frame, and the area where the photo control is located in the video frame is treated as a specific area that needs to be paid attention to.

[0434] When the camera control of the mobile phone is clicked, the control will shrink, such as Figure 30 As shown in Figure B, Figure 30 Compared to Figure A in the video, the photo control is smaller. Based on this, the present application uses a frame difference method to perform inter-frame differences on the area containing the photo control in adjacent video frames in the video to be tested, obtaining a frame difference value. When the frame difference value is greater than a preset difference value, it indicates that a significant difference has occurred between the adjacent video frames, indicating that the photo control has begun to change. The first frame with a frame difference greater than the preset difference value is regarded as the first frame where the photo control begins to change, and the first frame where the photo control begins to change is regarded as the starting frame.

[0435] When using the frame difference method to perform inter-frame differencing between adjacent video frames, you can first convert the video frames to grayscale before performing inter-frame differencing. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0436] Among them, the recognition process of the photo control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0437] In an embodiment of the present application, by matching the area where the photo control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy and reducing the amount of data used to determine whether the pixels have changed, thereby improving the detection efficiency.

[0438] There are two types of time recognition for a revolving screen: one is based on the first frame (i.e., the starting frame) when the camera control begins to change; the other is based on a high-definition image (e.g., a captured image). The starting frame is a video frame, which has a lower resolution than the captured image. Therefore, different methods are required for each type of time recognition.

[0439] The first method, for video frames, has low resolution and low brightness, the outline of the lights that are not emitting light is not clear, and there are interferences such as focus lines on the dynamic timer, such as Figure 31 As shown, the focus line is a shooting auxiliary line used to help determine the horizontal degree of shooting. Figure 31 It is shown in the figure that when shooting a dynamic timer, shooting auxiliary lines such as horizontal lines, grid lines, etc. will appear in the preview interface. Figure 31 The shooting auxiliary lines shown in the will be superimposed on the preview interface. To facilitate viewing the attached image, Figure 30 Figure A in the figure does not show the shooting auxiliary lines, which will interfere with the recognition process of the lights. Therefore, non-luminous lights are difficult to identify, while luminous lights are easy to identify. Based on this, the algorithm for identifying the time of the revolving lantern in the video frame is as follows: first read the entire video frame of the video to be detected, save the video frame where the photo control begins to change (i.e., the starting frame), and identify the position of the luminous lamp in each video frame (i.e., the outline of the lit lamp). Then all the identified positions of the lit lights are filtered and deduplicated to obtain the positions of all the lights, and the revolving lantern position marking is completed (i.e., the first coordinate matrix). The revolving lantern position marking is as follows: Figure 32 As shown, Figure 32 This is just an example of the outline of the range of a revolving light. By identifying the position of the lights in each video frame and combining the identified positions of the lights, we can get the following: Figure 32 The outline of the area where the revolving lantern is located is shown.

[0440] Then identify the light position in the saved starting frame and get the light position in the starting frame, such as Figure 33 The schematic diagram of the position of the light-emitting lamp is shown, wherein, Figure 33 Other places with bright lights are also shown in the figure. In practical applications, we can only focus on the lights that are lit. Match the light position in the starting frame with the position mark of the revolving lantern above to get the row and column where the light position in the starting frame is located, that is, Figure 33 The light position in the starting frame is placed at Figure 32 The row and column of the light position in the starting frame are obtained. The time is determined according to the row and column of the light position in the starting frame, that is, the time of the dynamic timer corresponding to the starting frame is obtained.

[0441] The second method is to take pictures, such as Figure 30 As shown in Figure C, the resolution is high, the outlines of the lights are clear, and there are no interference factors. The position of the unlit lights can be directly identified and the position is marked. Then, the position of the lit lights is identified and position matching is completed to obtain the row and column where the lit lights are located. Figure 30 The time corresponding to the light shown in Figure C is 5×100+90+3.

[0442] In an embodiment of the present application, the first method is used to identify the time of the dynamic timer in the video frame. The dynamic timer controls the light to display in a cycle according to a preset frequency, and the outline of the lighted light can be detected by color recognition. Each video frame in the video to be detected is traversed, and color recognition is performed on each video frame to obtain the outline of the lighted light in each video frame; the outlines of the lighted lights in multiple video frames of the video to be detected are deduplicated and superimposed together to construct the outlines of all the lights in the dynamic timer and obtain the first coordinate matrix of the dynamic timer. The starting frame is then color recognized to obtain the outline of the lighted light in the starting frame; the position of the outline in the starting frame is matched with the first coordinate matrix to obtain the position of the outline in the first coordinate matrix, and then the time of the dynamic timer corresponding to the starting frame is determined.

[0443] In this embodiment, a second method is used to identify the time of a dynamic timer in a captured image. Shape recognition is performed on the captured image based on a preset shape to determine the positions of all unlit lamps in the image. A second coordinate matrix is ​​then constructed based on the positions of all unlit lamps. Color recognition is then performed on the captured image to determine the outlines of the lit lamps in the image. The outlines of the lit lamps are then matched against the second coordinate matrix to determine their positions within the second coordinate matrix. The time of the dynamic timer in the image is then determined.

[0444] Among them, the preset shape is a configuration parameter set by technical personnel in this field based on the physical shape of the lamp used in a large number of experiments, such as triangle, square, rectangle, circle, ellipse, etc. The lamp can only be identified based on the shape, and this embodiment of the application does not limit this.

[0445] In the embodiment of the present application, different methods are used to identify the state (glowing or not) and time of the lamp for low-resolution video frames and high-definition pictures taken, respectively. For low-resolution video frames, the entire video to be detected is traversed to identify the glowing lamp in each video frame, thereby constructing a coordinate matrix of a dynamic timer, and then the contour of the glowing lamp in the starting frame is identified, and template matching is performed with the coordinate matrix to obtain the row and column where the glowing lamp in the starting frame is located, thereby obtaining the time corresponding to the starting frame. For the high-definition pictures taken, the position of the non-glowing lamp is directly identified, and a coordinate matrix is ​​constructed. The position of the glowing lamp is then identified, and template matching is performed with the coordinate matrix to obtain the row and column where the glowing lamp in the picture is located, thereby obtaining the time corresponding to the picture taken. By identifying the digitized video, the row and column where the lamp in the picture and the starting frame is located is determined, and automatic reading is achieved. No manual detection is required, and the automatic detection process of the end frame is realized, thereby improving the efficiency and accuracy of video frame detection.

[0446] The above scenario 8 involves the interaction between mobile phones, high-speed cameras and computer equipment, such as Figure 34 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Figure 34 ,The interaction process includes the following steps.

[0447] S81. A high-speed camera films the screen of a mobile phone.

[0448] S82. Trigger the photo control in the mobile phone shooting interface.

[0449] S83. The mobile phone takes a picture in response to the triggering operation.

[0450] S84. The high-speed camera sends the captured video to be detected to the computer device.

[0451] S85. The mobile phone sends the captured image to the computer device.

[0452] S86. The computer device uses a frame difference method to identify whether the camera control has started to change in the video to be detected, and uses the first frame in which the camera control starts to change as the starting frame. The computer device also identifies the time of the carousel in the starting frame and the time of the carousel in the captured image.

[0453] As an example, the video frame detection method performed by the computer device in scene 8 is as follows: Figure 35 Please refer to Figure 35 , the method comprises the following steps:

[0454] S861. Obtain the video to be detected and the corresponding operation control diagram.

[0455] The operation control diagram includes a photo taking control, such as Figure 30 As shown in Figure A.

[0456] S862. Based on the position of the photo control in the operation control diagram, perform region matching on the video frame in the video to be detected to obtain a first preset region of the photo control in the video frame.

[0457] Based on the positional features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the photo control in the video frame, and the area where the photo control is located in the video frame (i.e., the first preset area) is treated as a specific area that needs attention.

[0458] S863. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0459] S864. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0460] S865. Determine whether the frame difference is greater than a preset difference value?

[0461] If yes, execute S866; if no, execute S862.

[0462] S866. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the camera control to start changing.

[0463] S867. Perform color recognition on each video frame in the video to be detected to obtain the outline of the light in each video frame.

[0464] S868. Construct a first coordinate matrix of the dynamic timer based on the outline of the glowing lights in each video frame.

[0465] S869. Perform color recognition on the first frame in which the photo control starts to change to obtain a first outline; the first outline is the outline of the light in the first frame in which the photo control starts to change.

[0466] S870. Perform position matching in the first coordinate matrix based on the position of the first contour in the first frame in which the photo-taking control starts to change, and obtain the position of the first contour in the first coordinate matrix.

[0467] S871. Determine, based on the position of the first outline in the first coordinate matrix, the first time indicated by the dynamic timer in the first frame in which the photographing control begins to change.

[0468] S872. Perform shape recognition on the captured image according to a preset shape to obtain a second coordinate matrix of the dynamic timer; the preset shape is the physical shape of the lamp, and the second coordinate matrix indicates the position of the unlit lamp.

[0469] S873. Perform color recognition on the captured image to obtain a second outline; the second outline is the outline of the illuminated lamp in the captured image.

[0470] S874. Perform position matching in the second coordinate matrix based on the position of the second contour in the captured image to obtain the position of the second contour in the second coordinate matrix.

[0471] S875. Determine the second time indicated by the dynamic timer in the captured image based on the position of the second contour in the second coordinate matrix.

[0472] In this application Figure 34 and Figure 35Introducing the scene from receiving the trigger operation of the photo control to completing the dynamic timer shooting (shutterlag), which can include high light and low light. Its implementation process and the beneficial effects achieved can be seen in the above Figures 29-33 The description is not repeated here.

[0473] Below through Figure 36-Figure 39 Scenario 9 (a scenario in which a process is executed in response to a triggering operation on a photographing control) is described.

[0474] The camera shooting rate refers to the number of frames a camera can shoot per second. The camera shooting rate scenario can be simulated by observing the changes in the screen display in the preview interface and the changes in the dynamic timer display during the mobile phone's dynamic timer shooting process. In the actual test process, the shooting rate scenario is tested by detecting the start frame and the end frame. When detecting the camera shooting rate, the mobile phone triggers the photo control based on the test instruction to realize the automatic triggering operation of the photo control. In response to the triggering operation of the photo control, the screen display will change from black to bright (or from black to bright) and the dynamic timer display will freeze in the preview interface of the mobile phone. The order of the screen display changing from black to bright and the dynamic timer display freezing is uncertain. Therefore, it is necessary to identify the screen display changing from black to bright and the dynamic timer display freezing. In this process, the key nodes are the start of the change in the photo control, the screen changing from black to bright and the dynamic timer display freezing. The dynamic timer includes a dynamically displayed light array, which includes a plurality of lights arranged in rows and columns. For an explanation of the dynamic timer, please refer to the above Figure 29 The description in will not be repeated here.

[0475] In the embodiment of the present application, the first frame where the photo control begins to change is used as the starting frame, and the video frame sorted after the first frame where the screen changes from black to bright and the last frame where the dynamic timer displays a freeze is used as the ending frame. The camera shooting rate can be determined based on the starting frame and the ending frame. The present application can use a high-speed camera to capture the mobile phone's response process of executing the photo operation of the dynamic timer to obtain a video to be detected, and the video to be detected includes multiple consecutive video frames.

[0476] For example, Figure 36 As shown, Figure 36 The following figure shows four representative video frames in the video to be detected corresponding to the scene of taking a photo of the dynamic timer. It is understandable that the video to be detected in this scene also includes other video frames. The process of triggering the photo control to take a photo of the dynamic timer is as follows: Use an automated method to click the photo control on the shooting interface of the camera application, such as Figure 36 As shown in Figure A; the photo control is reduced, as shown in Figure 36 As shown in Figure B, then Figure 36As shown in Figure C, the camera controls return to normal; next, a black frame and a frame that changes from black to light will appear on the preview interface of the phone. The preview interface will go through the following steps: Figure 36 The blackening process shown in Figure C, and the Figure 36 The brightening process is shown in Figure D. The lights displayed dynamically in the preview interface may appear to be stuck, such as Figure 37 As shown, Figure 37 Four consecutive video frames are shown, Figure 37 Figure A in Figure 37 Figure B and Figure 37 The video frame shown in Figure C shows a flickering light until Figure 37 As shown in Figure D.

[0477] The video frame detection method provided in this application is used to detect the video to be detected, and the key nodes where the behavior occurs in the camera operation response process are identified. The key nodes include the start frame (the first frame indicating that the photo control starts to change) and the end frame (the video frame sorted after the first frame indicating that the screen changes from black to bright and the last frame where the dynamic timer displays a freeze). The camera operation response process involves the recognition of photo controls, the recognition of the screen changing from black to bright, and the recognition of the dynamic timer display freezing. The changes in the photo controls can be identified by the frame difference method. The process of the screen changing from black to bright is identified by brightness comparison. The following describes the recognition of photo controls, the recognition of the screen changing from black to bright, and the recognition of the dynamic timer display freezing.

[0478] The video frame detection method provided by this application is applied in the testing process of a mobile phone. The mobile phone can be operated in a relatively static state. Therefore, the operation control diagram can be obtained in advance. The operation control diagram includes a photo taking control, such as Figure 36 As shown in Figure A in the figure, the computer device pre-acquires an operation control diagram with a photo control. The operation control diagram and the video frame including the photo control are images from the same perspective, and the position of the photo control in the operation control diagram is the same as its position in the video frame. When detecting whether the photo control begins to change, the detection can be performed on the area where the photo control is located in the video frame (i.e., the first preset area), without paying attention to other areas. Based on the positional features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame, the position coordinates of the photo control are determined in the video frame, and the area where the photo control is located in the video frame is treated as a specific area that needs to be paid attention to.

[0479] When the camera control of the mobile phone is clicked, the control will shrink first and then return to normal. Figure 36 As shown in Figure B, Figure 36 Compared with Figure A in the figure, the camera control is reduced, and then Figure 36As shown in Figure C, the camera control has returned to normal. Based on this, the present application uses a frame difference method to perform inter-frame differences on the area containing the camera control in adjacent video frames in the video to be tested, obtaining a frame difference value. When the frame difference value is greater than a preset difference value, it indicates that a significant difference has occurred between the adjacent video frames, indicating that the camera control has begun to change. The first frame with a frame difference greater than the preset difference value is regarded as the first frame where the camera control begins to change, and the first frame where the camera control begins to change is regarded as the starting frame.

[0480] When using the frame difference method to perform inter-frame differencing between adjacent video frames, you can first convert the video frames to grayscale before performing inter-frame differencing. Using grayscale images for inter-frame differencing can reduce the amount of computation and increase data processing speed. Grayscale images can also eliminate interference caused by color differences and reduce the interference of video flicker on detection, thereby improving the accuracy of data processing results.

[0481] Among them, the recognition process of the photo control is consistent with the above-mentioned recognition process of the camera application icon using the frame difference method. Its implementation process and the technical effect achieved can be referred to the above-mentioned recognition process of the camera application icon using the frame difference method, and will not be repeated here.

[0482] In an embodiment of the present application, by matching the area where the photo control is located in the video frame, targeted area detection can be performed when using the frame difference method to detect whether there is a large difference in the pixels of adjacent video frames, thereby improving the detection accuracy and reducing the amount of data used to determine whether the pixels have changed, thereby improving the detection efficiency.

[0483] In the scene of the camera taking pictures of the dynamic timer, after detecting the starting frame of the video to be detected, the computer device will then detect the picture on the preview interface of the mobile phone, which includes the dynamic display light. The preview interface will have black frames and frames that change from black to bright. The preview interface will go through the following steps: Figure 36 The blackening process shown in Figure C, and the Figure 36 The brightening process is shown in Figure D. The lights displayed dynamically in the preview interface may appear to be stuck, such as Figure 37 As shown, Figure 37 Four consecutive video frames are shown, Figure 37 Figure A in Figure 37 Figure B and Figure 37 The video frame shown in Figure C shows a flickering light until Figure 37 As shown in Figure D. Compared with Figure 37 Figure C in Figure 37 The D diagram in the figure shows that the position of the light changes from constant to variable. Figure 37 Figure C in the figure shows the last frame of the dynamic timer display freeze.

[0484] It should be noted that Figure 37 is located in Figure 36 The video frame after the B picture in Figure 36 After the C figure in the figure, it can also be Figure 36 That is, the video frame where the screen changes from black to bright can be before or after the video frame where the dynamic timer shows a freeze.

[0485] The process of brightening the preview screen is a gradual process, brightening step by step. The change in brightness from black to bright can be used to identify the change from black to bright. The following two methods can be used to identify the change from black to bright. The first method is to pre-set a black threshold (i.e., the first threshold). If the brightness of a video frame is less than the black threshold, the video frame is judged to be black. At the same time, a black-to-bright threshold (i.e., the second threshold) is pre-set. When the current video frame is judged to be black, if the brightness difference obtained by subtracting the brightness of the previous video frame from the brightness of the current video frame is greater than the black-to-bright threshold, the current video frame is regarded as the first frame to change from black to bright. The second method is to count the brightness of multiple video frames in the video to be detected, average the brightness of the multiple video frames, and determine the brightness threshold based on the average value. For example, the average value is used as the brightness threshold, or the brightness threshold is obtained by multiplying the average value by a coefficient. As the brightness of multiple video frames after the starting frame gradually increases, the video frame with a brightness greater than the brightness threshold is regarded as the first frame to change from black to bright. This method can adapt to different lighting conditions, does not rely on a fixed threshold, and increases the applicable scenarios of the detection method.

[0486] Among them, the first threshold and the second threshold are configuration parameters set by technical personnel in this field based on the black threshold and the black-to-light threshold adopted in a large number of experimental processes. As long as it can be determined whether the video frame is a black video frame and whether the video frame is a black-to-light video frame, the embodiments of the present application are not limited to this.

[0487] In the embodiments of the present application, a black threshold is set to determine the brightness of a video frame, and a black-to-light threshold is set to determine the brightness difference between adjacent video frames. This allows the detection of video frames that transition from black to light in multiple video frames following the starting frame. This eliminates the need for manual detection, achieving automatic detection of video frames that transition from black to light, thereby improving both the efficiency and accuracy of video frame detection.

[0488] Furthermore, a brightness threshold can be set according to the brightness of multiple video frames, and the brightness threshold is related to the brightness of multiple video frames. For example, a video frame that changes from black to bright can be detected among multiple video frames after the starting frame, thereby increasing the diversity of implementation methods of the judgment process.

[0489] The dynamic timer controls the light to display in a cyclic manner at a preset frequency, and the outline of the lighted light can be detected by color recognition. Color recognition is performed on each video frame after the starting frame to obtain the outline of the lighted light in each video frame (i.e., the third outline). When consecutive video frames with the same outline are detected, it indicates that the dynamic timer display is stuck, and the detection is continued until a video frame with an outline different from the consecutive video frames is detected, and multiple video frames with the same outline are obtained. The video frames with the same outline are used as target video frames, and multiple target video frames are obtained. The outline of the next video frame of the last frame in the multiple target video frames is different from the outline of the multiple target video frames. The last frame in the multiple target video frames is used as the last frame of the dynamic timer display stuck.

[0490] In this embodiment, the outline of a light in a video frame is identified, and then multiple target video frames with the same outline are determined. The last frame in these target video frames is used as the last frame where the dynamic timer display freezes. This eliminates the need for manual detection and automatically detects video frames where the dynamic timer display freezes, improving both the efficiency and accuracy of video frame detection.

[0491] After detecting the first frame where the screen changes from black to bright and the last frame where the dynamic timer displays a freeze, the video frame sorted after the first frame where the screen changes from black to bright and the last frame where the dynamic timer displays a freeze is taken as the end frame. No manual detection is required, thus realizing the automatic detection process of the end frame and improving the efficiency and accuracy of video frame detection.

[0492] In the embodiment of the present application, after the start frame and the end frame are detected, the number of shooting frames is obtained by subtracting the two, and the camera shooting rate is determined according to the number of shooting frames.

[0493] The above scenario 9 involves the interaction between mobile phones, high-speed cameras and computer equipment, such as Figure 38 As shown, the camera operation is triggered on the screen of the mobile phone, and the mobile phone executes the camera operation response process and displays the corresponding interface on the screen. The high-speed camera shoots the screen of the mobile phone to obtain the video to be detected, and the video frame detection method provided in the embodiment of the application is used to detect the video to be detected to obtain the start frame and the end frame. Please refer to Figure 38 ,The interaction process includes the following steps.

[0494] S91. A high-speed camera films the screen of a mobile phone.

[0495] S92. Trigger the photo control in the mobile phone shooting interface.

[0496] S93. In response to the triggering operation, the mobile phone displays the picture in the preview interface.

[0497] S94. The high-speed camera sends the captured video to be detected to the computer device.

[0498] S95. The computer device uses a frame difference method to identify whether the camera control begins to change in the video to be tested, and uses the first frame where the camera control begins to change as the starting frame. The computer device also identifies the first frame where the screen changes from black to bright in the preview interface and the last frame where the dynamic timer freezes, and uses the video frame that is sorted later as the ending frame.

[0499] As an example, the video frame detection method performed by the computer device in scene 9 is as follows: Figure 39 Please refer to Figure 39 , the method comprises the following steps:

[0500] S951. Obtain the video to be detected and the corresponding operation control diagram.

[0501] The operation control diagram includes a photo taking control, such as Figure 36 As shown in Figure A.

[0502] S952. Perform region matching on the video frame in the video to be detected based on the position of the photo control in the operation control diagram to obtain a first preset region of the photo control in the video frame.

[0503] Based on the positional features of the area where the photo control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the photo control in the video frame, and the area where the photo control is located in the video frame (i.e., the first preset area) is treated as a specific area that needs attention.

[0504] S953. Perform grayscale transformation on the video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame.

[0505] S954. Perform inter-frame difference on the first preset area of ​​the first grayscale image corresponding to adjacent video frames to obtain a frame difference value.

[0506] S955. Determine whether the frame difference is greater than a preset difference value?

[0507] If yes, execute S956; if no, execute S952.

[0508] S956. The first frame whose frame difference is greater than the preset difference value is used as the first frame for the camera control to start changing.

[0509] S957. Detect the video frame located after the starting frame in the video to be detected. When the brightness of the current video frame is less than a first threshold value and the brightness difference between the current video frame and the previous video frame of the current video frame is greater than a second threshold value, use the current video frame as the first frame where the picture changes from black to bright; or, determine the brightness threshold value based on the brightness of each video frame in the video to be detected; in the process where the brightness of multiple video frames located after the starting frame gradually increases, use the video frame whose brightness is greater than the brightness threshold value as the first frame where the picture changes from black to bright.

[0510] S958. Perform color recognition on each video frame after the start frame in the video to be detected to obtain a third outline of each video frame; the third outline is the outline of the light in the video frame after the start frame.

[0511] S959. Determine multiple target video frames in each video frame located after the starting frame; wherein the third contours of the multiple target video frames are the same, and the third contour of the next video frame of the last frame in the multiple target video frames is different from the third contours of the multiple target video frames.

[0512] S960. Use the last frame among the multiple target video frames as the last frame of the dynamic timer display freeze.

[0513] S961. The video frame that is sorted after the first frame where the screen changes from black to bright and the last frame where the dynamic timer displays a freeze is used as the end frame corresponding to the camera operation response process.

[0514] In this application Figure 38 and Figure 39 Introducing the scene of executing the process (shot2preview) in response to the trigger operation of the photo control, which can include high light and low light. Its implementation process and the beneficial effects achieved can be seen in the above Figure 36-Figure 37 The description is not repeated here.

[0515] Figure 40 An exemplary flowchart of a video frame detection method provided in an embodiment of the present application.

[0516] Please refer to Figure 40 , Figure 40 This is a flow chart of a video frame detection method provided by an embodiment of the present application. Figure 40 As shown, the method includes the following steps:

[0517] S111. Obtain a video to be detected and a corresponding operation control diagram. The video to be detected is obtained by shooting the screen of a target terminal that executes a camera operation response process. The operation controls in the operation control diagram are used to start the camera operation response process.

[0518] In a possible implementation, the operation controls include a camera application icon, a photo control, an image echo control, a return control, a flip control, and a video control. Figure 4 Figure A in the above Figure 9 Figure A in the above Figure 13 Figure A in the above Figure 17 Figure A in the above Figure 20 Figure A and the above Figure 23 Figure A in .

[0519] S112. Perform region matching on the video frame in the video to be detected according to the position of the operation control in the operation control map to obtain a first preset region of the operation control in the video frame.

[0520] In one possible implementation, the operation control diagram and the video frame including the operation control are images taken from the same perspective, and the position of the operation control in the operation control diagram is the same as its position in the video frame. Based on the positional features of the area where the operation control is located in the operation control diagram, template matching is performed in the video frame to determine the position coordinates of the operation control in the video frame, and the area where the operation control is located in the video frame is treated as a specific area of ​​interest (i.e., a first preset area). When detecting whether the operation control has begun to change, the detection can be performed on the area in the video frame where the operation control is located (i.e., the first preset area), without paying attention to other areas.

[0521] The first preset area can refer to the above Figure 4 The area where the camera application icon is located in Figure A, the above Figure 9 The area where the photo control is located in Figure A, the above Figure 13 The area where the image echo control is located in Figure A, the above Figure 17 Returns the area where the control is located in Figure A, the above Figure 20 The area where the flip control is located in Figure A and the above Figure 23 The area where the video control is located in Figure A.

[0522] S113. Detect pixel changes in a first preset area of ​​adjacent video frames in the video to be detected, and obtain the first frame where the operation control starts to change. The first frame where the operation control starts to change is the starting frame corresponding to the camera operation response process.

[0523] In one possible implementation, a frame difference method is used to perform inter-frame difference on a first preset area of ​​adjacent video frames in the video to be detected to obtain a frame difference value. The frame difference value reflects the pixel change in the first preset area, thereby detecting the video frame where the operation control begins to change, and the first frame where the operation control begins to change is used as the starting frame corresponding to the camera operation response process.

[0524] S114. Detect pixel changes in the video frame after the start frame or in adjacent video frames in the video to be detected to obtain a response end video frame, where the response end video frame is the end frame corresponding to the camera operation response process.

[0525] In one possible implementation, after detecting the start frame, pixel changes in frames following the start frame in the video being tested are detected to determine whether a scene has appeared. Alternatively, pixel changes in adjacent frames following the start frame in the video being tested are detected to determine whether differences exist between adjacent frames and whether the image has appeared completely. For different detection scenarios, a detection method that matches the scenario can be selected to detect the end frame.

[0526] The relevant contents of S111-S114 can refer to the description of the above video frame detection process, which will not be repeated here. Figure 4-Figure 39 The description of the relevant content will not be repeated here.

[0527] In this application, a video to be detected is obtained by capturing the screen of a target terminal while executing a camera operation response process. The video to be detected comprises multiple consecutive video frames. The camera operation response process is the process by which the target terminal operates different shooting functions. Different operation controls are used during the operation process. A control diagram corresponding to the camera operation response process can be pre-acquired. The control diagram includes control controls, and different camera operation response processes correspond to different control controls. The control diagram and the video frame containing the control controls are images taken from the same perspective. The position of the control controls in the control diagram is the same as in the video frame. Based on the position of the control controls in the control diagram, a region matching is performed within the video frame to obtain a first predetermined region within the video frame. Whether pixel changes occur in the first predetermined region in adjacent video frames is detected. Specifically, whether there are significant differences in the regions containing the control controls between adjacent video frames is determined. This allows the video frame where the control changes begin to be detected. The first frame where the control changes begin is the starting frame of the camera operation response process. By matching the regions containing the control controls within the video frames, targeted region detection is achieved, improving detection accuracy. Furthermore, the amount of data required to determine whether a pixel change occurs is reduced, thereby improving detection efficiency. After detecting the starting frame, the pixel changes in the video frame after the starting frame or the adjacent video frames in the video to be detected are detected to determine whether there is a picture or whether the image appears completely. The ending frame is detected by detecting whether there is a pixel change in the video frame or whether there is a difference between adjacent video frames, thereby realizing the automatic detection process of the video frame and improving the efficiency and accuracy of video frame detection.

[0528] The following combination Figure 41 This paper introduces the video frame detection process.

[0529] For the different scenarios described above, an operation control diagram including different operation controls can be pre-acquired. The operation controls include one or more of a camera application icon, a photo control, an image playback control, a return control, a flip control, and a video control. Furthermore, one or more of an operation response diagram, a shooting interface diagram, a baffle diagram, a component diagram, and a distant view diagram can be pre-acquired. The operation response diagram, shooting interface diagram, baffle diagram, component diagram, and distant view diagram can be screenshots of the captured content in the detection scene. Different configuration parameters are pre-set, including: a preset difference value for determining whether the operation control begins to change; a preset similarity for determining whether the color of the camera application icon begins to change; a first number for determining whether the image appears in the preview interface; a second number for determining whether the image is blurred in the preview interface; a first preset area for determining whether the thumbnail appears completely or stops zooming; a second preset area for determining whether the display image is fully opened or stops zooming; a preset duration; a third preset area for determining whether the distant view in the preview interface stops zooming; a preset shape; a first threshold for determining whether the image is a black frame; and a second threshold for determining whether the image changes from black to light. The preset difference value is proportional to the flicker degree of the camera in the terminal.

[0530] In this application, the above-mentioned camera operation response process involves multiple scenarios, and each scenario requires the identification of the start frame and the end frame. The recognition of video frames can be divided into the following types: operation control recognition, picture recognition in the preview interface (referring to the first frame of the picture that has just appeared, and the picture has not yet been fully displayed), complete picture recognition (referring to the picture has stopped zooming), picture recognition from black to bright, recognition of the complete removal of the baffle, and carousel recognition (including recognition of lights in video frames and recognition of lights in captured pictures).

[0531] The process for identifying control states is as follows: Through inter-frame difference analysis, the computer device identifies changes in the state of the control, using the first frame where the control begins to change as the starting frame. When the control changes from a static state to a clicked state, the area where the control resides changes significantly. Using the frame difference method to calculate the difference between adjacent video frames, this change is detected and the video frame where the control begins to change is identified. This process eliminates the need for manual template matching based on judgment criteria and is unaffected by changes in the style or position of the control, making it simple and efficient.

[0532] The image recognition process in the preview interface is as follows: Edge detection is performed using an edge detection operator to identify characteristic content in the preview interface, such as white lines. The first video frame whose line feature indicates a number of lines greater than a first number is used as the first frame appearing in the preview interface. When a picture first appears in the preview interface, it will contain a large amount of edge information. The Canny edge detection algorithm can be used to detect lines in the picture. This detection algorithm relies on edge features and can effectively detect these edges without being affected by the specific content of the picture, thereby improving detection accuracy.

[0533] The complete image recognition process is as follows: By analyzing frame differences, the computer device can detect the image zooming process. When large frame differences cease, the zooming process is considered complete. While the image is being zoomed, the differences between adjacent video frames are large. After the zooming process is complete, the differences between adjacent frames are small. Using the frame difference method, the video frame where the zooming process has ceased is detected. This eliminates the need for manual template matching based on judgment criteria and can adapt to video frames of varying resolutions, expanding the application scenarios of the detection method.

[0534] The process for identifying transitions from dark to bright is as follows: By setting a black frame threshold (i.e., the first threshold) and a black-to-bright threshold, the brightness difference between adjacent video frames can be determined to identify transition frames from dark to bright. Alternatively, by statistically analyzing the brightness information of all video frames, a brightness threshold is set, and the brightness of the current frame is detected. When the brightness exceeds a certain percentage of the brightness threshold, it indicates that the image has brightened, and the video frame where the image transitions from dark to bright can be detected. This method can adapt to different lighting conditions and does not rely on a fixed threshold, which increases the applicable scenarios of the detection method.

[0535] The process for identifying a completely removed baffle is as follows: Baffle movement is detected through grayscale image analysis and frame difference analysis, and changes in the component's state are detected during the initial movement of the baffle. Video frames are converted to grayscale images, and inter-frame differences are performed on the grayscale images of adjacent video frames to detect whether the baffle has moved and identify when the baffle has completely been removed from the preview interface. If the baffle has moved and the component's outline does not appear, video frames are processed further to detect whether the component's outline appears. If the baffle has moved and the component's outline appears, video frames are processed further to detect whether the image appears in the preview interface.

[0536] The process of identifying the time of the revolving lantern is as follows: different methods are used to identify the revolving lantern state (glowing lights and non-glowing lights) and the revolving lantern time for low-resolution video frames and high-definition pictures respectively.

[0537] Based on the multiple recognition processes and various detection scenarios introduced above, the key node detection method implemented by computer equipment is further explained. This method is applied to the camera frame counting process and automatically detects key nodes in the camera video through feature analysis and recognition algorithms.

[0538] Please refer to Figure 41 , Figure 41 This is a flow chart of a detection method provided in an embodiment of the present application. Figure 41 As shown, the method includes the following steps:

[0539] S211. Obtain target video, operation control diagram, and configuration parameters.

[0540] The input component of the computer device executing the video frame detection method provided in this application includes: the target video, the detection scene (i.e., the camera operation response process), and scene screenshots corresponding to each detection scene, such as the operation control diagram, operation response diagram, shooting interface diagram, baffle diagram, component diagram, and perspective diagram. The input component may also include configuration parameters and related interfaces that need to be adjusted during actual detection. The configuration parameters are thresholds obtained through pre-training during a large number of experiments.

[0541] The target video can be a video to be detected in the application phase or a video in the pre-training phase.

[0542] The configuration parameters include thresholds for determining whether pixels between adjacent video frames have changed, so as to determine whether significant visual changes have occurred in adjacent video frames. For example, a preset difference value for determining whether an operating control has begun to change, a preset similarity for determining whether the color of a camera application icon has begun to change, a first preset area for determining whether a thumbnail has appeared completely or has stopped zooming, a second preset area for determining whether a display image has been fully opened or has stopped zooming, a third preset area for determining whether a distant view in the preview interface has stopped zooming, and a second threshold for determining whether the image has changed from a black frame to a bright frame. The configuration parameters also include thresholds for determining whether pixels in a video frame have changed, so as to determine whether significant visual changes have occurred in the video frame. For example, a first number for determining whether an image appears in the preview interface, a second number for determining whether an image is blurred in the preview interface, and a first threshold for determining whether an image is a black frame.

[0543] S212: Match a first preset area of ​​the operation control in the video frame of the target video according to the position of the operation control in the operation control diagram.

[0544] The first preset area is a specific area that needs attention. Targeted area detection is performed according to the coordinates of the specific area, which can improve detection accuracy and detection efficiency.

[0545] S213 , reading the video frames in the target video frame by frame, performing grayscale processing on the video frames, and obtaining a grayscale image.

[0546] The target video is read frame by frame, broken down into individual frames, and processed individually. Converting the frames to grayscale before performing inter-frame differencing reduces computational complexity and effectively suppresses flicker, bypassing complex de-strobing algorithms and facilitating subsequent processing.

[0547] S214 , performing inter-frame difference on a first preset area of ​​the grayscale images of adjacent video frames using a frame difference method to obtain a frame difference value.

[0548] Key features can be extracted based on different detection scenarios. A frame difference method is used to calculate the inter-frame differences between the first preset region of the grayscale images of adjacent video frames to detect changes in the area where the control is located. The first frame where the frame difference exceeds the preset difference value is considered the first frame where the control begins to change.

[0549] For the camera application startup scenario, you can detect the color change of the video frame before performing grayscale conversion on the video frame to detect whether the color of the camera application icon starts to change.

[0550] S215: Determine whether adjacent video frames have changed based on the frame difference value and the preset difference value in the configuration parameters.

[0551] If not, continue to execute S214. If yes, execute S216.

[0552] Detect changes between adjacent video frames by comparing inter-frame differences between consecutive video frames. Specifically, determine whether pixels within adjacent video frames have changed. By comparing the inter-frame difference feature data with the configuration parameters, detect whether significant changes have occurred between adjacent video frames. If changes are detected between adjacent video frames, it indicates that the control has begun to change. The first frame where the control begins to change is considered the starting frame, and further analysis is performed on the video frames following the starting frame.

[0553] S216 , continue to perform grayscale processing on the video frames after the first frame where the operation control starts to change, to obtain a grayscale image.

[0554] S217: Detect pixel changes in the grayscale image of the video frame or the adjacent video frame after the first frame where the operation control starts to change, and determine whether a picture appears or whether the image appears completely.

[0555] If not, continue to execute S216, and if so, execute S218.

[0556] Key features can be extracted based on different detection scenarios. For example, edge detection algorithms can be used to extract line features to determine whether a scene is present, while frame difference methods can be used to calculate the inter-frame differences between adjacent video frames to determine whether the image is complete.

[0557] S218: Output the frame number of the first frame where the operation control starts to change, and the frame number of the video frame where the picture appears or the frame number of the video frame where the image appears completely.

[0558] The first frame where the control begins to change is the start frame. The frame number of the video frame where the screen appears or the video frame where the image appears completely is the video frame where the response ends, and the video frame where the response ends is the end frame.

[0559] The system detects whether the image appears in the preview interface by determining whether pixels in the video frame have changed. If the number of lines corresponding to the detected pixel changes in the video frame exceeds a first number, the frame number of the video frame is output. A frame difference method is used to determine whether the image appears completely. If the frame difference data between adjacent video frames indicates that the area of ​​the image outline is less than a preset area, indicating that the difference between adjacent video frames is smaller, the image appears completely, that is, the image has stopped scaling, and the frame number of the video frame is output.

[0560] For example, taking the switching from rear-facing shooting to front-facing shooting as an example, the switching scene is explained in combination with the above S211-S218. The key nodes that need to be detected in the switching scene include the first frame where the flip control begins to change and the first frame where the picture appears in the preview interface. For the switching scene, a screenshot of the detection target is obtained, which is used for area positioning and recognition in video frame processing. The detection target is matched, and the detection target includes the flip control and the detection object in the switched picture.

[0561] Based on the position of the flip control in the operation control diagram, the flip control is regionally located. Based on the position of the detection object (e.g., a still image) in the switched image, the detection object is regionally located. After completing the region matching, the video to be detected is traversed frame by frame. After grayscale processing is performed on adjacent video frames, the frame difference between the located regions of the adjacent video frames is calculated. When the frame difference value is greater than a preset difference value, it indicates that the flip control has begun to change, and the first frame where the flip control begins to change is used as the starting frame. After the starting frame is detected, an edge detection algorithm is used to draw differential edges for the video frames following the starting frame, detect white lines in the video frames, and calculate the number of lines using relevant computer vision methods. When the number of lines is below a lower threshold (i.e., less than a second number), the corresponding image is determined to be blurred, indicating that the image has switched from the rear camera's image. When the number of lines is above an upper threshold (i.e., greater than the first number), the corresponding image is determined to be clear again, indicating that the image has switched to the front camera's image, and the first clear frame is used as the ending frame.

[0562] Furthermore, the detection scenario also includes the implementation of specific functions. Specific functions include revolving lantern time recognition and revolving lantern state recognition. For low-resolution video frames, focus on brightness changes and specific shapes (for example, the circular outline of the lamp); for high-resolution images (such as high-definition pictures taken), directly identify the state of the lamp (that is, whether the lamp is lit) and position. For a description of revolving lantern time recognition and application scenarios, please refer to the above Figure 29-Figure 39 , I will not go into details here.

[0563] Specific functions also include baffle movement recognition, which can determine whether the baffle is completely removed by analyzing the difference in images before and after the baffle. For a description of baffle recognition and application scenarios, please refer to the above Figure 26-Figure 28 , I will not go into details here.

[0564] The configuration parameters are threshold values ​​obtained through pre-training during a large number of experiments. They can be obtained by executing the above-mentioned S211-S218. In S211, the pre-training stage needs to collect videos under various detection scenarios. Collect digital image data containing detection targets, such as screenshots of operating controls, pictures of the first and second objects detected, screenshots of baffles, screenshots of components, distant pictures, etc. These data are used for subsequent automated target capture and recognition. The collected video data has an authentication label of the detection scene. The video data under each detection scene should cover various possible test environments, such as different lighting conditions, different backgrounds, etc.

[0565] During the pre-training phase, the coordinates of specific areas of interest within the video frame are manually or automatically calibrated using methods such as template matching, based on the digital image data of the detection target collected during the video collection process. This regional localization reduces the amount of data processing required for subsequent analysis of specific areas within the video frame.

[0566] After processing the target video through steps S211-S218, the recognition result output from S218 can be compared with the actual situation to verify its accuracy. Based on the discrepancy between the recognition result and the actual situation, algorithm parameters, such as the threshold value, color feature extraction method, and region matching method set in the configuration parameters, can be adjusted to improve recognition accuracy and processing speed.

[0567] The embodiment of the present application provides a camera video key node matching algorithm based on template matching and edge detection, which realizes the automatic detection process of camera video key nodes. Based on image grayscale processing and feature matching, feature analysis and recognition are performed on video data, and automatic detection can be performed for the occurrence of some specific events in the video data. When detecting key nodes, video frames with pixel changes in specific areas can be detected by automatically matching specific areas in the video data. Compared with manual detection, the detection efficiency and adaptability of the detection scene are improved, and the accuracy and consistency of the key nodes are improved. The detected key nodes can be used for camera video key duration testing, which improves the reliability and convenience of the camera performance test process, and contributes to the automation of the camera performance test process (for example, the duration of the photo process).

[0568] In the embodiment of the present application, subjective and objective tests and log analysis can be used for relevant detection scenarios to determine whether the video frame detection scheme provided in the present application is the same. For test schemes and video frame automation test systems similar to those for detecting and identifying the duration of a photo-taking process, the same faults or specific samples are input into the detection system and the computer equipment of the present application. The specific samples are tested using the video frame detection method provided in the present application to obtain the test results, and the test results of the test system are captured. The test results of the two are compared. If the test results are highly consistent, it can be determined that the test system may have adopted the video frame detection method provided in the present application. Furthermore, the log of the test system can be obtained, and by analyzing the log, it can be determined whether the test scheme adopted by the test system is the same as the video frame detection method provided in the present application.

[0569] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (such as a coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0570] The above are optional embodiments provided for this application and are not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the technical scope disclosed in this application should be included in the scope of protection of this application.

Claims

1. A video frame detection method, characterized in that: The method comprises: Obtaining a video to be detected and a corresponding operation control diagram, wherein the video to be detected is obtained by capturing the screen of a target terminal executing a camera operation response process, and the operation controls in the operation control diagram are used to initiate the camera operation response process; performing region matching on a video frame in the video to be detected according to a position of the operation control in the operation control map to obtain a first preset region of the operation control in the video frame; Detecting pixel changes in a first preset area of ​​adjacent video frames in the video to be detected to obtain a first frame in which the operation control begins to change, where the first frame in which the operation control begins to change is a starting frame corresponding to the camera operation response process; Detecting a video frame located after the start frame in the video to be detected to obtain a response end video frame, where the response end video frame is an end frame corresponding to the camera operation response process; The detecting of the video frame after the start frame in the video to be detected to obtain the video frame at the end of the response includes: Sequentially extracting edge features from video frames located after the starting frame in the video to be detected to obtain line features of each video frame located after the starting frame; The first frame of the video frame in which the number of the line feature indicating lines is greater than the first number is used as the video frame in which the response ends.

2. The method according to claim 1, wherein The detecting pixel changes in a first preset area of ​​adjacent video frames in the video to be detected to obtain a first frame where the operation control begins to change includes: Performing inter-frame difference on a first preset area of ​​adjacent video frames in the video to be detected to obtain a frame difference value; The first frame in which the frame difference value is greater than the preset difference value is used as the first frame in which the operation control starts to change.

3. The method according to claim 2, wherein The performing inter-frame difference on a first preset area of ​​adjacent video frames in the video to be detected to obtain a frame difference value includes: Performing grayscale transformation on a video frame in the video to be detected to obtain a first grayscale image corresponding to the video frame; An inter-frame difference is performed on a first preset area of ​​a first grayscale image corresponding to adjacent video frames to obtain the frame difference value.

4. The method according to claim 1, wherein Before extracting edge features from the video frames after the starting frame in the video to be detected to obtain line features of each video frame after the starting frame, the method further includes: Obtaining an operation response diagram corresponding to the camera operation response process; performing region matching on the video frame in the video to be detected according to the position of the first object in the operation response graph to obtain a second preset region of the first object in the video frame; The step of sequentially extracting edge features from the video frames after the starting frame in the video to be detected to obtain line features of each video frame after the starting frame includes: Edge features are sequentially extracted from second preset areas of the video frames located after the starting frame in the video to be detected to obtain line features of each video frame located after the starting frame.

5. The method according to any one of claims 1 to 4, characterized in that The camera operation response process includes the camera application jumping from the gallery interface to the preview interface, the operation control is a return control, the first frame in which the return control starts to change is the start frame corresponding to the camera operation response process, and the first frame in which the picture appears in the preview interface is the end frame corresponding to the camera operation response process.

6. The method according to any one of claims 1 to 5, characterized in that: The camera operation response process includes switching from front-facing shooting to rear-facing shooting or from rear-facing shooting to front-facing shooting, the operation control is a flip control, the first frame in which the flip control starts to change is the start frame corresponding to the camera operation response process, and the first frame that appears in the preview interface is the end frame corresponding to the camera operation response process; The step of taking the first frame of the video frame in which the number of the line feature indicating lines is greater than the first number as the video frame in which the response ends includes: In a process in which the number of lines indicated by the line features of multiple video frames located after the starting frame gradually decreases, if it is detected that the number of lines is less than the second number, and in a process in which the number of lines indicated by the line features of the multiple video frames gradually increases, if it is detected that the number of lines is greater than the first number, the first frame of the video frames in which the number of lines is greater than the first number is used as the first frame of the picture appearing in the preview interface.

7. The method according to any one of claims 1 to 4, characterized in that The camera operation response process includes starting a camera application for shooting, the operation control is a camera application icon, the first frame in which the camera application icon starts to change is the start frame corresponding to the camera operation response process, and the first frame in which the picture appears in the preview interface is the end frame corresponding to the camera operation response process.

8. The method according to claim 1 or 4, wherein: The camera operation response process includes starting a camera application to shoot, the operation control is a camera application icon, and the first frame in which the camera application icon starts to change is the starting frame corresponding to the camera operation response process; The detecting pixel changes in a first preset area of ​​adjacent video frames in the video to be detected to obtain a first frame where the operation control begins to change includes: Performing color extraction on a first preset area of ​​a video frame in the video to be detected to obtain a color feature; determining color similarity between adjacent video frames based on the color features; The first frame of the video frame whose color similarity is greater than the preset similarity is used as the first frame at which the camera application icon starts to change.

9. The method according to any one of claims 1 to 3, wherein: The camera operation response process includes receiving a trigger operation on a photo control and generating a thumbnail corresponding to the latest captured image in an image echo control, wherein the image echo control is used to trigger the display of the latest captured image, the operation control is a photo control, the first frame in which the photo control begins to change is the start frame corresponding to the camera operation response process, and the last frame in which the thumbnail changes is the end frame corresponding to the camera operation response process; The video frame located after the start frame in the video to be detected is detected, and the video frame at the end of the response is obtained and replaced with: Performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain first frame difference data; The video frame in which the area of ​​the picture outline indicated by the first frame difference data is smaller than the first preset area is used as the last frame of the thumbnail change.

10. The method according to claim 9, wherein Before performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain first frame difference data, the method further includes: Get the shooting interface image; Performing region matching on the video frame in the video to be detected according to the position of the image echo control in the shooting interface diagram to obtain a third preset region of the thumbnail in the video frame; The performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain first frame difference data includes: Performing grayscale conversion on a video frame located after the start frame in the video to be detected to obtain a second grayscale image corresponding to the video frame; Inter-frame difference is performed on a third preset area of ​​the second grayscale image corresponding to adjacent video frames to obtain the first frame difference data.

11. The method according to any one of claims 1 to 3, wherein: The camera operation response process includes receiving a trigger operation on an image echo control and displaying a display image corresponding to a thumbnail in the image echo control. The operation control is the image echo control, and the image echo control is used to trigger the display of the latest captured image. The first frame in which the image echo control begins to change is the start frame corresponding to the camera operation response process, and the first frame in which the display image is fully opened is the end frame corresponding to the camera operation response process. The video frame located after the start frame in the video to be detected is detected, and the video frame at the end of the response is obtained and replaced with: Performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain second frame difference data; The video frame in which the area of ​​the picture outline indicated by the second frame difference data is smaller than the second preset area is used as the first frame of the fully opened display image.

12. The method according to claim 1, wherein The camera operation response process includes switching from close-up shooting to long-range shooting, the first frame in which the baffle of the close-up shooting is completely moved is the start frame corresponding to the camera operation response process, and the first frame in which the long-range shooting stops zooming is the end frame corresponding to the camera operation response process, and the operation control diagram is a baffle diagram; The performing region matching on the video frame in the to-be-detected video according to the position of the operation control in the operation control map to obtain a first preset region of the operation control in the video frame includes: Performing region matching on the video frame in the video to be detected according to the position of the baffle in the baffle map to obtain the baffle region in the video frame; The detecting pixel changes in a first preset area of ​​adjacent video frames in the video to be detected to obtain a first frame where the operation control begins to change includes: performing inter-frame difference on the baffle regions of adjacent video frames in the video to be detected to obtain third frame difference data; When the third frame difference data indicates that the baffle has moved, performing inter-frame difference on adjacent video frames in the video to be detected to obtain fourth frame difference data; The first frame in which the fourth frame difference data indicates that the outline of the target component appears is used as the first frame in which the baffle is completely removed, and the target component is the component displayed after the baffle is completely removed; The video frame located after the start frame in the video to be detected is detected, and the video frame at the end of the response is obtained and replaced with: Performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain fifth frame difference data; The video frame in which the area of ​​the picture outline indicated by the fifth frame difference data is smaller than the third preset area is used as the first frame for stopping zooming of the distant picture.

13. The method according to claim 12, wherein: The method further comprises: Get the component diagram; Performing region matching on the video frame in the video to be detected according to the position of the target component in the component map to obtain a component region of the target component in the video frame; When the third frame difference data indicates that the baffle has moved, performing inter-frame difference on adjacent video frames in the video to be detected to obtain fourth frame difference data includes: When the third frame difference data indicates that the baffle has moved, grayscale conversion is performed on the video frame in the video to be detected to obtain a third grayscale image corresponding to the video frame; Inter-frame difference is performed on the component area of ​​the third grayscale image corresponding to adjacent video frames to obtain the fourth frame difference data.

14. The method according to claim 12 or 13, wherein: Before performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain fifth frame difference data, the method further includes: Get a vision map; performing region matching on the video frame in the to-be-detected video according to the position of the second object in the distant view, to obtain a fourth preset region of the second object in the video frame; The performing inter-frame difference on adjacent video frames located after the start frame in the video to be detected to obtain fifth frame difference data includes: Performing grayscale conversion on a video frame located after the start frame in the video to be detected to obtain a fourth grayscale image corresponding to the video frame; Inter-frame difference is performed on a fourth preset area of ​​the fourth grayscale image corresponding to adjacent video frames to obtain the fifth frame difference data.

15. The method according to any one of claims 1 to 3, wherein: In the case where the shooting scene includes a dynamic timer, the camera operation response process includes from receiving a trigger operation on a shooting control to completing shooting of the dynamic timer, the operation control is a shooting control, and the first frame in which the shooting control begins to change is the starting frame corresponding to the camera operation response process; The method further comprises: Determine the first time indicated by the dynamic timer in the first frame where the photo control starts to change, and determine the second time indicated by the dynamic timer in the captured picture, where the captured picture refers to the picture obtained by the target terminal after the dynamic timer completes the shooting.

16. The method according to claim 15, wherein The dynamic timer includes a light array for dynamic display, wherein the light array includes a plurality of lights arranged in rows and columns; Determining the first time indicated by the dynamic timer in the first frame in which the photographing control begins to change includes: Performing color recognition on each video frame in the video to be detected to obtain the outline of the light emitting lamp in each video frame; constructing a first coordinate matrix of the dynamic timer according to the outlines of the lights emitting light in each video frame; Performing color recognition on the first frame in which the photo control starts to change to obtain a first outline; the first outline is the outline of the light emitting in the first frame in which the photo control starts to change; Performing position matching in the first coordinate matrix based on the position of the first contour in the first frame in which the photographing control begins to change, to obtain the position of the first contour in the first coordinate matrix; The first time is determined according to the position of the first contour in the first coordinate matrix.

17. The method according to claim 15 or 16, wherein: The determining the second time indicated by the dynamic timer in the captured picture includes: Performing shape recognition on the image according to a preset shape to obtain a second coordinate matrix of the dynamic timer; the preset shape is the physical shape of the lamp, and the second coordinate matrix indicates the position of the unlit lamp; Performing color recognition on the image to obtain a second outline; the second outline is an outline of a light in the image; Performing position matching in the second coordinate matrix according to the position of the second contour in the image to obtain the position of the second contour in the second coordinate matrix; The second time is determined according to the position of the second contour in the second coordinate matrix.

18. The method according to any one of claims 15 to 17, wherein: After determining the first time indicated by the dynamic timer in the first frame in which the photographing control begins to change, and determining the second time indicated by the dynamic timer in the captured picture, the method further includes: Determine a delay duration for a camera photographing of the target terminal according to the first time and the second time.

19. The method according to any one of claims 1 to 3, wherein: In a case where the shooting scene includes a dynamic timer, the dynamic timer includes a dynamically displayed light array, the light array includes a plurality of lights arranged in rows and columns, the camera operation response process includes executing a process in response to a triggering operation of a photo control, the operation control is a photo control, and the first frame in which the photo control begins to change is the starting frame corresponding to the camera operation response process; The video frame located after the start frame in the video to be detected is detected, and the video frame at the end of the response is obtained and replaced with: Detecting the video frames after the start frame in the video to be detected, determining the first frame in which the screen in the preview interface changes from black to bright and the last frame in which the dynamic timer displays a freeze, wherein the position of the light in the video frames before and after the dynamic timer displays a freeze changes from being unchanged to being changed; The video frame that is sorted after the first frame in which the screen changes from black to bright and the last frame in which the dynamic timer displays a freeze is used as the end frame corresponding to the camera operation response process.

20. The method according to claim 19, wherein The detecting of the video frames after the start frame in the video to be detected, and determining the first frame in which the screen in the preview interface changes from black to bright, includes: When the brightness of the current video frame is less than a first threshold and the brightness difference between the current video frame and a previous video frame of the current video frame is greater than a second threshold, the current video frame is used as the first frame of the screen changing from black to bright; Alternatively, a brightness threshold is determined based on the brightness of each video frame in the video to be detected; in the process of gradually increasing brightness of multiple video frames located after the starting frame, the video frame with brightness greater than the brightness threshold is used as the first frame in which the picture changes from black to bright.

21. The method according to claim 19 or 20, wherein: The detecting the video frame located after the start frame in the video to be detected and determining the last frame of the dynamic timer displaying a freeze includes: Performing color recognition on each video frame after the start frame in the video to be detected to obtain a third outline of each video frame; the third outline is the outline of the light emitting lamp in the video frame after the start frame; Determining a plurality of target video frames from among the video frames following the start frame; wherein the third contours of the plurality of target video frames are all the same, and the third contour of a video frame subsequent to the last frame in the plurality of target video frames is different from the third contours of the plurality of target video frames; The last frame among the multiple target video frames is used as the last frame of the dynamic timer display freeze.

22. The method according to any one of claims 1 to 21, wherein: After detecting the video frame located after the start frame in the video to be detected and obtaining the video frame for the response end, the method further includes: Determining a target number of frames according to the start frame and the end frame; The duration of the camera operation response process or the shooting rate is determined according to the target frame number.

23. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 22 when executed by the processor.

24. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 22.

25. A computer program product comprising instructions, characterized in that When the method is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 22.

Images