Transition dynamic effect detection method, electronic equipment, readable storage medium and chip

Abstract

The invention provides a transition dynamic effect detection method, electronic equipment, a readable storage medium and a chip, and relates to the technical field of testing. The method comprises the steps of performing shared element tracking on a transition video to be detected, and determining tracking information of a tracking object; according to the tracking information, the motion time interval of the tracking object and the motion information of the tracking object in the motion time interval are determined, and the motion information comprises the frame length, the speed and the acceleration; and determining a transition mode of the tracking object in the motion time interval according to the motion information. According to the method provided by the embodiment of the invention, the transition mode of the tracking object can be accurately determined through the motion information such as the frame length, the speed and the acceleration of the tracking object.

Description

Technical Field

[0001] This application relates to the field of testing technology, and in particular to a method for detecting transition effects, an electronic device, a readable storage medium, and a chip. Background Technology

[0002] Transition animations refer to the smooth transitions between different interfaces or elements in an application when switching interfaces or changing elements within the interface. They are also known as transition effects or motion effects in general. To ensure the quality of application motion effects, it is necessary to test them, determining the transition method, whether the transition process is natural and smooth, and whether the transition method matches the application scenario, thereby ensuring that the motion effects meet transition design specifications. Currently, the YOLOv8 model is commonly used for motion effect testing. However, the YOLOv8 model has poor motion effect detection performance, for example, incomplete or inaccurate detection. Therefore, there is an urgent need to provide a new method for detecting transition animations. Summary of the Invention

[0003] This application provides a method, electronic device, readable storage medium, and chip for detecting transition effects, which can improve the detection effect of transition effects.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] In a first aspect, embodiments of this application provide a method for detecting transition animation effects. The method includes: performing shared element tracking on the transition video to be detected to determine the tracking information of the tracking object; determining the motion time interval of the tracking object and the motion information of the tracking object within the motion time interval based on the tracking information, the motion information including border length, speed and acceleration; and determining the transition mode of the tracking object within the motion time interval based on the motion information.

[0006] The method provided in this application embodiment can accurately determine the transition mode of the tracked object based on motion information such as the bounding box length, speed and acceleration of the tracked object, and has a good motion effect detection effect.

[0007] In some embodiments, shared element tracking is performed on the transition video to be detected to determine the tracking information of the tracking objects. This includes: dividing the transition video into multiple video frames; determining the tracking objects in each video frame and the tracking information of each tracking object; identifying abnormal tracking objects in each video frame based on the tracking information; and generating a tracking object set W based on the remaining tracking objects in each video frame, excluding the abnormal tracking objects. The tracking object set W includes at least one tracking object. The method provided in this embodiment can accurately identify tracking objects and avoid the impact of abnormal tracking objects on the subsequent motion effect detection process.

[0008] In some embodiments, the tracking information for the tracked object includes boundaries, type, tracking identifier ID, and confidence level.

[0009] In some embodiments, the shared element is a search box, and based on the tracking information, the identification of abnormal tracking objects in each video frame includes: identifying tracking objects with a confidence level less than a confidence level threshold as abnormal tracking objects; and / or identifying tracking objects whose type is not an input box as abnormal tracking objects; and / or identifying tracking objects whose type is a text box as abnormal tracking objects.

[0010] In some embodiments, before determining the motion time interval of the tracked object based on the tracking information, the method further includes: if there is a first tracked object and a second tracked object in the transition video, and the first tracked object disappears and the second tracked object appears after a first time interval, then within the first time interval, determining whether there is a virtual keyboard in the transition video based on the number of single text characters in each video frame of the transition video; if there is a virtual keyboard in the transition video within the first time interval, determining the missing boundary within the first time interval; and merging the missing boundary within the first time interval and the boundary of the second tracked object into the first tracked object.

[0011] The method provided in this embodiment can generate a boundary for a shared element when the shared element exists but the model cannot detect it, and merge and unify multiple tracking IDs when the same shared element is assigned multiple tracking IDs.

[0012] In some embodiments, if a virtual keyboard exists in the transition video during a first time period, determining the missing boundary during the first time period includes: if a virtual keyboard exists in the transition video during the first time period, determining whether the first tracking object and the second tracking object are substantially at the same horizontal position; if the first tracking object and the second tracking object are substantially at the same horizontal position, determining an average boundary based on the last M video frames of the first tracking object and the first N video frames of the second tracking object; and using the average boundary as the missing boundary during the first time period; if the first tracking object and the second tracking object are not at the same horizontal position, using the last boundary of the first tracking object as the missing boundary during the first time period.

[0013] In some embodiments, before determining the motion information of the tracked object based on the boundary of the tracked object within the motion time interval, the method further includes: determining the HSV values ​​of all pixels of the tracked object within the boundary range; determining the key HSV values ​​of the HSV values ​​of all pixels of the tracked object; and calibrating the boundary of the tracked object based on the maximum and minimum values ​​of brightness in the key HSV values.

[0014] The method provided in this embodiment can correct the boundaries of the tracked object and improve its pixel-level accuracy, so as to accurately identify its transition type and whether the one-shot transition is smooth.

[0015] In some embodiments, determining the motion time interval of the tracked object based on the tracking information includes: linearly fitting the coordinates of the boundary center of the tracked object in a first direction to obtain a linear function, wherein the first direction includes the horizontal and vertical directions; and repeatedly performing the following operations (1) to (3) until the function segments of the linear function cannot be merged, thereby obtaining at least one merged time interval.

[0016] (1) Determine the slope and time interval of each function segment of the linear function.

[0017] (2) Determine the motion state of the tracked object within each function segment based on the slope; wherein, the motion state includes moving or stationary. When the slope is greater than or equal to the slope threshold, the tracked object moves within the corresponding function segment; when the slope is less than the slope threshold, the tracked object remains stationary within the corresponding function segment.

[0018] (3) Merge adjacent function segments and time intervals with the same motion state. Finally, determine the motion time interval from at least one merged time interval based on the slope of the linear function and the motion state of the tracked object.

[0019] The method provided in this embodiment can determine the time range of motion of the tracked object, so as to subsequently determine and analyze its motion information.

[0020] In some embodiments, determining the transition mode of the tracked object within a motion time interval based on motion information includes: determining motion information of the tracked object in a first direction based on the boundary of the tracked object within the motion time interval, the first direction including lateral and longitudinal directions; and determining the transition mode of the tracked object in the first direction based on the motion information.

[0021] In some embodiments, determining the transition mode of the tracked object in the first direction based on motion information includes: if the tracked object simultaneously meets the following conditions (1) to (4) within the motion time interval, then determining that the transition mode of the tracked object in the first direction is displacement entry.

[0022] (1) The length of the border of the tracked object gradually increases in the first direction.

[0023] (2) The first few elements of the acceleration of the tracked object in the first direction are not monotonically increasing.

[0024] (3) The tracking object moves without acceleration in the first direction.

[0025] (4) The tracked object comes to a stop after decelerating in the first direction.

[0026] In some embodiments, determining the transition mode of the tracked object in the first direction based on motion information includes: if the tracked object simultaneously meets the following conditions (1) to (3) within the motion time interval, then determining that the transition mode of the tracked object in the first direction is displacement exit.

[0027] (1) The length of the bounding box of the tracking object gradually decreases in the first direction.

[0028] (2) The last element of the acceleration is greater than 0.

[0029] (3) The tracked object does not exist in the time interval after the movement time interval.

[0030] In some embodiments, determining the transition mode of the tracked object in the first direction based on motion information includes: if the tracked object simultaneously meets the following conditions (1) to (3) within the motion time interval, then determining that the transition mode of the tracked object in the first direction is a single shot.

[0031] (1) The tracked object has a stationary time interval in the first direction that is located before and adjacent to the motion time interval.

[0032] (2) The tracked object has a stationary time interval in the first direction that is located after the motion time interval and adjacent to the motion time interval.

[0033] (3) Each video frame within the motion time interval includes the tracked object.

[0034] The above embodiments detect whether the object being tracked is a displacement entry, displacement exit, or a one-shot transition based on changes in the object's border length, speed, and acceleration. The detection results have high accuracy.

[0035] In some embodiments, when the transition of the tracked object is a single-shot sequence, the method further includes: determining, within the motion time interval, the actual speed value and the fitted speed value of the tracked object along the first direction in each video frame of the transition video; and determining video frames in which the difference between the actual speed value and the fitted speed value is greater than a threshold as video frames with unsmooth transitions along the first direction.

[0036] And / or, within the motion time interval, determine the acceleration of the tracked object along a first direction in each video frame of the transition video; if the acceleration of the first video frame is positive, but there is a second video frame with negative acceleration before the first video frame, and the acceleration of the second video frame is greater than an acceleration threshold, then determine that the transition of the second video frame along the first direction is not smooth; wherein, the number of frames between the first video frame and the second video frame is less than the threshold; if the acceleration of the first video frame is negative, but there is a second video frame with positive acceleration after the first video frame, then determine that the second video frame is an abnormal video frame.

[0037] In a second aspect, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method shown in the first aspect above.

[0038] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method shown in the first aspect above.

[0039] Fourthly, embodiments of this application provide a chip including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the method shown in the first aspect above.

[0040] Fifthly, embodiments of this application provide a computer program product comprising a computer program that, when executed by an electronic device, causes the electronic device to implement the method shown in the first aspect above.

[0041] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0042] Figure 1 This is a schematic flowchart of a motion effect detection method provided in one embodiment of this application;

[0043] Figure 2A This is a schematic diagram illustrating the identification result of a tracking ID provided in one embodiment of this application;

[0044] Figure 2B This is a schematic diagram of the identification result of the tracking ID provided in another embodiment of this application;

[0045] Figure 3 This is a schematic diagram of image coordinates provided in one embodiment of this application;

[0046] Figure 4This is a schematic diagram of a virtual keyboard display scenario provided in one embodiment of this application;

[0047] Figure 5 This is a schematic diagram of a transition effect detection device provided in one embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of a chip provided in one embodiment of this application. Detailed Implementation

[0049] The technical solutions provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0050] It should be understood that in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0051] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0052] For ease of description and understanding, some terms used in this application are explained below.

[0053] (I) Transition

[0054] A transition refers to the process of switching the display interface of an electronic device from one interface to another. During this process, the two interfaces switching can be different interfaces of the same application, different interfaces of different applications, or the main screen of the electronic device and any application interface.

[0055] (II) Transition Dynamic Effects

[0056] Transition animations, also known as transition effects or simply transition animations, refer to the animated transitions between different interfaces or elements when an application switches screens or changes elements within the interface. Transition animations help enhance the visual smoothness of interface changes and improve the user experience. At the same time, transition animations can guide users' attention to key elements on the interface, helping them understand the logical relationships between the interface and its elements.

[0057] (III) Transition Methods

[0058] Depending on how the elements in the transition animation change during the transition, transition methods include one-shot transitions, displacement entry, and displacement exit. See below for details.

[0059] (1) One-shot sequence

[0060] A seamless transition, also known as a shared transition, is an choreography technique that uses shared elements, containers, and momentum to create smooth transitions. This helps improve user efficiency and enhances visual fluidity. In other words, the transition interface features a persistent or continuously changing interface element—the shared element. A seamless transition makes the switching between two interfaces more fluid, resulting in an uninterrupted user experience.

[0061] Shared elements, defined as individual elements, are persistent or continuously changing interface elements in the transitional interface. Shared elements can establish a natural connection and transition between pages; these elements are usually key elements that require user attention, enhancing the sense of continuity in the transition. It should be noted that during the transition, elements in the status bar at the top of the interface on an electronic device may persist, as may elements such as virtual navigation buttons at the bottom of the interface, but these are not shared elements in the embodiments of this application.

[0062] Shared containers are units that hold a group of shared elements. Page transitions are achieved through the continuous movement of the components and changes in the elements within them. When a group of shared elements has clear boundaries during a transition, the size and border curvature of the container can be tweened, and the elements within the container can transition smoothly using fade-in / fade-out or shared element techniques.

[0063] Shared motion refers to transitions that cannot be smoothly achieved through tweening. Instead, usable shared transformation properties can be extracted to create smooth transitions between the preceding and following parts. Commonly used shared motion properties include displacement, scaling, and rotation.

[0064] Tweening refers to the automatic manipulation of content (such as the position, size, angle, and transparency of elements) between keyframes using computer calculations to create smoother transitions between them. Examples include transforming a square into a circle and vice versa.

[0065] In one example, when a user interacts with the search box in the application interface, the search box typically moves to the top of the screen to make way for elements such as the virtual keyboard and suggested search terms during the search process. This movement is usually accompanied by a localized change in the appearance of the search box. Throughout this search transition, the search box is a shared element that remains on the screen; therefore, this transition is a typical one-shot transition.

[0066] (2) Displacement entry

[0067] A displacement entry can be understood as an interface element gradually appearing during a transition. For example, a certain interface element may not be present in the first interface, but during the transition from the first interface to the second interface, this interface element gradually appears from left to right in the transition interface and reaches a complete state in the second interface. This interface element can be called an entry element or a field entry element.

[0068] (3) Displacement and exit

[0069] Displacement exit can be understood as a process where a UI element gradually moves and disappears during a transition. For example, the first UI may contain a certain UI element, but during the transition from the first UI to the second UI, this UI element gradually moves from right to left and disappears in the transition UI, and then completely disappears in the second UI. This UI element can also be called an exit element or a departure element.

[0070] To ensure the quality of application animations, it is necessary to test them, determining aspects such as transition methods, the smoothness of transitions, and whether the transition methods match the application scenario. This ensures that the animations meet transition design specifications. For example, for search transitions, it is recommended to use shared elements. The search box serves as a persistent shared element connecting the two screens. Other elements can use fade-in / fade-out or other transition methods. Transitions should not be done by switching directly in a single frame or using non-shared elements, and the transition animations should be natural and smooth.

[0071] Currently, the YOLOv8 model is commonly used for transition animation detection. However, the YOLOv8 model has poor performance in detecting transition animations. For example, it cannot detect transitions that are continuous in one shot, it cannot detect various graphic user interface (GUI) controls, it cannot detect whether the movement of target elements is smooth, and it lacks the ability to detect anomalies based on target tracking results. GUI controls are visual components that integrate display and control functions, including image controls, button controls, text controls, container controls (such as list boxes, dialog boxes, and menus), and utility controls (such as progress bars and scroll bars). This embodiment does not impose any restrictions on these.

[0072] Therefore, embodiments of this application provide a method for detecting transition animation effects, which can improve the detection effect of transition animation effects. For example, it can accurately determine the transition mode based on the motion information of shared elements (such as boundaries, velocity, and acceleration).

[0073] Figure 1This is a schematic flowchart illustrating a motion effect detection method provided in one embodiment of this application. The method specifically includes the following steps S1 to S7.

[0074] S1: Perform shared element tracking on the transition video to be detected to determine the tracking information of the tracking object.

[0075] In this embodiment, there may be one or more tracked objects; this embodiment does not impose any limitation on this. The tracking information for each tracked object includes: boundary, type, tracking identifier (ID), and confidence level.

[0076] S2: Correct the tracking information of the tracked object.

[0077] It should be noted that S2 is an optional step. For example, correcting the tracking information of a tracking object can be done by: if the same shared element is misidentified as multiple tracking objects, then filling in the missing boundaries between the multiple tracking objects and merging the multiple tracking objects into a single tracking object.

[0078] S3: Perform pixel-level correction on the boundaries of each tracked object in each video frame.

[0079] It should be noted that S3 is an optional step. Boundary correction can make the boundaries in the tracking information more accurate.

[0080] S4: Determine the time interval of the tracked object's movement based on the tracking information.

[0081] S5: Based on the tracking information of the tracked object, determine the movement information of the tracked object within the movement time interval. Also, based on the movement information of the tracked object, determine the transition method of the tracked object within the movement time interval.

[0082] For example, the transition methods include displacement entry, displacement exit, and one-shot transition.

[0083] S6: Based on the motion information of the tracked object, determine whether the one-shot transition process of the tracked object is smooth.

[0084] S7: Output the test report.

[0085] The transition animation detection method provided in this embodiment can be applied to software such as DevecoTesting to assist developers in detecting and locating animation specification issues. Alternatively, it can be integrated into a cloud testing platform, where developers can upload application packages to conduct user experience (UX) application experience standard testing to ensure that the display effect of animations within the application on different devices complies with the OpenHarmony ecosystem standard, and can be used to support application market listing testing.

[0086] In the specific implementation process, the embodiments of this application use remote procedure call (RPC) communication to capture virtual screen data of OpenHarmony single-frame device on personal computer (PC) for parsing in-application transition effects. The present invention can determine whether the transition is performed in the form of shared transition (one-shot), detect whether the one-shot transition is smooth and fluid, and output video frames with abnormalities.

[0087] The following example, using the transition video as an example where the search transition video is the shared element in the transition video, is the search box, will provide a detailed explanation and description of the above steps S1 to S7.

[0088] S1: Perform target tracking on shared elements in the transition video to be detected, and determine the tracking information of the target.

[0089] For example, S1 specifically includes the following steps S10 to S15.

[0090] S10, acquire the transition video to be detected.

[0091] In this embodiment, the transition video can be a video obtained through screen recording, filming, or other methods. For example, when a user searches for information using application A installed on an electronic device, they can control the electronic device to record the screen of the search transition process of application A, thus obtaining a search transition video. Exemplarily, the shared element in the search transition video is typically the search box.

[0092] S11: Divide the search transition video into frames and determine the timestamp t of each video frame.

[0093] In some embodiments, the electronic device uses an open-source computer vision library (OpenCV) to segment the transition video into multiple video frames. Then, each frame is assigned a unique timestamp t according to its playback order in the video. t This represents the video frame corresponding to the timestamp t.

[0094] S12: Determine the F value for each video frame. t The tracking information for the tracked object includes its boundaries, type, tracking ID, and confidence level.

[0095] In some embodiments, the electronic device may input video frames into a YOLOv8 model (hereinafter referred to as the model) trained on the RICO dataset in ascending order of timestamp t, and configure the tracker of the model to remain between different video frames to obtain the boundary b of the tracked object contained in each video frame. i,t The file contains the type (w.cls), tracking ID, and confidence level (w.conf).

[0096] It should be noted that in this embodiment, when the YOLOv8 model performs target tracking on the search transition video, it expects to track the search box as the target object. However, there may actually be some misidentified objects, such as text boxes, virtual keyboards, folders, etc., whose outlines are similar to the search box, and mistakenly identify them as the search box. For example... Figure 2A and Figure 2B As shown, while correctly identifying the search box, the device mistakenly identified some text display areas as tracking objects and assigned different tracking IDs to each object, such as tracking ID0 to tracking ID8. Therefore, the electronic device needs to filter the tracking objects and remove unreliable tracking results.

[0097] S13: For each video frame F t Based on the tracking information of the tracked object, from TRACK(F t Remove tracking objects with a confidence level lower than the confidence threshold or whose type is not an input box.

[0098] In this embodiment, taking a confidence threshold of 0.7 as an example, S13 can be expressed as:

[0099]

[0100] Among them, TRACK(F t ) represents the set of tracked objects tracked in the video frame with timestamp t, w represents a tracked object, w.conf represents the confidence level of the tracked object, w.id represents the tracking ID of the tracked object, and EditText represents the input box.

[0101] In addition, this embodiment does not impose any restrictions on the confidence threshold. For example, the confidence threshold can also be 0.65, 0.75, etc.

[0102] S14: For each video frame F t Remove tracking objects of type textbox from the tracking results.

[0103] In some embodiments, during a search transition, the number of video frames containing ordinary text boxes is typically less than a threshold, or they are located in the lower half of the screen. The electronic device can use this characteristic to filter out ordinary text boxes from the tracked objects. For example, see... Figure 3As shown, taking the coordinates of the top-left corner of the video frame as (0,0) as an example, the horizontal coordinate of the image gradually increases from left to right along the diagram, and the vertical coordinate of the image gradually increases from top to bottom along the diagram. Based on this, when the average value of the center vertical coordinate of the tracked object in each video frame is greater than K times (such as 2 / 5, 1 / 2 times) the screen height, the tracked object is considered to be located in the lower half of the screen, and is considered to be a regular text box.

[0104] Taking a threshold of 5 and K = 2 / 5 as an example, S15 specifically means: excluding tracking objects with a video frame count less than 5 or a mean center vertical axis coordinate greater than 0.4 times the screen height, that is:

[0105]

[0106] Where, φ x (·) is used to obtain the center x-axis coordinate of the area where the tracked object is located, φ y (·) is used to obtain the center vertical axis coordinate of the area where the tracked object is located, and HEIGHT(·) is used to obtain the height of the video frame.

[0107] S15: Generate a set of tracking objects W based on the remaining tracking objects in each video frame.

[0108] It should be noted that in search scenarios, S1 can track various tracking objects, such as GUI controls, and filter out non-search box controls, resulting in relatively accurate tracking results.

[0109] S2: Correct the tracking information of the tracked object.

[0110] In this embodiment, the electronic device can use optical character recognition (OCR) technology to correct the tracking ID of the tracked object, ensuring that the same shared element has the same tracking ID during the tracking process. Additionally, it can identify video frames with abnormal transitions and add them to the corresponding abnormal video frame set E. bug middle.

[0111] For example, the process specifically includes the following steps S20 to S210.

[0112] S20: Determine the F of each video frame t The number of objects being tracked is determined, and the set of abnormal video frames E is initialized. bug .

[0113] For example, each video frame F can be... t The tracking results TRACK(F) tCoexistence detection is performed to determine the number of tracked objects in the video frame. Then, based on the number of tracked objects, the video frame F is determined. t Check for any abnormal transition methods and determine the type of abnormality. Additionally, initialize the abnormal video frame set E. bug This can be understood as clearing E. bug So that E bug It is an empty set, that is

[0114] After determining the number of tracked objects in each video frame, the video frames with transition anomalies can be identified based on the number of tracked objects. Examples include video frames with transition anomalies caused by the simultaneous presence of multiple tracked objects, video frames with a virtual keyboard, video frames with missing motion effects for tracked objects, video frames where tracked objects move into the frame, and video frames where tracked objects move out of the frame. See S21 to S25 for details.

[0115] S21: If video frame F t If the video frame contains multiple tracking objects, then the F-frame will be... t Video frames marked as causing transition anomalies due to the simultaneous presence of multiple tracked objects are then assigned an F-string. t Add to E bug .

[0116] Taking the search box as an example, typically in a search scenario, the application interface only has one search box for the user to enter search keywords. Therefore, when the electronic device detects video frame F... t When multiple search boxes appear in the video frame, it is determined that the video frame is abnormal, and the video frame should be F... t Add to the set E of video frames containing the anomaly bug .

[0117] In this embodiment, video frame F t Add to E bug The condition can be expressed as the following formula:

[0118] E bug =E bug ∪{F t ||TRACK(F t )|>1,t=1,2,...}

[0119] S22: If there are two different tracking objects in the tracking object set W, and the first tracking object disappears and the second tracking object appears after a first time interval, then the video frames within the first time interval are subjected to keyboard detection.

[0120] The following explanation of S22 will take the search box as an example.

[0121] In some scenarios, such as Figure 4 As shown, in a text messaging application, when a user clicks the search box, the electronic device controls the search box to gradually move upwards to the top of the display screen, and controls each text message to move downwards until it disappears, in order to make way for the virtual keyboard and control the virtual keyboard to gradually appear from bottom to top. However, during the process of the virtual keyboard appearing from nothing, the search box may actually exist in several video frames (e.g., 1 to 4 video frames) but cannot be recognized by the model, resulting in the same search box being recognized as different search boxes by the model.

[0122] For example, see Figure 4 As shown in (a) above, before the virtual keyboard is displayed, the search box is recognized by the model as a tracking object with a tracking ID of ID1. (See also...) Figure 4 As shown in (b), during the process of the virtual keyboard appearing from nothing, the model cannot recognize the search box due to the influence of the virtual keyboard, resulting in the search box having no tracking ID. See also Figure 4 As shown in (c), after the virtual keyboard is successfully displayed, the search box will be recognized as another tracking object with tracking ID 2.

[0123] In this embodiment, the duration of the tracked object can be represented by the number of video frames. Therefore, the first time period can be the duration of one or more video frames, for example, it can be (0,4) video frames.

[0124] Based on this, in some embodiments, for video frame F t If the difference between the minimum timestamp of the video frame containing the later-appearing tracking object (i.e., the second tracking object) and the maximum timestamp of the video frame containing the earlier-appearing tracking object (i.e., the first tracking object) is greater than 0 and does not exceed 4, then it is considered that the search box may be missing in the tracking results due to the virtual keyboard. Furthermore, the virtual keyboard may exist during the time period when the tracking object is missing. Therefore, keyboard detection needs to be performed on the video frames within that time period.

[0125] In this embodiment, the conditions for keyboard detection can be expressed as follows:

[0126]

[0127] Among them, w i ,w j To represent two different tracking objects (such as a search box), MIN(T) j ) indicates w j The minimum timestamp of all video frames in the specified location, MAX(T) i ) indicates w iThe maximum value of the timestamps of all the video frames in which it appears.

[0128] In this embodiment, the first time period can be represented as [p,q], where p = MAX(T) i )+1, q=MIN(T j Based on this, keyboard detection is required for video frames with timestamps t∈[p,q].

[0129] The following section provides a detailed explanation of the content of S22, referring to S221 to S2210.

[0130] S221: For each video frame with timestamp t∈[p,q], if the number of single-character texts contained in the video frame is greater than the threshold, it is determined that the video frame includes a virtual keyboard.

[0131] It's understandable that virtual keyboards typically contain many virtual keys, and most of these keys contain only one character (such as English letters, numbers, punctuation marks, etc.). Therefore, electronic devices can use OCR technology to recognize the text contained in a video frame. In the OCR recognition result, if the number of single-character texts in the video frame exceeds a threshold (e.g., 5), then the video frame is determined to contain a virtual keyboard. Furthermore, if any video frame in the timestamp t ∈ [p, q] contains a virtual keyboard, then the target tracking in the interval [p, q] is considered to have failed due to a virtual keyboard.

[0132] For each interval [p,q] where target tracking fails due to the presence of a virtual keyboard, the electronic device determines two tracking objects that actually correspond to the same shared element from the tracking object set W through S222~S210, fills in the missing boundary between the two tracking objects, and merges the multiple tracking objects into the same tracking object to ensure that each shared element corresponds to a tracking ID.

[0133] S222: Identify the text boundary set B of search-related text within the video frames [p-3, q+3]. text .

[0134] Since [p,q] is the interval where the tracked object is missing due to the presence of a virtual keyboard, the missing boundaries within [p,q] are compensated by using a boundary of a time range larger than [p,q].

[0135] In this embodiment, the search box usually includes texts with search meanings such as the Chinese character "搜索" and the English word "search" to prompt users that it has a search function. For the sake of convenience in description, in this embodiment, texts with search meanings such as the Chinese character "搜索" and the English word "search" are referred to as search texts, and all or part of the search box texts, such as substrings of the search box texts like "search", "earch", "arch", "ear", "搜索", "搜", "索", etc., are referred to as search box related texts.

[0136] Based on this, for each interval [p, q] where target tracking fails due to the presence of a virtual keyboard, the electronic device extracts video frames with timestamps t ∈ [p - 3, q + 3], performs text recognition on each frame using OCR technology, records the boundaries of the search box related texts containing the substring "search" or "搜索", and denotes the set of these text boundaries as B text .

[0137] S223: According to the text boundary set B text , determine whether each tracking object in the tracking object set W within the first time period is basically at the same horizontal position.

[0138] In some embodiments, taking the tracking object as a search box as an example, if the difference between the maximum value and the minimum value of the central vertical axis coordinates of the search box within [p - 3, q + 3] is less than or equal to the threshold, it is determined that the search box is basically at the same horizontal position within [p - 3, q + 3]. On the contrary, if the difference between the maximum value and the minimum value of the central vertical axis coordinates of the search box within [p - 3, q + 3] is greater than the threshold, it is determined that the search box is not at the same horizontal position within [p - 3, q + 3].

[0139] Taking the threshold as 5 as an example, the condition for determining that the tracking object is basically at the same horizontal position can be expressed as:

[0140] MAX({φ y (b)|b ∈ B text}) - MIN({φ y (b)|b ∈ B text}) ≤ 5

[0141] Taking the threshold as 5 as an example, the condition for determining that the tracking object is not at the same horizontal position can be expressed as:

[0142] MAX({φ y (b)|b ∈ B text}) - MIN({φ y (b)|b ∈ B text}) > 5

[0143] If the first tracking object and the second tracking object are substantially at the same horizontal position, then steps S224-228 are executed to determine an average boundary based on the last M video frames of the first tracking object and the first N video frames of the second tracking object; and this average boundary is used as the missing boundary within the first time period. M and N can be the same or different, and this embodiment does not impose any restrictions on this.

[0144] If the first tracked object and the second tracked object are not at the same horizontal position, then execute S229, and take the last boundary of the first tracked object as the missing boundary in the first time period.

[0145] The first object to be tracked is w i The second object being tracked is w j For example, S224-228 and S229 are shown below.

[0146] S224: Extract the tracked object w i The boundaries of the last three video frames, and the tracked object w. j The boundaries of the first three video frames, denoted as B. box .

[0147] S225: Calculate B box The set of boundary center coordinates for each tracked object is denoted as S. x The set of ordinates is denoted as S. y S y ={φ y (b)|b∈B box}, calculate S x mean and standard deviation σ x Calculate S y mean and standard deviation σ y .

[0148] S226: Remove B box The boundary does not meet the requirements. For example, for any b∈B box Calculate its center coordinates (φ) x (b),φ y (b)), if b does not satisfy Let B box =B box -{b}.

[0149] S227: Remove B according to S226 box After the boundary does not meet the requirements, calculate the average boundary b. avg ,

[0150] S228: In the tracked object w i Add b to the boundary set i,p ,b i,p+1 ,…,b i,q-1 ,b i,q These q-p+1 boundary values ​​are all related to b. avg same.

[0151] That is, according to the average boundary b avg In each of the q-p+1 video frames where the tracked object is missing, a boundary is added to make up for the missing boundaries of the tracked object due to the display of the virtual keyboard, so that the tracked object is more continuous in each video frame.

[0152] S229: Record the tracking object w i The timestamp of the last frame is l = MAX(T) i ), w i The boundary of the last frame is b. i,l In tracking object w i Add b to the tracking boundary record i,p ,b i,p+1 ,…,b i,q-1 ,b i,q The values ​​of these q-p+1 boundaries are all related to b. i,l same.

[0153] That is, according to w i The boundary of the last frame is b. i,l In each of the q-p+1 missing video frames, a boundary is added to fill in the missing boundaries of the tracked object due to the display of the virtual keyboard, making the tracking information of the tracked object more continuous.

[0154] S2210: In tracking object w i In the tracking boundary record, add the tracking object w j All tracking boundary records, i.e., b i,q+1 =b j,q+1 ,b i,q+2 =b j,q+2 ... The object to be tracked is w i and tracking object w j Merge into tracking object w i To remove the tracking object w j .

[0155] It should be noted that the method provided by S22 can effectively reduce the situation where the same shared element is assigned multiple tracking IDs by the YOLOv8 model, thereby reducing the difficulty of subsequent detection of anomalies in one-shot transitions.

[0156] S23: If there are two different tracking objects in the tracking object set W, and the first tracking object disappears and the second tracking object appears after a video frame interval, then it is determined that there is a missing motion effect in the video frame.

[0157] It's important to note that the YOLOv8 model, when tracking a target object, identifies objects that appear consecutively across multiple video frames with continuous trajectories as a single tracking object. This means that if a shared element appears in two discontinuous video frames due to issues such as missing animation (i.e., a frame jump), it will be identified as a different target object by the YOLOv8 model. For example, if search box 1 appears consecutively in video frames [T1,T2] and (T2,T3), but not in the video frame corresponding to T2, then search box 1 will be identified as two different search boxes.

[0158] In other words, S23 can also be expressed as: if the first tracking object and the second tracking object appear successively in the tracking object set W, and the difference between the minimum value of the timestamp of the video frame where the second tracking object is located and the maximum value of the timestamp of the video frame where the first tracking object is located is equal to 1, then it is determined that the search transition video has missing motion effects.

[0159] In this embodiment, the first tracking object is w i The second object being tracked is w j For example, the condition in S23 that determines that the video frame has missing motion effects can be expressed as:

[0160]

[0161] Among them, w i ,w j Indicates two different tracking objects, MIN(T) j ) indicates w j The minimum timestamp of all video frames in the specified location, MAX(T) i ) indicates w i The maximum value of the timestamps of all the video frames in which it appears.

[0162] In this embodiment, the time period in which the animation is missing can be represented as [p,q], where p = MAX(T) i ), q = MIN(T j Based on this, it is determined that video frames with timestamps t∈[p,q] have missing animation effects, i.e., the page jumps by one frame, and the video frames with missing animation effects F are... t Add the set E of video frames containing anomalies. bug This process can be expressed by the following formula:

[0163] E bug =E bug ∪{Ft |t∈[p,q]}

[0164] S24: Identify video frames where the search transition is abnormal due to displacement entry.

[0165] In the transition method of displacement entry, the tracked object will gradually appear in the display interface and be displayed in full. However, this transition method is obviously not suitable for search scenarios. Therefore, it is necessary to identify the tracked object of displacement entry.

[0166] However, when identifying search boxes, the YOLOv8 model recognizes complete search boxes as tracking objects, while incomplete search boxes are not. Therefore, taking the search box as an example, text detection can be used to determine the video frame where the object is located.

[0167] In some embodiments, for any search box w i , remember w i The timestamp of the first frame is p = MIN(T) i If the electronic device detects F using OCR technology p b i,p The text txt exists within the range i,p Then for F p Text detection is performed on multiple consecutive video frames. If in Fp p If a portion of the text of the search box is detected in a previous video frame (i.e., a previous video frame where the search box was tracked), it indicates that the search box is a new search box that has been shifted in, and that video frame is added to E. bug .

[0168] For example, for any search box w i The electronic device iterates through the video frames with timestamps t∈[p-5,p-1] in reverse order and performs text detection on them. If the b of a certain video frame... i,p There is a txt file within the pixel range. i,p If a substring (i.e., a portion of the text in the search box) is found, the video frame is marked as having an abnormal search transition caused by the new search box shifting in, and the video frame is added to E. bug middle.

[0169] S25: Identify video frames where abnormal transitions occur due to displacement exit.

[0170] In the transition method of displacement exit, the tracked object will gradually disappear from the display interface. It can be seen that this transition method is obviously not suitable for search scenarios. Therefore, it is necessary to identify the tracked object that is displacement entry.

[0171] In some embodiments, for any search box w i , remember wi The timestamp of the last frame is q = MAX(T) i If F is detected by OCR technology p b 1,q The text txt exists within the range i,q Then for F p Text detection is performed on multiple consecutive video frames following the search box (i.e., multiple video frames that cannot be tracked due to the incompleteness of the search box). If in F... p b detected in multiple consecutive video frames afterwards i,p There is a txt file within the pixel range i,p If a substring (i.e., a portion of the text in the search box) is found, the video frame is marked as having an abnormal search transition caused by the search box shifting out, and the video frame is added to E. bug middle.

[0172] S3: Perform pixel-level correction on the boundaries of each tracked object in each video frame.

[0173] For example, the boundaries of each tracked object (e.g., a search box) in each video frame can be corrected at the pixel level based on HSV color representation. The specific process of correcting the boundaries at the pixel level for each tracked object in each video frame includes S31 to S36.

[0174] S31: Determine the HSV values ​​of all pixels within the boundary range of the tracked object.

[0175] That is, determine any video frame F t Any tracking object w i In b i,t The HSV values ​​of all pixels within the range.

[0176] In this embodiment, HSV includes hue (H), saturation (S), and value (V). Hue (H) represents color information, expressed in angles, and ranges from 0° to 360°. Saturation (S) ranges from 0.0 to 1.0; and value (V) ranges from 0.0 (black) to 1.0 (white).

[0177] Subsequently, the key HSV values ​​of all pixels of the tracked object are determined through S32 to S33.

[0178] S32: Determine w i Among all the HSV values ​​of all pixels, the M HSV values ​​that appear most frequently.

[0179] It's important to note that in search scenarios, the shared element being tracked is specifically the search box. The color information of the search box is usually quite simple; for example, the boundary of the search box is typically represented by one HSV value, and the interior of the search box by another. Of course, there will also be other interfering pixels with HSV values, such as noise pixels and transition pixels. Therefore, by determining the key HSV values, the influence of interfering pixels on the tracked object can be filtered out.

[0180] For example, M = 8, 10, 15, etc., but this embodiment does not limit this. For example, F can be... t Chinese w i The different HSV values ​​of all pixels are counted, and the 10 most frequent HSV values ​​are determined. Through this step, the electronic device can determine w i This mainly includes color information, so that subsequent analysis of w is possible. i The border was corrected.

[0181] S33: Determine the key HSV value from the M HSV values ​​obtained in S32.

[0182] In this embodiment, the key HSV value refers to a large number of HSV values.

[0183] In some embodiments, HSV values ​​with an occurrence rate greater than a threshold can be identified as key HSV values. It can be understood that one HSV value corresponds to one pixel, therefore the number of times an HSV value appears is equal to the number of pixels with that HSV value. Therefore, the occurrence rate of an HSV value can be determined by the ratio of the number of times an HSV value appears to the pixel area of ​​the tracked object. The pixel area of ​​the tracked object can be determined based on its boundary b. i,t To determine this, for example, S32 is traversed in descending order of frequency of occurrence to obtain M HSV values. If the ratio of the frequency of occurrence of the currently traversed HSV value to the pixel area of ​​the tracked object is less than 0.1, then the HSV value is determined as a key HSV value.

[0184] In other embodiments, S32 can be traversed in descending order of frequency of occurrence to obtain M HSV values. If the ratio of the frequency of occurrence of the currently traversed HSV value to that of the previous traversed HSV value is less than 0.5, the traversal ends, and all traversed HSV values ​​are determined as key HSV values.

[0185] Then, through steps S34 to S36, the boundary of the tracked object is calibrated based on the maximum and minimum values ​​of brightness in the key HSV values ​​of the tracked object.

[0186] S34: Determine the maximum and minimum values ​​of lightness in the key HSV values.

[0187] For example, the set of key HSV values ​​determined in S33 can be denoted as S. bg By traversing S bg Obtain the maximum value of brightness v in the HSV color representation. max and minimum value v min .

[0188] S35: Determine the mask of color information within the video frame where the tracked object is currently located, based on the maximum and minimum values ​​of brightness in the key HSV values.

[0189] For example, take the hue range [0°, 179°], saturation range [0, 5], and brightness range [v]. min ,v max Use the OpenCV library to get the current video frame F. t Mask M within the aforementioned HSV color range i,t .

[0190] S36: Based on the mask, calibrate the boundary of the tracked object in the current video frame.

[0191] For example, for M i,t Each element p in jk Take the satisfied The extreme values ​​of j and k are denoted as j min j max and k min , k max , b′ i,t = <j min ,k min ,j max ,k max >As a tracking object (such as a search box) w i In video frame F t The calibration boundary is used as the boundary b obtained in S12 to distinguish it. i,t This can be called the original boundary.

[0192] The method provided by S3 can significantly improve the pixel-level accuracy of tracking object boundaries, so as to determine the transition mode of the tracked object based on the boundary and judge whether the transition of a one-shot is smooth.

[0193] S4: Determine the time interval of the tracked object's movement based on the tracking information.

[0194] Optionally, in S4, the pwlf library can be used to analyze the boundary center coordinates of the tracked object in each video frame. Based on the horizontal / vertical coordinates of the boundary center, a linear function is fitted to the motion of the tracked object along the horizontal / vertical coordinates, thereby determining the motion time interval of the tracked object along the horizontal / vertical coordinates based on the fitted linear function.

[0195] For example, the specific implementation process of S4 includes S41 to S48.

[0196] S41: Determine each tracking object w i The set of x-axis coordinates of the boundary center i and the set of Y coordinates of the boundary center's vertical axis. i .

[0197] That is, X i ={φ x (b i,t )|t∈T i}, Y i ={φ y (b i,t )|t∈T i}

[0198] S42: If the tracking object w i If the x-coordinate remains essentially unchanged, then the tracking object w is considered to be... i It remains stationary throughout the entire horizontal axis.

[0199] For example, if MAX(X) i )-MIN(X i If )≤12, then the tracking object w is considered to be... i It remains stationary throughout the entire horizontal axis.

[0200] S43: Tracking object w i If the x-coordinate changes significantly, then let the order of the function to be fitted be 1 and the number of segments be 10. Use the pwlf library to fit the function f. i :T i →X i Perform piecewise linear fitting, where f i (t)=φ z (b i,t ).

[0201] For example, if MAX(X) i )-MIN(X i If )>12, then the tracking object w is considered to be... i The x-axis varies considerably.

[0202] It should be noted that in S42 and S43, the threshold "12" and the number of segments "10" can also be replaced with other possible values. This embodiment does not limit their specific values.

[0203] S44: Based on the fitted piecewise linear function, determine the slope and time interval of each segment of the linear function.

[0204] That is, the function f is obtained from S43. i The linear function expression for each segment, denoted as a, where the slope of the fitted linear function for the j-th segment is a. i,j The timestamp interval (i.e., the time interval) where the j-th segment is located is [t i,j ,t i,j+1 ], where j = 1, 2, ..., 10.

[0205] S45: Determine the motion state of the tracked object within each function segment based on the slope. This motion state includes both motion and stillness.

[0206] For example, for the function f in S44 i For each segment j, if its slope |a i,j If | < 0.5, then mark the tracking object w. i In the interval [t] i,j ,t i,j+1 [Still within], otherwise mark the tracking object w i In the interval [t] i,j ,t i,j+1 Internal movement.

[0207] S46: For function f i The 10 segments are divided into two parts. If the motion or stationary markers of adjacent segments are the same, their intervals are merged, and the motion state of the new interval is consistent with the motion state of the interval before the merger.

[0208] For ease of subsequent description, the interval of the j-th segment after merging is denoted as [t′]. i,j ,t′ i,j+1 ].

[0209] S47: Repeat S46 until no more intervals can be merged. Then, define the time interval with the motion state as "motion" as the motion time interval. Alternatively, the time interval with the motion state as "stationary" can also be defined as the stationary time interval.

[0210] S48: X i Replace with Y i φ x (·) is replaced with φ y (·), execute S42-47 again to obtain the tracking object w i Vertical movement within each timestamp interval.

[0211] S5: Based on the boundary of the tracked object within the motion time interval, determine the motion information of the tracked object in the first direction; and based on the motion information of the tracked object, determine the transition mode of the tracked object in the first direction.

[0212] In this embodiment, the first direction includes the horizontal axis (i.e., the horizontal direction) and the vertical axis (i.e., the vertical direction) of the pixel coordinates. The motion information includes the border length, velocity, and acceleration, etc. The transition methods include displacement entry, displacement exit, and one-shot transition, etc.

[0213] For example, the specific implementation process of S5 includes the following steps S51 to S55.

[0214] S51: Determine the lateral motion information of the tracked object within its motion time interval.

[0215] In this embodiment, for ease of description, for any tracking object w i and its j-th segment interval (see S46), X i,j ={φ x (b i,t )|t∈[t′ i,j ,t′ i,j+1 ]} represents w i The set of x-axis coordinates of the boundary centers within the j-th segment interval. Indicates w i The set of horizontal axis velocities within the j-th segment interval. Indicates w i The set of horizontal axis accelerations within the j-th segment interval, where X represents i,j The kth element, V represents i,j The kth element.

[0216] S52: Determine whether the transition method of the tracked object in the horizontal direction is displacement entry.

[0217] Specifically, within the motion time interval, if the tracked object simultaneously meets the following conditions (1) to (4), then the transverse transition mode of the tracked object is determined to be displacement entry.

[0218] Condition (1): The horizontal border length of the tracked object increases with the timestamp. That is, the parts of the tracked object are displayed gradually, or in other words, for each tracked object w... i The interval [t′] within which the movement occurs i,j ,t′ i,j+1 ], w i The horizontal length is in [t′ i,j ,t′ i,j+1The value increases as the timestamp increases within the interval.

[0219] Condition (2): The first N elements of the lateral acceleration of the tracked object are not monotonically increasing.

[0220] Condition (3): The tracked object has no acceleration process during lateral motion. For example, V i,j With a length not less than The monotonically decreasing interval ends. This is because when the displacement enters the field, the shared element goes from motion to rest, which is usually a decelerating motion.

[0221] Condition (4): The tracked object comes to a stop after decelerating. For example, in the tracked object w i The time interval of motion [t′] i,j ,t′ i,j+1 After that, there exists an interval [t′] i,j+1 ,t′ i,j+2 ], and the tracking object w i Remain still within it.

[0222] Additionally, in tracking object w i The motion time interval is [t′ i,j ,t′ i,j+1 After transitioning by displacement entry, [t′] i,j ,t′ i,j+1 Video frames F within ] t All are added to the abnormal video frame set, that is, let E bug =E bug ∪{F t |t∈[t′ i,j ,t′ i,j+1 ]}.

[0223] S53: Determine whether the transition method of the tracked object in the horizontal direction is displacement exit.

[0224] Specifically, within the motion time interval, if the tracked object simultaneously meets the following conditions (1) to (3) in the lateral direction, then the transition mode of the tracked object in the lateral direction is determined to be displacement exit.

[0225] Condition (1): The horizontal border length of the tracked object decreases as the timestamp increases. That is, the parts of the tracked object gradually disappear, or in other words, the tracked object w... i The length of [t′ i,j ,t′ i,j+1 The value decreases as the timestamp increases within the interval.

[0226] Condition (2): The last element of the lateral acceleration of the tracked object is greater than 0. That is, the velocity of the tracked object is not 0 when it disappears laterally.

[0227] Condition (3): The tracked object does not exist after the motion time interval. That is, the tracked object disappears after the motion time interval, at time t′. i,j+1 The frame is w i The last frame in which it is located, i.e., t′ i,j+1 =MAX(T) i ).

[0228] Additionally, in tracking object w i The motion time interval is [t′ i,j ,t′ i,j+1 After transitioning by displacement, [t′] i,j ,t′ i,j+1 Video frames F within ] t All were added to the abnormal video frame set, i.e., E bug =E bug ∪{F t |t∈[t′ i,j ,t′ i,j+1 ]}.

[0229] S54: Determine whether the horizontal transition of the tracked object is a single, continuous shot.

[0230] Specifically, within the motion time interval, if the tracked object simultaneously meets the following conditions (1) to (3) in the horizontal direction, then the transition mode of the tracked object in the horizontal direction is determined to be a one-shot sequence.

[0231] Condition (1): The tracked object has a stationary time interval that is located before and adjacent to the motion time interval in the horizontal direction. That is, for the motion time interval [t′ i,j ,t′ i,j+1 The tracking object w i There exists a time interval [t′ i,j-1 ,t′ i,j ], and w i It remains still within it. Or, to put it another way, the tracked object exists in a static state within the interface before it moves.

[0232] Condition (2): The tracked object has a stationary time interval that is located after and adjacent to the motion time interval in the horizontal direction. That is, for the motion time interval [t′ i,j ,t′ i,j+1 The tracking object w i There exists an interval [t′ i,j+1 ,t′ i,j+2 ], and w i It remains still within it. Or, in other words, the tracked object exists in a static state in the interface after it has moved.

[0233] Condition (3): Every video frame within the motion time interval includes the tracked object. That is, the tracked object w i Exists in the interval [t′ i,j ,t′ i,j+1 In each frame of ], {t|t′ i,j ≤t≤t′ i,j+1 ,t∈T i}={t′ i,j ,t′ i,j +1,…,t′ i,j+1 -1,t′ i,j+1}

[0234] It should be noted that if the object being tracked is w i During the motion time interval [t′ i,j ,t′ i,j+1 If the scene does not transition using displacement entry, displacement exit, or a single continuous shot, then it is assumed that the tracked object w... i During the motion time interval [t′ i,j ,t′ i,j+1 Transitions within [t′ in an undefined manner, and [t′] are used within the transition. i,j ,t′ i,j+1 All video frames within the specified range are added to the abnormal video frame set, i.e., E is set to... bug =E bug ∪{F t |t∈[t′ i,j ,t′ i,j+1 ]}:1);

[0235] S55: X i,j Replace with Y i,j ={φ y (b i,t )|t∈[t′ i,j ,t′ i,j+1 ]}, execute S51-54 again to determine the tracking object w i The longitudinal transition method within each movement time interval.

[0236] In summary, step S5 can determine the transition methods of shared elements in the transition video, such as displacement entry, displacement exit, one-shot, and undefined methods, and identify video frames with abnormal transitions.

[0237] S6: Determine whether the transition of the tracked object in a one-shot transition is smooth.

[0238] For objects tracked in a one-shot transition, there may be issues such as stuttering or inconsistent transition speed during the transition, resulting in a poor user experience or visual quality.

[0239] For example, the smoothness of the transition of the tracked object in a one-shot sequence can be detected through S61 to S66.

[0240] S61: For a tracked object that transitions in one continuous shot, fit its velocity curve along the first direction within the motion time interval.

[0241] In some embodiments, S6 uses the NumPy library to fit the velocity data in a first direction within the transition interval (i.e., the motion time interval) of the tracked object (such as a search box) in a single shot. The first direction includes both horizontal and vertical directions.

[0242] For example, taking the first direction as horizontal as an example, for the tracking object w i The time interval marked as a transition in a single shot [t′] i,j ,t′ i,j+1 (See S46), let T′ i,j ={t|t′ i,j +1≤t≤t′ i,j+1}, using the NumPy library to fit the function g i,j :T′ i,j →V i,j (V i,j The calculation formula is shown in S51. Let the fitting order start from 1 and increase by 1 each time. When the fitting order reaches 6, or the mean absolute error (MAE), the result is obtained. When the current order is determined as the optimal fitting order, then... Indicates T′ i,j The kth element.

[0243] Additionally, let V i,j The calculation formula is V i,j Indicates w i The set of ordinate velocities (Y) within the j-th segment interval i,j See S55), and repeat the above fitting process to adjust w. i One-shot transition interval [t′ i,j ,t′ i,j+1 Curve fitting was performed on the vertical axis velocity data within the range.

[0244] Subsequently, by analyzing the velocity and acceleration of the tracked object within the transition interval of a single-shot sequence through S63 to S65, the video frames with abnormally unsmooth transitions in a single-shot sequence were located.

[0245] S62: Based on the fitted curve, determine the actual velocity value and the fitted velocity value of the tracked object along the first direction in each video frame.

[0246] S63: Video frames whose difference between the actual speed value and the fitted speed value is greater than a threshold are identified as video frames with unsmooth transitions along the first direction.

[0247] For example, for any tracked object w i Any time interval of motion [t′ i,j ,t′ i,j+1 ], calculate the optimal order fit of g in S61. i,j Mean Absolute Deviation (MAE) opt , will [t′ i,j +1,t′ i,j+1 The absolute value of the difference between the actual and fitted velocity values ​​within the interval exceeds 3 times the MAE. opt Video frames that (correspond to the threshold in S63) are marked as having choppy transitions and added to the abnormal video frame set E. bug In the middle. That is to say...

[0248] S64: Within the motion time interval, determine the acceleration of the tracked object along the first direction in each video frame of the transition video.

[0249] In this embodiment, based on the acceleration along the first direction in each video frame, video frames with abnormal acceleration can be identified.

[0250] S65: For adjacent first and second video frames, if the acceleration of the first video frame is positive but the acceleration of the second video frame is negative, and the acceleration of the second video frame is greater than the acceleration threshold, it is determined that the transition of the second video frame along the first direction is not smooth, wherein the timestamp of the first video frame is less than that of the second video frame.

[0251] S65 is used to determine video frames that experience excessive acceleration or deceleration during acceleration. For example, for any tracked object w i A time interval of motion [t′ i,j ,t′ i,j+1 ], let T″ i,j ={t|t′ i,j +2≤t≤t′ i,j+1}, A i,j Indicates w i The set of accelerations within this interval will [t′ i,j +2,t′ i,j+1 Video frames with excessive acceleration within a given interval, immediately followed by an excessively large reverse acceleration, are marked as having an anomaly in the search transition and are added to the set of anomaly video frames E. bug In the middle. That is to say...

[0252] S66: If the acceleration of the first video frame is negative, but there is a second video frame with positive acceleration after the first video frame, then the second video frame is determined to be an abnormal video frame.

[0253] S66 is used to determine video frames that are decelerated and then accelerated. For example, for any tracked object w i Any one-shot transition interval [t′ i,j ,t′ i,j+1 ], will [t′ i,j +2,t′ i,j+1 Video frames with positive acceleration within the interval but which have decelerated before this video frame are marked as having unsmooth transitions and added to the abnormal video frame set E. bug In the middle. That is to say...

[0254] S67: Let V i,j The calculation formula is Indicates w i The set of ordinate velocities (Y) within the j-th segment interval i,j See S55), then execute S64-66 again to obtain the tracking object w. i During the motion time interval [t′ i,j ,t′ i,j+1 Video frames with choppy transitions along the inner vertical axis.

[0255] S7: Generate a test report.

[0256] Specifically, it generates reports on abnormal transitions between non-one-shot and non-smooth transitions between one-shot and non-smooth transitions, and extracts video frames with abnormalities from the screen recording of motion effects.

[0257] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0258] Based on the transition effect detection method provided in the above embodiments, this application also provides the following technical solutions.

[0259] Figure 5 This is a schematic diagram of a transition effect detection device provided in one embodiment of this application. The device includes a shared element tracking module 501, a tracking patch module 502, a boundary calibration module 503, a motion range positioning module 504, a transition mode judgment module 505, a curve fitting module 506, an abnormal frame screening module 507, and an abnormal reporting module 508.

[0260] The shared element tracking module 501 is used to perform shared element tracking on the transition video to be detected and to determine the tracking information of the tracking object.

[0261] Tracking patch module 502 is used to correct duplicate tracking IDs in the tracking results of shared elements.

[0262] The boundary calibration module 503 is used to perform pixel-level correction of the boundary of the tracked object based on HSV color representation.

[0263] The motion interval positioning module 504 determines the motion information of the tracked object within the motion time interval based on the tracking information of the tracked object.

[0264] The transition mode determination module 505 is used to determine the transition mode of the tracked object within the motion time interval based on the motion information of the tracked object. Possible transition modes include displacement entry, displacement exit, one-shot, and undefined mode.

[0265] The curve fitting module 506 fits the velocity curve of a tracked object that transitions in a single shot over a time interval.

[0266] The abnormal frame filtering module 507 locates video frames with unsmooth transitions in a single shot based on the fitted velocity curve of the tracked object.

[0267] The anomaly reporting module 508 generates reports on non-one-shot transition anomalies and one-shot transition non-smoothness anomalies, and extracts video frames with anomalies from the screen recording of motion effects.

[0268] This application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the transition motion effect detection method as shown in the above embodiments.

[0269] This application also provides a chip, see [link to example]. Figure 6 As shown, the chip includes a processor and a memory, in which a computer program is stored. When the computer program is executed by the processor, it implements the transition effect detection method in the above embodiments.

[0270] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the transition effect detection method provided in the above embodiments.

[0271] This application also provides a computer program product, which includes a computer program that, when run by an electronic device, enables the electronic device to implement the transition effect detection method provided in the above embodiments.

[0272] It should be understood that the processor mentioned in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0273] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0274] In the embodiments provided in this application, the division of each framework or module is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple frameworks or modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0275] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0276] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0277] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0278] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting transition animation effects, characterized in that, include: Shared element tracking is performed on the transition video to be tested to determine the tracking information of the tracking object; Based on the tracking information, the motion time interval of the tracked object is determined, as well as the motion information of the tracked object within the motion time interval, wherein the motion information includes the border length, velocity, and acceleration; Based on the motion information, determine the transition mode of the tracked object within the motion time interval.

2. The method according to claim 1, characterized in that, The process of tracking shared elements in the transition video to be detected, and determining the tracking information of the tracking object, includes: The transition video is divided into frames to obtain multiple video frames; Determine the tracking object in each video frame, and the tracking information for each tracking object; Based on the tracking information, identify abnormal tracking objects in each video frame; The tracking object set W is generated based on the remaining tracking objects in each video frame, excluding the abnormal tracking object, and the tracking object set includes at least one of the tracking objects.

3. The method according to claim 1 or 2, characterized in that, The tracking information of the tracked object includes its boundaries, type, tracking identifier ID, and confidence level.

4. The method according to claim 3, characterized in that, The shared element is a search box, and the step of identifying abnormal tracking objects in each video frame based on the tracking information includes: The tracking objects with a confidence level less than the confidence threshold are identified as the abnormal tracking objects; and / or, Identify the tracking objects whose type is not an input box as the abnormal tracking objects; and / or, The text box type is identified as the exception tracking object.

5. The method according to any one of claims 1 to 4, characterized in that, Before determining the motion time interval of the tracked object based on the tracking information, the method further includes: If there is a first tracking object and a second tracking object in the transition video, and the first tracking object disappears and the second tracking object reappears after a first time interval, then within the first time interval, the presence of a virtual keyboard in the transition video is determined based on the number of single text characters in each video frame of the transition video. If the virtual keyboard is present in the transition video during the first time period, then the missing boundary during the first time period is determined. The missing boundaries within the first time period and the boundaries of the second tracked object are merged into the first tracked object.

6. The method according to claim 5, characterized in that, If the virtual keyboard is present in the transition video during the first time period, then determining the missing boundaries within the first time period includes: If a virtual keyboard is present in the transition video during the first time period, it is determined whether the first tracking object and the second tracking object are basically at the same horizontal position. If the first tracked object and the second tracked object are basically at the same horizontal position, then the average boundary is determined based on the last M video frames of the first tracked object and the first N video frames of the second tracked object; and the average boundary is used as the missing boundary in the first time period. If the first tracked object and the second tracked object are not at the same horizontal position, then the last boundary of the first tracked object is taken as the missing boundary in the first time period.

7. The method according to any one of claims 1 to 6, characterized in that, Before determining the motion information of the tracked object based on the boundaries of the tracked object within the motion time interval, the method further includes: Determine the HSV values ​​of all pixels of the tracked object within the boundary range; Determine the key HSV values ​​for all pixels of the tracked object; The boundaries of the tracked object are calibrated based on the maximum and minimum values ​​of brightness in the key HSV values.

8. The method according to any one of claims 1 to 7, characterized in that, Determining the motion time interval of the tracked object based on the tracking information includes: A linear function is obtained by linearly fitting the coordinates of the boundary center of the tracked object in a first direction, where the first direction includes the horizontal and vertical directions. The following operations (1) to (3) are repeated until the segments of the linear function can no longer be merged, thereby obtaining at least one merged time interval; (1) Determine the slope and time interval of each function segment of the linear function; (2) Determine the motion state of the tracked object within each function segment based on the slope; wherein, the motion state includes movement or stillness, when the slope is greater than or equal to the slope threshold, the tracked object moves within the corresponding function segment; when the slope is less than the slope threshold, the tracked object remains still within the corresponding function segment; (3) Merge adjacent function segments and time intervals with the same motion state; The motion time interval is determined from the at least one merged time interval based on the slope of the linear function and the motion state of the tracked object.

9. The method according to any one of claims 1 to 8, characterized in that, Based on the motion information, determining the transition mode of the tracked object within the motion time interval includes: Within the motion time interval, the motion information of the tracked object in a first direction is determined based on the boundary of the tracked object, the first direction including the horizontal and vertical directions; Based on the motion information, the transition mode of the tracked object in the first direction is determined.

10. The method according to claim 9, characterized in that, Based on the motion information, determining the transition mode of the tracked object in the first direction includes: If the tracking object simultaneously meets the following conditions (1) to (4) within the motion time interval, then the transfer mode of the tracking object in the first direction is determined to be displacement entry. (1) The length of the border of the tracked object gradually increases in the first direction; (2) The first few elements of the acceleration of the tracked object in the first direction are not monotonically increasing; (3) The tracking object moves without acceleration in the first direction; (4) The tracked object comes to a stop after decelerating in the first direction.

11. The method according to claim 9, characterized in that, Based on the motion information, determining the transition mode of the tracked object in the first direction includes: If the tracked object simultaneously meets the following conditions (1) to (3) within the motion time interval, then the transition mode of the tracked object in the first direction is determined to be displacement exit. (1) The length of the border of the tracked object gradually decreases in the first direction; (2) The last element of the acceleration is greater than 0; (3) The tracked object does not exist in the time interval after the movement time interval.

12. The method according to claim 9, characterized in that, Based on the motion information, determining the transition mode of the tracked object in the first direction includes: If the tracking object simultaneously meets the following conditions (1) to (3) within the motion time interval, then the transition mode of the tracking object in the first direction is determined to be a one-shot sequence. (1) The tracking object has a stationary time interval in the first direction that is located before the movement time interval and adjacent to the movement time interval; (2) The tracking object has a stationary time interval in the first direction that is located after the movement time interval and adjacent to the movement time interval; (3) Each video frame within the motion time interval includes the tracked object.

13. The method according to claim 12, characterized in that, When the transition of the tracked object is a single, continuous shot, the method further includes: Within the motion time interval, the actual speed value and the fitted speed value of the tracked object along the first direction are determined in each video frame of the transition video; and video frames in which the difference between the actual speed value and the fitted speed value is greater than a threshold are determined as video frames with unsmooth transitions along the first direction. and / or, Within the motion time interval, determine the acceleration of the tracked object along the first direction in each video frame of the transition video; If the acceleration of the first video frame is positive, but there is a second video frame with negative acceleration before the first video frame, and the acceleration of the second video frame is greater than the acceleration threshold, then it is determined that the transition of the second video frame along the first direction is not smooth; wherein, the number of frames between the first video frame and the second video frame is less than the threshold. If the acceleration of the first video frame is negative, but there is a second video frame with positive acceleration after the first video frame, then the second video frame is determined to be an abnormal video frame.

14. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 13.

16. A chip, characterized in that, The chip includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 13.

Images