Detection method and device, electronic equipment, storage medium and program product

Abstract

The application discloses a detection method and device, electronic equipment, a storage medium and a program product; the application obtains a screen recording video, the screen recording video records a process of entering a video interface from a target access entrance and playing a video in the video interface; frame processing is performed on the screen recording video to obtain a video frame sequence; a target detection strategy corresponding to the target access entrance is determined from multiple detection strategies; scene switching detection processing is performed on the video frame sequence based on the target detection strategy to obtain a first scene frame and a second scene frame in the video frame sequence, the first scene frame corresponds to a time point of triggering the target access entrance, and the second scene frame corresponds to a time point of the video interface appearing a video to be played; and the video opening time consumption is determined according to the first scene frame and the second scene frame. In the application, the video opening time consumption can be accurately calculated according to the two scene frames. Therefore, the application can improve the accuracy of video opening time consumption detection.

Description

Technical Field

[0001] This application relates to the field of computers, and specifically to a detection method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid advancement of information technology, various applications are gradually permeating all aspects of people's daily lives, providing a rich variety of services. Taking video platforms as an example, they not only greatly enrich people's entertainment options but also meet the needs of different viewers by offering diverse video content. For video platforms, a crucial indicator of user experience is the waiting time from when a user accesses a video to when it begins playing. In other words, reducing this waiting time has become a key factor in improving user satisfaction and platform competitiveness.

[0003] Current technologies calculate user waiting time by detecting the real-time timing of events such as "user clicks on video" and "video starts playing." However, since event detection itself requires processing time, the detected event time often lags behind the actual event time, leading to a discrepancy between the calculated waiting time and the user's actual waiting time. This discrepancy affects the accurate assessment of user experience, hindering effective platform optimization and ultimately preventing the reduction of user waiting time, thus impacting user satisfaction. Summary of the Invention

[0004] This application provides a detection method, apparatus, electronic device, storage medium, and program product, which can improve the accuracy of video start-up time detection, help accurately evaluate user experience, further support effective platform optimization, and improve user satisfaction.

[0005] This application provides a detection method, including:

[0006] The screen recording video records the process of entering the video interface from the target access point and playing the video on the video interface.

[0007] The screen recording video is segmented into frames to obtain a video frame sequence;

[0008] Determine the target detection strategy corresponding to the target access point from multiple detection strategies;

[0009] Based on the target detection strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access entry is triggered, and the second scene frame corresponds to the time point when the video interface shows the video to be played.

[0010] The video startup time is determined based on the first scene frame and the second scene frame.

[0011] This application embodiment also provides a detection device, including:

[0012] The acquisition unit is used to acquire screen recording videos. The screen recording videos record the process of entering the video interface from the target access point and playing the video on the video interface.

[0013] The framing unit is used to process the screen-recorded video into frames to obtain a video frame sequence.

[0014] The strategy determination unit is used to determine the target detection strategy corresponding to the target access entry from multiple detection strategies;

[0015] The detection unit is used to perform scene switching detection processing on the video frame sequence based on the target detection strategy to obtain the first scene frame and the second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access entry is triggered, and the second scene frame corresponds to the time point when the video interface shows the video to be played.

[0016] The time consumption determination unit is used to determine the video start-up time based on the first scene frame and the second scene frame.

[0017] In some embodiments, the target detection strategy includes a first sub-strategy and a second sub-strategy. The first sub-strategy is used to detect the video frame corresponding to the triggering of the target access entry, and the second sub-strategy is used to detect the video frame corresponding to the appearance of the video interface when the video to be played appears.

[0018] The detection unit includes a first subunit and a second subunit:

[0019] The first subunit is used to perform scene switching detection processing on the video frame sequence based on the first sub-strategy to obtain the first scene frame in the video frame sequence;

[0020] The second subunit is used to perform scene switching detection processing on the video frame sequence based on the second sub-strategy to obtain the second scene frame in the video frame sequence.

[0021] In some embodiments, the first subunit is used for:

[0022] In response to the first access scenario, the grayscale difference between two adjacent frames is obtained based on the grayscale pixel difference between two adjacent frames in the video frame sequence.

[0023] Determine multiple image features of a specific frame in a video frame sequence between two adjacent frames, where the specific frame is one of the two adjacent frames;

[0024] The first scene frame is determined from the video frame sequence based on the difference in grayscale values ​​between two adjacent frames and multiple image features of a specific frame in the two adjacent frames.

[0025] In some embodiments, based on the first sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain the first scene frame in the video frame sequence, which further includes:

[0026] In response to the second access scenario, the first mean difference is determined based on the mean grayscale image of two adjacent frames in the video frame sequence;

[0027] The first scene frame is determined from the video frame sequence based on the first mean difference, the grayscale difference between two adjacent frames, and multiple image features of a specific frame in two adjacent frames.

[0028] In some embodiments, the second subunit is used for:

[0029] Based on the second sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain the second scene frame in the video frame sequence, including:

[0030] The second mean difference is determined based on the mean grayscale image of two adjacent frames in the remaining frame sequence. The remaining frame sequence includes frames in the video frame sequence that are located after the first scene frame.

[0031] The second scene frame is determined from the remaining frame sequence based on the mean grayscale image of two adjacent frames and the difference between the second mean.

[0032] In some embodiments, the first subunit is used for:

[0033] In response to a third access scenario, the difference in grayscale values ​​between two adjacent frames in the first preset region is determined based on the difference in grayscale pixels between two adjacent frames in the video frame sequence in the first preset region.

[0034] Based on the difference in grayscale values ​​between two adjacent frames in the first preset region, multiple feature points of two adjacent frames in the second preset region are paired to obtain multiple feature point matching pairs between two adjacent frames.

[0035] The degree of inter-frame motion between two adjacent frames is determined based on the position information of multiple feature point matching pairs between two adjacent frames.

[0036] The first scene frame is determined from the video frame sequence based on the degree of motion between frames.

[0037] In some embodiments, pairing multiple feature points of two adjacent frames in a second preset region to obtain multiple feature point matching pairs of two adjacent frames includes:

[0038] The additional visual element regions in two adjacent frames are masked to obtain the updated two adjacent frames.

[0039] Multiple feature points of two adjacent frames are paired after the update to obtain multiple feature point matching pairs of two adjacent frames.

[0040] In some embodiments, determining the inter-frame motion degree between two adjacent frames based on the position information of multiple feature point matching pairs between two adjacent frames includes:

[0041] Based on the position information of multiple feature point matching pairs in two adjacent frames, determine the vertical displacement component of the second frame relative to the first frame in the two adjacent frames.

[0042] The degree of motion between adjacent frames is determined based on the position component in the vertical direction.

[0043] In some embodiments, determining the inter-frame motion degree between two adjacent frames based on the position information of multiple feature point matching pairs between two adjacent frames includes:

[0044] Based on the position information of the feature point matching pairs in two adjacent frames, determine the displacement distance of the feature point matching pairs in multiple directions;

[0045] Based on the displacement distance of the feature point matching pair in multiple directions and the preset displacement distance in multiple directions, the target feature point matching pair is determined from multiple feature point matching pairs in two adjacent frames.

[0046] Determine the proportion of target feature point matching pairs in multiple feature point matching pairs in two adjacent frames;

[0047] Based on the proportion information, determine the degree of inter-frame movement between two adjacent frames.

[0048] In some embodiments, determining a first scene frame from a video frame sequence based on the degree of inter-frame motion includes:

[0049] When the inter-frame motion between two adjacent frames is within the range of weak inter-frame motion, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames. The second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0050] Based on the difference in grayscale values ​​between two adjacent frames in the first preset region, multiple feature points of the two adjacent frames in the second preset region are paired to obtain multiple feature point matching pairs of the two adjacent frames.

[0051] Based on the position information of multiple feature point matching pairs in the next two adjacent frames, the inter-frame motion degree of the next two adjacent frames is determined.

[0052] When the inter-frame motion of the next two adjacent frames falls within the range of weak inter-frame motion, the first frame of the two adjacent frames is determined as the first scene frame.

[0053] In some embodiments, the second subunit is used for:

[0054] In response to the third access scenario, if the number of edge pixels in the second frame of two adjacent frames in the video frame sequence is less than the preset number of pixels, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames, and the second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0055] Based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0056] When the contour information of the second frame in the next two adjacent frames meets the preset contour conditions, the second frame in the next two adjacent frames is determined as the second scene frame.

[0057] In some embodiments, based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain contour information of the second frame of the next two adjacent frames, including:

[0058] Based on multiple image features of the next two adjacent frames, the target feature conditions are determined from multiple preset image feature conditions;

[0059] The scores of the preset conditions corresponding to the target feature conditions are statistically analyzed to obtain the total target score;

[0060] Based on the target total score, the number of edge pixels in the second frame of the next two adjacent frames, and the difference in grayscale values ​​between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0061] In some embodiments, the second subunit is used for:

[0062] In response to the third access scenario, if the number of edge pixels of the second frame of two adjacent frames in the video frame sequence is not less than the preset number of pixels, and the number of edge pixels of the second frame in the content entry area of ​​two adjacent frames is less than the preset number of edge pixels, then a number of consecutive video frames after the second frame in two adjacent frames are determined from the video frame sequence. The content entry area refers to the starting area in the video interface where the content to be displayed begins to replace the currently displayed content.

[0063] Content recognition processing is performed on multiple consecutive video frames to obtain the video content;

[0064] Based on the video content, the second frame in a pair of adjacent frames is determined as the second scene frame.

[0065] In some embodiments, determining the second frame of two adjacent frames as the second scene frame based on video content includes:

[0066] When the video content does not belong to the preset cover image, the second frame of the two adjacent frames is determined as the second scene frame;

[0067] When the video content belongs to the preset cover image, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames. The second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0068] Based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0069] When the contour information of the second frame in the next two adjacent frames meets the preset contour conditions, the second frame in the next two adjacent frames is determined as the second scene frame.

[0070] This application also provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any of the detection methods provided in this application.

[0071] This application also provides a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute steps in any of the detection methods provided in this application.

[0072] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in any of the detection methods provided in this application.

[0073] This application embodiment can acquire screen recording video, which records the process of entering the video interface from the target access point and playing the video on the video interface; perform frame segmentation processing on the screen recording video to obtain a video frame sequence; determine the target detection strategy corresponding to the target access point from multiple detection strategies; perform scene switching detection processing on the video frame sequence based on the target detection strategy to obtain the first scene frame and the second scene frame in the video frame sequence, where the first scene frame corresponds to the time point when the target access point is triggered and the second scene frame corresponds to the time point when the video to be played appears on the video interface; and determine the video start time based on the first scene frame and the second scene frame.

[0074] In this application, the acquired screen recording video can be segmented into frames to obtain a video frame sequence. Since the screen recording video documents the process of entering the video interface from the target access point and playing the video on the video interface, a target detection strategy can be determined from multiple detection strategies based on this target access point. Thus, based on this target detection strategy, scene switching detection processing can be performed on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence. Because the first scene frame corresponds to the time point when the target access point is triggered, and the second scene frame corresponds to the time point when the video to be played appears on the video interface, the video startup time can be accurately calculated based on these two scene frames. This method improves the accuracy of video startup time detection, helps to accurately assess user experience, further supports effective platform optimization, helps reduce video startup time, improves video startup efficiency, and thus enhances user satisfaction. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1a This is a schematic diagram of a scenario for the detection method provided in the embodiments of this application;

[0077] Figure 1b This is a schematic flowchart of the detection method provided in the embodiments of this application;

[0078] Figure 2a This is a schematic diagram illustrating the application of the detection method provided in this application embodiment in a video platform;

[0079] Figure 2b This is a schematic diagram of switching live broadcast screens provided in an embodiment of this application;

[0080] Figure 2c This is a schematic diagram illustrating a scenario where the detection video takes a long time to start, as provided in an embodiment of this application.

[0081] Figure 2d This is a schematic diagram illustrating a scenario where a live video platform detects the time it takes to start a video, as provided in an embodiment of this application.

[0082] Figure 2e This is a flowchart of the detection process for the first scene frame in the first access scenario provided in the embodiments of this application;

[0083] Figure 2f This is a flowchart of the detection process for the second scene frame in the first access scenario provided in the embodiments of this application;

[0084] Figure 2g This is a flowchart of the detection process for the first scene frame in the third access scenario provided in the embodiments of this application;

[0085] Figure 2h This is a flowchart of the detection process for the second scene frame in the third access scenario provided in the embodiments of this application;

[0086] Figure 3 This is a schematic diagram of the detection device provided in the embodiments of this application;

[0087] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0088] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0089] This application provides a detection method, apparatus, electronic device, storage medium, and program product.

[0090] Specifically, the detection device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0091] In some embodiments, the detection device may also be integrated into multiple electronic devices, such as multiple servers, with the detection method of this application being implemented by multiple servers.

[0092] In some embodiments, the terminal can also be used as a server to implement some or all of the functions of a server.

[0093] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0094] It is understood that in the specific implementation of this application, screen recording videos and other related data are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0095] It is understood that the terms “first,” “second,” etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another.

[0096] For example, refer to Figure 1a The electronic device can acquire screen recording video, which records the process of entering the video interface from the target access point and playing the video on the video interface; the screen recording video is processed into frames to obtain a video frame sequence; the target detection strategy corresponding to the target access point is determined from multiple detection strategies; based on the target detection strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access point is triggered, and the second scene frame corresponds to the time point when the video to be played appears on the video interface; the video start time is determined based on the first scene frame and the second scene frame.

[0097] This application allows for frame-by-frame processing of the acquired screen recording video to obtain a video frame sequence. Since the screen recording video documents the process of entering the video interface from the target access point and playing the video, a target detection strategy can be determined from multiple detection strategies based on this target access point. Thus, based on this target detection strategy, scene switching detection processing can be performed on the video frame sequence to obtain the first scene frame and the second scene frame. Because the first scene frame corresponds to the time point when the target access point is triggered, and the second scene frame corresponds to the time point when the video to be played appears on the video interface, the video startup time can be accurately calculated based on these two scene frames. This method improves the accuracy of video startup time detection, helps to accurately assess user experience, further supports effective platform optimization, reduces video startup time, improves video startup efficiency, and thus enhances user satisfaction.

[0098] Furthermore, current technologies often require real-time detection of the events "user clicks on video" and "video starts playing," which consumes computational resources and may negatively impact video playback smoothness. In contrast, this application first acquires a screen recording video, then performs frame-by-frame processing to obtain a video frame sequence, and further detects the first and second scene frames needed to calculate the video start time from this sequence. Since this method does not rely on real-time detection, it does not consume additional computational resources during video playback, thus ensuring smooth playback and providing users with a superior viewing experience.

[0099] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0100] In this embodiment, a detection method is provided, such as Figure 1b As shown, the specific process of this detection method can be as follows:

[0101] 101. Obtain the screen recording video. The screen recording video records the process of entering the video interface from the target access point and playing the video on the video interface.

[0102] Screen recording videos are video files generated by capturing and recording the content displayed on the screen of an electronic device using screen recording tools.

[0103] In some embodiments, screen recording video can be a video file generated by a screen recording tool that starts when a preset trigger condition (such as accessing a target access point) is detected, capturing and recording the content displayed on the screen of an electronic device.

[0104] In some embodiments, screen recording videos can also be generated by directly launching a screen recording tool to capture and record the content displayed on the screen of an electronic device, resulting in a video file.

[0105] In some embodiments, the screen recording video records multiple access processes, which refer to the process of entering the video interface from the target access point and playing the video on the video interface.

[0106] For example, a screen recording documentes the process of a user entering video interface 1 from target access point 1 and playing a video on video interface 1. Afterwards, the user exits this process and enters video interface 2 from target access point 2, playing a video on video interface 2. It can be considered that the screen recording contains multiple independent access processes, and the start and end of each access process are recorded and detected separately.

[0107] The target entry point refers to the point where the user enters the video interface and triggers video playback. For example, the target entry point can be a Uniform Resource Locator (URL) (such as a video link, web address, etc.), a video cover on a content aggregation page, a referral video, or a video containing a video recommendation entry point.

[0108] The content aggregation page is a page that gathers multiple video thumbnails, containing various video resources. On this page, some video thumbnails are displayed in a fixed position, while others are displayed dynamically. Specifically, certain video thumbnails remain in the core area of ​​the content aggregation page and can be scrolled based on user actions. Furthermore, the position of the video thumbnails can also change as the page is scrolled.

[0109] The referral video can guide viewers to the next video interface. For example, by clicking, using voice commands, or other triggering methods, viewers can enter the video interface of the next video (such as a short video, long video, or live video). The specific operation to "enter" the next video can be swiping down, swiping up, swiping left, swiping right, clicking, autoplay, or gestures.

[0110] For example, users can access live videos from short videos by swiping down, swiping up, swiping left, swiping right, clicking, autoplaying, or using gestures.

[0111] Videos that include video recommendation links can be those that display related recommended videos during playback. Alternatively, based on a recommendation mechanism, users can navigate from the current video to the next video using actions such as swiping down, swiping up, swiping left, swiping right, clicking, autoplay, or gestures. The current video can be considered a video containing video recommendation links. Especially in live streaming scenarios, this recommendation mechanism can seamlessly guide users to the next live stream channel or replay content that they might be interested in, thus providing a coherent and personalized viewing experience.

[0112] The video interface is where the video content is displayed. It typically includes various control elements such as play / pause buttons, volume adjustment, progress bars, and full-screen transitions. It's the area where users interact with the video, displaying not only the video itself but also video-related functions. On streaming platforms, the video interface may also feature comment sections, likes, favorites, and tipping functions, further enhancing the user experience.

[0113] In some embodiments, the video interface may be an interface for displaying video content such as live streams, short videos, or long videos.

[0114] In some embodiments, screen recording videos can be obtained locally or remotely.

[0115] For example, acquiring screen recordings locally can refer to detecting the time it takes for a video to start on a local device. Acquiring screen recordings remotely, on the other hand, refers to a server collecting screen recordings from multiple electronic devices and analyzing the time it takes for the videos to start.

[0116] 102. Perform frame segmentation on the screen recording video to obtain a video frame sequence.

[0117] A video frame sequence refers to a collection of consecutive frames that divide a screen-recorded video into chronological order. Each frame represents the image content of the screen-recorded video at a specific moment, and all frames are arranged in chronological order to form a sequence. By performing frame segmentation processing on the screen-recorded video, it can be transformed into a series of static image frames, which, when combined, display the dynamic changes of the screen-recorded video.

[0118] In some embodiments, to extract the video frame sequence of a screen-recorded video, the screen-recorded video is segmented into frames to obtain the video frame sequence, including:

[0119] A frame-by-frame parsing strategy is used to perform temporal decomposition on the screen recording video to obtain a continuous sequence of video frames. Each video frame is a static image snapshot of the screen recording video at a specific point in time.

[0120] Frame-by-frame parsing is a strategy used to decompose video and extract each frame of image. For example, frame-by-frame parsing strategies can be multimedia processing tools, open-source computer vision libraries, or custom video parsers. When the frame-by-frame parsing strategy is a multimedia processing tool, a decoder achieves frame-level video parsing. When the frame-by-frame parsing strategy is an open-source computer vision library, a frame extraction interface is provided to capture images frame by frame. When the frame-by-frame parsing strategy is a custom video parser, video frame data is obtained through demultiplexing and video decoding processes.

[0121] 103. Determine the target detection strategy corresponding to the target access point from multiple detection strategies.

[0122] The detection strategy is used to detect video frames in the screen recording that are related to scene changes, including scene changes when entering the video interface from the access point and scene changes when playing the video on the video interface.

[0123] The target detection strategy is the detection strategy used to detect video frames in the screen recording video that are related to scene changes, when the screen recording video is related to the target access point.

[0124] In some embodiments, different detection strategies may correspond to different access points.

[0125] For example, the detection strategies corresponding to multiple access points may include the detection strategy corresponding to the URL address, the detection strategy corresponding to the video cover in the content aggregation page, the detection strategy corresponding to the traffic-driving video, and the detection strategy corresponding to the video containing the video recommendation entry.

[0126] In some embodiments, the same detection strategy may correspond to multiple access points.

[0127] For example, screen recording can capture the process of entering the video interface displaying live content from the target access point and playing the live content on the video interface. This can be done through URL addresses, video covers in content aggregation pages, referral videos, videos containing video recommendation entry points, etc., and then playing the video on the video interface.

[0128] Video thumbnails are closely linked to specific URLs; clicking a thumbnail triggers the corresponding URL, leading to the video. However, video thumbnails typically appear on content aggregation pages, and these pages may contain multiple thumbnails, making the context of a video thumbnail more complex and diverse than that of a single URL. Nevertheless, video thumbnails and URLs share a similar function in guiding users to specific content. Therefore, when designing video thumbnail detection strategies, existing detection strategies for URL-based access points can be referenced and integrated. This approach not only simplifies the development process but also improves the uniformity and efficiency of managing different types of access points.

[0129] Since both entering the next video interface from the referral video and entering the next video interface from the video recommendation entry point involve accessing the next video through the same video, the scenarios are similar, so the same detection strategy can be used for these two types of access entry points.

[0130] 104. Based on the target detection strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access entry is triggered, and the second scene frame corresponds to the time point when the video to be played appears on the video interface.

[0131] The first scene frame is the video frame that is triggered when the target access point is accessed.

[0132] For example, the first scene frame could be the video frame corresponding to clicking a URL address, the video frame corresponding to clicking a video cover on a content aggregation page, the video frame corresponding to entering the next video interface from the referral video through clicks, voice commands, or other triggering methods, the video frame corresponding to entering the next video interface from a video containing a video recommendation entry, and so on.

[0133] A video waiting to be played is a video that is about to be played on the video interface. For example, a video waiting to be played could be a video that is loading and ready to play, or a video that has finished loading but has not yet had its play button clicked, and so on.

[0134] The second scene frame is the video frame that appears on the video interface when the video to be played appears. For example, the second scene frame could be the first frame when the video content appears on the video interface, or it could be the first frame after the video has finished loading on the video interface but before the play button has been clicked, and so on.

[0135] Video content refers to the content displayed on the video interface. Video content is not limited to the actual video being played, but may also include related additional visual elements (such as comment sections, bullet comments, and control controls). For example, video content can be live stream content, long video content, or short video content.

[0136] For example, when the video content is live, if the host only engages in voice chat and doesn't share their screen, the live stream might only include the user's avatar and the interactive area. If the host enables screen sharing, the live stream will include both the host's screen and the shared screen content. When the video content is either long or short, it typically consists of an intro and the actual video playback. The intro may include advertisements, opening animations, or recommendations for other videos.

[0137] In some embodiments, in order to accurately detect the first scene frame and the second scene frame, the target detection strategy includes a first sub-strategy and a second sub-strategy. The first sub-strategy is used to detect the video frame corresponding to the triggering of the target access entry, and the second sub-strategy is used to detect the video frame corresponding to the appearance of the video to be played on the video interface.

[0138] Based on the object detection strategy, scene transition detection processing is performed on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence, including:

[0139] Based on the first sub-strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame in the video frame sequence.

[0140] Based on the second sub-strategy, scene switching detection processing is performed on the video frame sequence to obtain the second scene frame in the video frame sequence.

[0141] The first sub-strategy is used to detect the video frame that triggers the target access entry from the video frame sequence.

[0142] The second sub-strategy is used to detect the video frame corresponding to the appearance of the video to be played in the video interface from the video frame sequence.

[0143] In some embodiments, when the target access point is a URL address, in order to accurately detect the first frame when entering the video interface from the URL address, scene switching detection processing is performed on the video frame sequence based on the first sub-strategy to obtain the first scene frame in the video frame sequence, including:

[0144] In response to the first access scenario, the grayscale difference between two adjacent frames is obtained based on the grayscale pixel difference between two adjacent frames in the video frame sequence.

[0145] Determine multiple image features of a specific frame in a video frame sequence between two adjacent frames, where the specific frame is one of the two adjacent frames;

[0146] The first scene frame is determined from the video frame sequence based on the difference in grayscale values ​​between two adjacent frames and multiple image features of a specific frame in the two adjacent frames.

[0147] The first access scenario is accessing the video interface through a URL address. The URL address can be a URL address shared in a dialog box, or a URL address obtained through social media, email, web links, or other channels. These links will all guide users to the video interface.

[0148] Two adjacent frames refer to two consecutive frames in a video frame sequence that appear in chronological order.

[0149] For example, in a video frame sequence {f1, f2, ... f... n}, (f1, f2), (f2, f3)…(f n-1 f n Both belong to two adjacent frames.

[0150] Gray-scale pixel difference refers to the difference in grayscale images between two adjacent frames, specifically, the first frame and the second frame. This difference is calculated by comparing the grayscale values ​​of each pixel in the first and second frames. In chronological order, the first frame appears before the second frame, so their grayscale pixel difference reflects the changes between these two frames.

[0151] The difference in grayscale values ​​can reflect the degree of change between two adjacent frames. For example, based on the difference in grayscale values, it can be determined whether the change between two adjacent frames is a "strong change" or a "weak change".

[0152] A specific frame is a frame among two adjacent frames used to provide a reference for changes. It is usually used as a comparison benchmark to analyze changes or differences between two adjacent frames. For example, a specific frame can be the first frame among two adjacent frames, or it can be the second frame among two adjacent frames.

[0153] Image features are the visual attributes of a specific frame among two adjacent frames. These attributes provide important information for quantifying the differences between frames when analyzing and comparing changes between adjacent frames. For example, multiple image features of a specific frame include the grayscale mean, the number of edge pixels, and the amount of text.

[0154] The grayscale mean of a specific frame refers to the average grayscale value of all pixels in that frame, and can reveal whether the frame contains image content.

[0155] The number of edge pixels in a specific frame refers to the total number of edge pixels identified in that specific frame.

[0156] The number of texts in a specific frame is used to measure whether there is text information in that frame, thus helping to determine whether an image contains text content.

[0157] The first scene frame is the first frame when you enter the video interface from the URL address.

[0158] For example, when a specific frame is the first frame of two adjacent frames, if the difference in grayscale values ​​between the two adjacent frames reflects a "strong change" between the two adjacent frames, and the average grayscale value, the number of edge pixels, and the number of text in the specific frame indicate that there is image content in the specific frame, then the first frame of the two adjacent frames can be taken as the first scene frame.

[0159] When a specific frame is the second frame of two adjacent frames, if the difference in grayscale values ​​between the two adjacent frames reflects a "strong change" between the two adjacent frames, and the average grayscale image value, number of edge pixels, and number of text in the specific frame indicate that there is no image content in the specific frame, or that the image content has not been fully loaded, then the first frame of the two adjacent frames can be taken as the first scene frame.

[0160] In some embodiments, in order to detect significant changes in the content of two adjacent image frames for further detection of the first scene frame, the grayscale difference between two adjacent frames is obtained based on the grayscale pixel difference between them in the video frame sequence, including:

[0161] Based on the grayscale pixel difference between two adjacent frames in the video frame sequence, determine the grayscale pixel difference that is greater than the first preset pixel difference.

[0162] The grayscale difference between two adjacent frames is determined based on the grayscale pixel difference greater than the first preset pixel difference and the grayscale pixel difference between two adjacent frames.

[0163] The first preset pixel difference is a pixel difference threshold set in advance in the first access scenario, which is used to measure the grayscale difference between pixels between two adjacent frames.

[0164] A grayscale pixel difference greater than the first preset pixel difference refers to a grayscale difference between two adjacent frames that exceeds the first preset pixel difference.

[0165] The grayscale difference between two adjacent frames is equal to the proportion of grayscale pixel differences greater than the first preset pixel difference among all grayscale pixel differences between two adjacent frames.

[0166] For example, calculate the grayscale pixel difference of each pixel between two adjacent frames. When the first preset pixel difference is 25, filter out grayscale pixel differences greater than 25 from the grayscale pixel differences between the pixels between two adjacent frames, and determine the proportion of the number of grayscale pixel differences greater than 25 in all grayscale pixel differences between two adjacent frames. When this proportion is greater than the preset proportion, it can reflect that there is a "strong change" between two adjacent frames. Here, the range of the first preset pixel difference is not limited.

[0167] In some embodiments, considering that a content aggregation page typically includes multiple video covers, and some video covers can move within the content aggregation page, and since screen recording can record the process of entering the video interface through the video cover and playing the video on the video interface, to avoid interference from the movement of the video cover on the first scene frame detection, based on the first sub-strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame in the video frame sequence, further comprising:

[0168] In response to the second access scenario, the first mean difference is determined based on the mean grayscale image of two adjacent frames in the video frame sequence;

[0169] The first scene frame is determined from the video frame sequence based on the first mean difference, the grayscale difference between two adjacent frames, and multiple image features of a specific frame in two adjacent frames.

[0170] The second access scenario refers to the scenario where one enters the video interface from the video cover in the content aggregation page.

[0171] The grayscale image mean of two adjacent frames includes the grayscale image mean of the first frame and the grayscale image mean of the second frame. The grayscale image mean of the first frame is the mean obtained by dividing the total grayscale value of all pixels in the first frame by the total number of pixels in the first frame. The grayscale image mean of the second frame is the mean obtained by dividing the total grayscale value of all pixels in the second frame by the total number of pixels in the second frame.

[0172] The first mean difference is equal to the difference between the mean grayscale image of the first frame and the mean grayscale image of the second frame in a video frame sequence.

[0173] For example, when the first mean difference is less than the first preset mean difference, it means that the two adjacent frames only show a change in the video cover moving within the content aggregation page, rather than a scene where the video cover moves from the content aggregation page into the video interface.

[0174] The first preset mean difference is set in advance in the second access scenario and is used to measure the difference between the mean grayscale images of two adjacent frames in the video frame sequence.

[0175] The first scene frame is the first frame when entering the video interface from the video cover on the content aggregation page.

[0176] In some embodiments, when the screen recording video captures the process of entering the video interface from a URL address or a video cover in a content aggregation page, and playing the video in the video interface, in order to accurately detect the second scene frame in these two processes, scene switching detection processing is performed on the video frame sequence based on the second sub-strategy to obtain the second scene frame in the video frame sequence, including:

[0177] The second mean difference is determined based on the mean grayscale image of two adjacent frames in the remaining frame sequence. The remaining frame sequence includes frames in the video frame sequence that are located after the first scene frame.

[0178] The second scene frame is determined from the remaining frame sequence based on the mean grayscale image of two adjacent frames and the difference between the second mean.

[0179] The remaining frame sequence includes frames in the video frame sequence that are located after the first scene frame.

[0180] For example, the video frame sequence is {f1, f2, ... f... n When the first scene frame is f2, the remaining frame sequence is {f3, ... f2}. n}

[0181] The second mean difference is equal to the difference between the mean grayscale image of the first frame and the mean grayscale image of the second frame in the remaining frame sequence.

[0182] The second scene frame is the video frame corresponding to the video to be played when the video interface appears in the first or second access scene.

[0183] For example, when the average grayscale value of the second frame of two adjacent frames in the remaining frame sequence is greater than the preset average grayscale value, the average grayscale value of the first frame is less than the preset average grayscale value, and the difference between the two average values ​​is greater than the difference between the two preset average values, the second frame of two adjacent frames in the remaining frame sequence is taken as the second scene frame.

[0184] The preset grayscale image mean is a threshold that is pre-set to measure the grayscale image mean in either the first or second access scenario.

[0185] The second preset mean difference is used to measure the difference between the mean grayscale images of two adjacent frames in the remaining frame sequence, under the first or second access scenario.

[0186] For example, when the preset grayscale image mean is 30 and the second preset mean difference is 10, if the grayscale image mean of the second frame of two adjacent frames in the remaining frame sequence is greater than 30, the grayscale image mean of the first frame is less than 30, and the second mean difference is greater than 10, then the second frame of two adjacent frames in the remaining frame sequence is taken as the second scene frame. Here, the range of the preset grayscale image mean and the second preset mean difference is not limited.

[0187] In some embodiments, considering that screen recording videos can record the process of transitioning from a referral video or a video containing a video recommendation entry point to the next video interface, and then playing the video on the next video interface, in order to accurately detect the first scene frame in these two processes, scene switching detection processing is performed on the video frame sequence based on the first sub-strategy to obtain the first scene frame in the video frame sequence, including:

[0188] In response to a third access scenario, the difference in grayscale values ​​between two adjacent frames in the first preset region is determined based on the difference in grayscale pixels between two adjacent frames in the video frame sequence in the first preset region.

[0189] Based on the difference in grayscale values ​​between two adjacent frames in the first preset region, multiple feature points of two adjacent frames in the second preset region are paired to obtain multiple feature point matching pairs between two adjacent frames.

[0190] The degree of inter-frame motion between two adjacent frames is determined based on the position information of multiple feature point matching pairs between two adjacent frames.

[0191] The first scene frame is determined from the video frame sequence based on the degree of motion between frames.

[0192] The third access scenario is the scenario where you enter the next video interface from a referral video or a video containing a video recommendation entry.

[0193] The first preset area is a specific area set in advance to eliminate the false impression of "entering the next video interface". For example, the first preset area could be the identification area of ​​the video publishing object, the video watermark area, the channel identifier area, or some fixed-position control areas. These controls typically do not move relative to each other during video playback. For example, play buttons, pause buttons, and other control elements usually remain in fixed positions and do not change with the scrolling of screen content.

[0194] The difference in grayscale value between two adjacent frames in the first preset area is the difference in grayscale pixel value between two adjacent frames in the first preset area, which can reflect the degree of change between two adjacent frames in the first preset area.

[0195] For example, when navigating from a referral video to the next video screen, or from a video recommendation entry point, the content of the current video (such as a referral video or a video containing a video recommendation entry point) itself contains a visual effect similar to "slide to the next video." To accurately determine whether the user has truly entered the next video screen, analysis can be performed based on the difference in grayscale pixels between two adjacent frames in the video frame sequence within the identifier area of ​​the video publishing object. When the difference in grayscale pixels between two adjacent frames in the identifier area of ​​the video publishing object is greater than a preset difference in grayscale pixels, it indicates that the user has truly entered the next video screen.

[0196] The preset grayscale difference is a pre-set difference threshold used to measure the grayscale difference between two adjacent frames in the first preset region.

[0197] The second preset area refers to the area other than the area containing additional visual elements in a traffic-driving video or a video containing a video recommendation entry.

[0198] The additional visual element area refers to the area occupied by various visual elements displayed on top of the video content. These areas include, but are not limited to, the comment section, the bullet screen area, and the interactive effects area (such as reward effects, effects triggered by the streamer's actions, etc.).

[0199] The tipping effects area refers to the area on the screen where special effects animations are displayed when viewers tip during a live stream.

[0200] The special effects area triggered by the streamer's behavior refers to the area where visual effects are triggered by specific actions of the streamer (such as initiating an interaction or making a certain action).

[0201] Feature points are pixels that exhibit stable and repeatable localization in a second preset region between two adjacent frames. For example, feature points can be edge pixels, corner pixels, etc., between two adjacent frames in the second preset region.

[0202] A feature point matching pair is a set of corresponding feature points between two adjacent frames, and the feature point matching pair exhibits similarity between the two adjacent frames. A feature point matching pair includes feature point 1 and feature point 2, where feature point 1 is located in the first frame of the two adjacent frames, and feature point 2 is a feature point in the second frame of the two adjacent frames that matches feature point 1.

[0203] Location information refers to the position of the feature points in a feature point matching pair within their respective frames.

[0204] For example, the location information includes the position of feature point 1 in the first frame of two adjacent frames, and the position of feature point 2 in the second frame of two adjacent frames.

[0205] The degree of inter-frame motion between two adjacent frames is quantified based on the positional information of multiple feature point matching pairs in the two adjacent frames, reflecting the relative movement that occurs between the two frames. According to the magnitude of the relative movement, the degree of inter-frame motion can be classified into strong inter-frame motion and weak inter-frame motion.

[0206] The first scene frame is the first frame when the user enters the next video interface from the referral video or the video containing the video recommendation entry.

[0207] For example, based on the positional information of multiple feature point matching pairs between two adjacent frames, the inter-frame motion degree between the two adjacent frames, calculated using various methods, can be determined. Based on the inter-frame motion degree calculated by these methods, the first scene frame can be identified from the video frame sequence. When a strong inter-frame motion degree is found among the inter-frame motion degrees calculated by these methods, the first frame among the two adjacent frames is taken as the first scene frame. These multiple calculation methods can include affine transformation matrix analysis, feature point distance difference, etc.

[0208] When the inter-frame motion level calculated based on the affine transformation matrix is ​​strong, while the inter-frame motion level calculated based on the feature point distance difference is weak, or when the inter-frame motion level calculated based on the affine transformation matrix is ​​weak, while the inter-frame motion level calculated based on the feature point distance difference is strong, if it is determined that a scene occurs in two adjacent frames where the user enters the next video interface from the traffic-driving video or a video containing a video recommendation entry, then the first frame in the two adjacent frames is taken as the first scene frame.

[0209] When the inter-frame motion level calculated based on the affine transformation matrix is ​​weak, and the inter-frame motion level calculated based on the feature point distance difference is also weak, it is determined that there is no scene in the two adjacent frames where the video enters the next video interface from the traffic-driving video or the video containing the video recommendation entry.

[0210] Alternatively, the inter-frame motion level of two adjacent frames can be determined based on the positional information of multiple feature point matching pairs between adjacent frames, calculated by the target calculation method. When the inter-frame motion level calculated by the target calculation method is strong, it is determined that a scene occurs where the video leads from a referral video or a video containing a video recommendation entry point to the next video interface, and the first frame of the two adjacent frames is taken as the first scene frame. When the inter-frame motion level calculated by the target calculation method is weak, it is determined that a scene does not occur where the video leads from a referral video or a video containing a video recommendation entry point to the next video interface. The target calculation method can be one of affine transformation matrix analysis, feature point distance difference, etc.

[0211] In some embodiments, to eliminate video sliding artifacts that may be caused by scrolling in the comment area, interactive effects, and changes in the video shooting perspective, and to ensure accurate detection of the first scene frame, the grayscale difference between two adjacent frames in the first preset region is determined based on the grayscale pixel difference between two adjacent frames in the video frame sequence, including:

[0212] Based on the grayscale pixel difference between two adjacent frames in the video frame sequence in the first preset region, determine the grayscale pixel difference that is greater than the second preset pixel difference.

[0213] The grayscale difference between two adjacent frames in the first preset region is determined based on the grayscale pixel difference greater than the second preset pixel difference and the grayscale pixel difference between two adjacent frames in the first preset region.

[0214] The second preset pixel difference is a pixel difference threshold set in advance in the third access scenario, which is used to measure the grayscale difference between pixels between two adjacent frames.

[0215] A grayscale pixel difference greater than the second preset pixel difference refers to a grayscale difference between two adjacent frames that exceeds the second preset pixel difference.

[0216] The proportion of grayscale pixel differences between two adjacent frames in the first preset region that are equal to or greater than the second preset pixel difference in all grayscale pixel differences between the two adjacent frames in the first preset region.

[0217] For example, the grayscale pixel difference between each pixel in a first preset region is calculated for two adjacent frames. When the second preset pixel difference is 15, grayscale pixel differences greater than 15 are selected from the absolute values ​​of these differences. The proportion of grayscale pixel differences greater than 15 in the total number of grayscale pixel differences is determined. Only when this proportion is greater than a preset proportion (e.g., 0.2) is it considered that "entering the next video interface" may have occurred between two adjacent frames. If the preset proportion does not reach this threshold, the judgment is terminated. The range of the second preset pixel difference and the preset proportion is not limited here.

[0218] In some embodiments, after excluding video sliding artifacts caused by scrolling in the comment area, in order to obtain multiple feature point matching pairs between adjacent frames required for calculating the degree of motion between frames, multiple feature points of adjacent frames in a second preset area are paired based on the grayscale difference between adjacent frames in a first preset area to obtain multiple feature point matching pairs between adjacent frames, including:

[0219] Based on the second preset region, feature points are extracted from two adjacent frames to obtain multiple feature points of the two adjacent frames in the second preset region. The multiple feature points include a first feature point and a second feature point. The first feature point is the feature point of the first frame in the second preset region of the two adjacent frames, and the second feature point is the feature point of the second frame in the second preset region of the two adjacent frames.

[0220] Determine at least one candidate second feature point from the second feature points that matches the first feature point;

[0221] Based on the distance information between the first feature point and the matching candidate second feature point, a target second feature point that matches the first feature point is determined from at least one candidate second feature point;

[0222] Based on the target second feature point matched by multiple first feature points, multiple feature point matching pairs are obtained between two adjacent frames.

[0223] The first feature point is the feature point of the first frame in the second preset region among two adjacent frames.

[0224] The second feature point is the feature point of the second frame in the second preset region among two adjacent frames.

[0225] Candidate second feature points are second feature points that have a potential matching relationship with the first feature point.

[0226] For example, by using matching algorithms (such as ratio testing and other screening methods), second feature points that have a potential matching relationship with the first feature points are selected from the second feature points. These second feature points with potential matching relationships are determined based on the similarity between feature descriptors. The purpose is to find possible corresponding feature points between two adjacent frames.

[0227] A feature descriptor is a mathematical representation in computer vision used to describe the local appearance information of a specific region (such as feature points) in an image. It transforms a local region in an image into a compact and discriminative vector, allowing similar regions in different images to be identified and matched by comparing these vectors. A feature descriptor can uniquely characterize the features of a local region, such as edge pixels, corner points, and texture.

[0228] The target second feature point is the candidate second feature point that has the highest matching degree with the first feature point.

[0229] The ratio test is a commonly used feature point matching filtering method, especially suitable for feature descriptor-based matching tasks. Its core idea is to filter high-quality feature point matching pairs by judging the ratio of the nearest neighbor distance and the second nearest neighbor distance. For each feature point in the first frame, the distance between its feature descriptor and the descriptors of all feature points in the second frame is calculated (usually using Euclidean distance or Hamming distance). The two candidate second feature points with the smallest distance are found, called the nearest neighbor and the second nearest neighbor, respectively. The ratio of the nearest neighbor distance to the second nearest neighbor distance is calculated. If this ratio is less than a certain preset threshold (usually between 0.7 and 0.8), the feature point matching pair is considered reliable; otherwise, the match is considered unreliable and is discarded. Through the ratio test, only feature point matching pairs with ratios below this preset threshold are retained. These feature point matching pairs are considered to have high-confidence matching relationships and can reflect the relative movement between adjacent frames.

[0230] For example, if the nearest neighbor distance is smaller than the distance to the second nearest neighbor multiplied by a preset threshold, then the match is likely correct. This is because if the distances between the best-matching candidate second feature point and the second-best-matching candidate second feature point are very close, it cannot be determined whether the best-matching candidate second feature point is correct. However, if the distance from the first feature point to the best-matching candidate second feature point is much smaller than the distance from the first feature point to the second-best-matching candidate second feature point, then the best match is highly likely to be correct.

[0231] In some embodiments, considering that outliers may exist in multiple feature point matching pairs due to noise, mismatches, or abnormal motion, in order to effectively remove these outliers and more accurately obtain feature point matching pairs between adjacent frames for calculating the inter-frame motion degree between adjacent frames, after obtaining multiple feature point matching pairs between adjacent frames based on the target second feature points matched by multiple first feature points, the method further includes:

[0232] A preset number of feature point matching pairs are randomly selected from multiple feature point matching pairs in two adjacent frames;

[0233] A homography matrix is ​​constructed based on a preset number of feature point matching pairs. The homography matrix reflects the mapping relationship between the first feature point in the first frame and the second feature point of the target in the second frame in two adjacent frames.

[0234] Based on the homography matrix, the first feature point in multiple feature point matching pairs is transformed to obtain the transformed feature point corresponding to the first feature point.

[0235] The error information is determined based on the transformed feature point corresponding to the first feature point and its matching target second feature point;

[0236] Based on the error information, multiple feature point matching pairs between two adjacent frames are filtered.

[0237] The preset number refers to the minimum number of feature point matching pairs that are pre-set before constructing the homography matrix. Since at least four feature point matching pairs are required to calculate the homography matrix, the preset number is usually set to be greater than four to ensure higher accuracy and robustness.

[0238] The homography matrix reflects the mapping relationship between the first feature point in the first frame and the second feature point of the target in the second frame between two adjacent frames.

[0239] The transformed feature point corresponding to the first feature point refers to the new position obtained by projecting and transforming the first feature point in the first frame to the second frame using the homography matrix.

[0240] Error information refers to the difference between the transformed feature point corresponding to the first feature point and its corresponding target second feature point during feature point matching. It is a distance metric between the position of the transformed feature point in the second frame and the target second feature point. This error reflects the degree of deviation between the transformation calculated by the homography matrix and the actual match.

[0241] In some embodiments, when the error information is greater than a preset error information, the feature point matching pair corresponding to the error information is deleted.

[0242] In some embodiments, to eliminate video sliding artifacts caused by comment area scrolling, interactive effects, etc., multiple feature points of two adjacent frames are paired in a second preset region to obtain multiple feature point matching pairs of two adjacent frames, including:

[0243] The additional visual element regions in two adjacent frames are masked to obtain the updated two adjacent frames.

[0244] Multiple feature points of two adjacent frames are paired after the update to obtain multiple feature point matching pairs of two adjacent frames.

[0245] The additional visual element area refers to the area occupied by various visual elements displayed on top of the video content. These areas include, but are not limited to, the comment section, the bullet screen area, and the interactive effects area (such as reward effects, effects triggered by the streamer's actions, etc.).

[0246] The updated two adjacent frames are obtained by masking the regions containing additional visual elements in the adjacent frames. This masking operation effectively avoids extracting feature point pairs in these regions, thus preventing them from interfering with subsequent calculations.

[0247] For example, because the text in the comment section rises to the top as new comments appear, it creates a "swiping-up illusion," and this same swiping-up effect also occurs when navigating from a referral video or a video recommended by another video. To reduce interference from the comment text, the text area is obscured, thus avoiding interference with feature point matching and preventing impact on subsequent calculations.

[0248] In some embodiments, in order to calculate the inter-frame motion degree between two adjacent frames, the inter-frame motion degree between two adjacent frames is determined based on the position information of multiple feature point matching pairs between the two adjacent frames, including:

[0249] Based on the position information of multiple feature point matching pairs in two adjacent frames, determine the vertical displacement component of the second frame relative to the first frame in the two adjacent frames.

[0250] The degree of motion between adjacent frames is determined based on the position component in the vertical direction.

[0251] The vertical direction is the direction perpendicular to the horizontal axis.

[0252] The displacement component refers to the total vertical movement of the second frame relative to the first frame in two adjacent frames.

[0253] In some embodiments, in order to calculate the inter-frame motion degree between two adjacent frames, the displacement component of the second frame relative to the first frame in the vertical direction is determined based on the position information of multiple feature point matching pairs between the two adjacent frames, including:

[0254] Based on the position information of multiple feature point matching pairs in two adjacent frames, an affine transformation is performed on the multiple feature point matching pairs in two adjacent frames to obtain the affine transformation matrix.

[0255] Based on the affine transformation matrix, determine the displacement component of the second frame relative to the first frame in the vertical translation direction.

[0256] The affine transformation matrix is ​​used to map feature points from the first frame of a matching pair to the corresponding feature points in the second frame through matrix transformation. This process typically includes several geometric transformations such as translation, rotation, scaling, and shearing, which can be combined to describe the coordinate transformation relationship between two image frames.

[0257] For example, multiple feature point matching pairs between two adjacent frames include (A, A'), (B, B'), (C, C'), and (D, D').

[0258] The location information of feature point A is (10,20), the location information of feature point B is (30,40), the location information of feature point C is (50,60), and the location information of feature point D is (70,80).

[0259] The location information of feature point A' is (10,18), the location information of feature point B' is (30,38), the location information of feature point C is (50,58), and the location information of feature point D is (70,78).

[0260] In the feature point matching pair, the feature points in the first frame are used as the source point set src, and the feature points in the second frame are used as the target point set dst.

[0261] src=[[10,20],[30,40],[50,60],[70,80]];

[0262] dst=[[10,18],[30,38],[50,58],[70,78]].

[0263] Calculate the affine transformation matrix from src to dst:

[0264]

[0265] In the affine transformation matrix, the third value in the first row reflects the horizontal displacement between two adjacent frames, and the third value in the second row reflects the vertical displacement between two adjacent frames.

[0266] This means:

[0267] There is no horizontal translation between two adjacent frames (tx = 0);

[0268] The displacement component of the second frame relative to the first frame in the vertical translation direction is 2 pixels (ty = -2).

[0269] In the calculation of the affine transformation matrix, a mask matrix is ​​used to label the feature point matching pairs used in the calculation. For feature point matching pairs considered outliers (such as those caused by noise or incorrect matching), the corresponding positions in the mask matrix are marked as 0, indicating that these feature point matching pairs will not be considered in the calculation of the affine transformation matrix. In this way, only reliable feature point matching pairs are included in the calculation, thereby improving the accuracy of the affine transformation matrix.

[0270] In some embodiments, in order to calculate the inter-frame motion degree between two adjacent frames, the inter-frame motion degree between two adjacent frames is determined based on the position information of multiple feature point matching pairs between the two adjacent frames, including:

[0271] Based on the position information of the feature point matching pairs in two adjacent frames, determine the displacement distance of the feature point matching pairs in multiple directions;

[0272] Based on the displacement distance of the feature point matching pair in multiple directions and the preset displacement distance in multiple directions, the target feature point matching pair is determined from multiple feature point matching pairs in two adjacent frames.

[0273] Determine the proportion of target feature point matching pairs in multiple feature point matching pairs in two adjacent frames;

[0274] Based on the proportion information, determine the degree of inter-frame movement between two adjacent frames.

[0275] Here, direction refers to the direction in which the feature points in the second frame move relative to the feature points in the first frame within the feature point matching pair. For example, multiple directions include, but are not limited to, the vertical direction and the horizontal direction.

[0276] Displacement distance refers to the amount of displacement of a feature point in the second frame relative to a feature point in the first frame in the direction of movement within a feature point matching pair. For example, the displacement distance of a feature point matching pair in multiple directions includes the displacement in the vertical direction, the displacement in the horizontal direction, and so on.

[0277] The preset displacement distance is a pre-set value used to measure the displacement of the feature point matching pair in multiple directions, in order to determine the movement of the feature point in the second frame relative to the feature point in the first frame in a specific direction (up, down, left, right, etc.).

[0278] The target feature point matching pair uses the displacement distance in multiple directions to compare with the preset displacement distance in the corresponding direction. When it meets a specific condition (such as greater than, equal to or less than), it can be determined that it meets the preset displacement condition.

[0279] For example, when "the vertical displacement of the feature point matching pair exceeds 1 pixel and the horizontal displacement is less than 3 pixels", it can be determined that the feature point in the second frame of the feature point matching pair has moved upward relative to the feature point in the first frame, and the feature point matching pair is identified as the target feature point matching pair.

[0280] The percentage information refers to the proportion of the number of target feature point matching pairs in the total number of multiple feature point matching pairs.

[0281] Using multiple feature point matching pairs from two adjacent frames, including (A, A'), (B, B'), (C, C'), and (D, D'), with the position information of feature point A as (10,20), feature point B as (30,40), feature point C as (50,60), and feature point D as (70,80), and the position information of feature point A' as (10,23), feature point B' as (30,42), feature point C' as (50,64), and feature point D' as (70,85), determine the displacement distance of the feature point matching pairs in the vertical and horizontal directions:

[0282] For feature point matching pairs (A, A'), the vertical displacement distance = 23 - 20 = 3, and the horizontal displacement distance = 10 - 10 = 0; for feature point matching pairs (B, B'), the vertical displacement distance = 42 - 40 = 2, and the horizontal displacement distance = 30 - 30 = 3; for feature point matching pairs (C, C'), the vertical displacement distance = 64 - 60 = 4, and the horizontal displacement distance = 50 - 50 = 0; for feature point matching pairs (D, D'), the vertical displacement distance = 85 - 80 = 5, and the horizontal displacement distance = 70 - 70 = 0.

[0283] Based on the preset displacement condition, when "the vertical displacement distance of the feature point matching pair exceeds 1 pixel upward and the horizontal displacement distance is less than 3 pixels," it can be determined that the feature point in the second frame of the feature point matching pair has moved upward relative to the feature point in the first frame. It can be seen that the vertical displacement distance of the feature point matching pairs (A, A`), (B, B`), (C, C`), and (D, D`) is greater than 1 pixel in the vertical direction and the horizontal displacement distance is less than 3 pixels in the horizontal direction. Thus, the feature point matching pairs (A, A`), (B, B`), (C, C`), and (D, D`) can be determined as the target feature point matching pairs.

[0284] In some embodiments, considering that the inter-frame motion between two adjacent frames may fall within a weak inter-frame motion range, and since a weak inter-frame motion range cannot effectively reflect real scene changes, in order to accurately detect whether a first scene frame exists between two adjacent frames, the first scene frame is determined from the video frame sequence based on the inter-frame motion level, including:

[0285] When the inter-frame motion between two adjacent frames is within the range of weak inter-frame motion, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames. The second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0286] Based on the difference in grayscale values ​​between two adjacent frames in the first preset region, multiple feature points of the two adjacent frames in the second preset region are paired to obtain multiple feature point matching pairs of the two adjacent frames.

[0287] Based on the position information of multiple feature point matching pairs in the next two adjacent frames, the inter-frame motion degree of the next two adjacent frames is determined.

[0288] When the inter-frame motion of the next two adjacent frames falls within the range of weak inter-frame motion, the first frame of the two adjacent frames is determined as the first scene frame.

[0289] The weak inter-frame motion range is used to define a low level of inter-frame motion between two adjacent frames, meaning that the inter-frame motion between two adjacent frames is not obvious and the displacement is low. In other words, when the inter-frame motion between two adjacent frames is below a certain threshold, it indicates that the inter-frame motion between two adjacent frames is relatively stable, and the scene is either not moving significantly or stationary.

[0290] The first frame of the next two adjacent frames is the same as the second frame of the next two adjacent frames.

[0291] For example, when the video frame sequence is {f1, f2, ... f... n When two adjacent frames are (f1, f2), the next two adjacent frames are (f2, f3).

[0292] The difference in grayscale value between the next two adjacent frames in the first preset area is the difference in grayscale pixel value between the next two adjacent frames in the first preset area, which can reflect the degree of change between the next two adjacent frames in the first preset area.

[0293] The feature point matching pair of the next two adjacent frames is a set of corresponding feature points between the next two adjacent frames, and the feature point matching pair shows similarity in the two adjacent frames.

[0294] The position information of the feature point matching pair in the next two adjacent frames is the position of the feature point in the feature point matching pair within their respective frames.

[0295] The degree of inter-frame motion between the next two adjacent frames is quantified based on the positional information of multiple feature point matching pairs in the next two adjacent frames, and can reflect the relative movement that occurs between the next two adjacent frames.

[0296] In some embodiments, when the inter-frame motion of two adjacent frames is within the range of weak inter-frame motion, and the inter-frame motion of the next two adjacent frames is within the range of weak inter-frame motion, then the first frame of the two adjacent frames is determined as the first scene frame.

[0297] In some embodiments, when the inter-frame motion of two adjacent frames is within the range of weak inter-frame motion and the inter-frame motion of the next two adjacent frames is within the range of strong inter-frame motion, the first frame of the two adjacent frames is determined as the first scene frame.

[0298] In some embodiments, when the inter-frame motion of two adjacent frames is within the range of weak inter-frame motion and the inter-frame motion of the next two adjacent frames is within the range of strong inter-frame motion, the first frame of the next two adjacent frames is determined as the first scene frame.

[0299] In some embodiments, considering that in short video or live streaming scenarios, users may see the next video content immediately upon swiping (i.e., an instant-on scenario), but may also need to wait for a period of time after triggering the swipe before seeing the next video content (i.e., a non-instant-on scenario), in order to accurately identify the second video frame in the video frame sequence in a non-instant-on scenario, a scene switching detection process is performed on the video frame sequence based on a second sub-strategy to obtain the second scene frame in the video frame sequence, including:

[0300] In response to the third access scenario, if the number of edge pixels in the second frame of two adjacent frames in the video frame sequence is less than the preset number of pixels, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames, and the second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0301] Based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0302] When the contour information of the second frame in the next two adjacent frames meets the preset contour conditions, the second frame in the next two adjacent frames is determined as the second scene frame.

[0303] The number of edge pixels in the second frame of two adjacent frames refers to the total number of edge pixels identified in the second frame of two adjacent frames.

[0304] The preset number of pixels is a pre-set number of pixels used to measure the number of edge pixels in the second frame between two adjacent frames.

[0305] The multiple image features of the next two adjacent frames are the visual attributes possessed by the next two adjacent frames. These attributes provide important basis for quantifying the differences between frames when analyzing and comparing the changes between the next two adjacent frames. For example, the multiple image features of the next two adjacent frames include grayscale mean, grayscale variance, sharpness, difference between grayscale maximum and minimum values, number of edge pixels, and difference in grayscale values.

[0306] The difference in grayscale values ​​between two adjacent frames can reflect the degree of change between them. For example, based on the difference in grayscale values, it can be determined whether the change between two adjacent frames is a "strong change" or a "weak change".

[0307] The outline information of the second frame in the next two adjacent frames is the area occupied by the outline of the image content in the second frame in the next two adjacent frames. The image content may include elements such as text and objects.

[0308] The preset contour condition is a pre-set contour condition used to measure whether there is image content in the second frame of the next two adjacent frames.

[0309] For example, if the number of edge pixels in the second frame of two adjacent frames in a video frame sequence is less than a preset number of pixels, it indicates that no video image appears in the second frame of the two adjacent frames, belonging to a non-instant-opening scenario. That is, the second frame of the two adjacent frames does not belong to the video frame corresponding to the appearance of the video to be played on the video interface. Thus, based on the two adjacent frames, the next two adjacent frames are determined from the video frame sequence, and the second frame of the next two adjacent frames is the same as the first frame of the next two adjacent frames. When multiple image features and grayscale differences between the next two adjacent frames suggest that the second frame of the next two adjacent frames may contain video imagery, the contour information of the second frame in the next two adjacent frames is further examined.

[0310] The preset contour conditions can be as follows: if the area of ​​the contour is greater than 1500, increment the count by 3; otherwise, if the area of ​​the contour is greater than 700 but less than 1500, increment the count by 2. If the area of ​​the contour is greater than 100, increment the count by 1. When the total count is not less than 3, it indicates that the second frame does indeed contain an image, and it can be determined that the second frame of the next two adjacent frames is the second scene frame.

[0311] In some embodiments, the contour information of the second frame in the next two adjacent frames is filtered based on a preset contour area.

[0312] The preset contour area is a pre-set area used to measure the contours that need to be filtered.

[0313] For example, the preset contour area can be 35. Contours with the preset contour area in the contour information are filtered out because such contours are likely to represent text rather than actual image content. The range of values ​​for the preset contour area is not limited here.

[0314] In some embodiments, in non-instantaneous opening scenarios, to determine whether the next two adjacent frames meet the condition of changing from a "black screen" or "one video frame" to "another video frame," and to determine whether to further detect the contour information of the second frame in the next two adjacent frames, contour detection processing is performed on the second frame in the next two adjacent frames based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, to obtain the contour information of the second frame in the next two adjacent frames, including:

[0315] Based on the first preset detection area, perform region scanning processing on the next two adjacent frames to obtain the first detection area in the next two adjacent frames.

[0316] Determine multiple image features in the first detection region for the next two adjacent frames;

[0317] Based on the grayscale pixel difference between the next two adjacent frames, determine the grayscale pixel difference that is greater than the third preset pixel difference.

[0318] The grayscale difference between the next two adjacent frames is determined based on the grayscale pixel difference greater than the third preset pixel difference and the grayscale pixel difference between the next two adjacent frames.

[0319] Based on multiple image features in the first detection region of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0320] The first preset detection area is a pre-set area where image features need to be detected. This area is used to determine whether there are image changes in the next two adjacent frames.

[0321] For example, in an area where new video content appears, the previous frame in that area could be either a black screen or the content of the first video frame. Therefore, the first preset detection area can include [screen resolution height multiplied by 0.25, screen resolution height multiplied by 0.5], [screen resolution height multiplied by 0.52, screen resolution height multiplied by 0.78], [screen resolution height multiplied by 0.72, screen resolution height multiplied by 0.985], used to determine whether the next two adjacent frames jump from a "black screen" or "first video frame" to the "second video frame".

[0322] The first region to be detected is the region in the next two adjacent frames that corresponds to the first preset detection region.

[0323] For example, based on the first preset detection area, the next two adjacent frames can be scanned to identify the area corresponding to the first preset detection area as the first detection area.

[0324] The multiple image features of the next two adjacent frames in the first detection area are the visual attributes that the next two adjacent frames have in the first detection area, which may include grayscale image mean, grayscale variance, sharpness, difference between grayscale maximum and minimum values, number of edge pixels, grayscale difference, etc.

[0325] The grayscale pixel difference between the next two adjacent frames refers to the difference between the grayscale images of the first frame and the second frame in the next two adjacent frames. Specifically, this difference is calculated by comparing the grayscale values ​​of each pixel in the first frame and the second frame.

[0326] The third preset pixel difference is a pixel difference threshold set in advance in the third access scenario and in a non-instant opening scenario. It is used to measure the grayscale difference of pixels between the next two adjacent frames.

[0327] The grayscale pixel difference greater than the third preset pixel difference value refers to the grayscale difference between pixels in the next two adjacent frames that exceeds the third preset pixel difference value.

[0328] For example, the grayscale pixel difference of each pixel in the first preset detection area is calculated between the next two adjacent frames. When the third preset pixel difference is 25, grayscale pixel differences greater than 25 are selected from the absolute values ​​of these grayscale pixel differences. The proportion of grayscale pixel differences greater than 25 in the total number of grayscale pixel differences in the first preset detection area is determined. Based on the multiple image features of the next two adjacent frames in the first detection area and the proportion greater than the preset proportion (such as 0.1), it is considered that there has been a significant change between the next two adjacent frames in the first preset detection area. Here, the range of the second preset pixel difference and the preset proportion is not limited.

[0329] In some embodiments, when there are multiple first detection regions, if the current first preset detection region changes significantly between the next two adjacent frames, then it is no longer necessary to check the next first detection region in the next two adjacent frames; otherwise, if the current first preset detection region does not change significantly, then the next first detection region in the next two adjacent frames will continue to be detected.

[0330] In some embodiments, in non-instantaneous opening scenarios, to determine whether the next two adjacent frames meet the condition of changing from a "black screen" or "one video frame" to "another video frame," contour detection processing is performed on the second frame of the next two adjacent frames based on multiple image features and the grayscale difference between the next two adjacent frames, to obtain the contour information of the second frame of the next two adjacent frames, including:

[0331] Based on multiple image features of the next two adjacent frames, the target feature conditions are determined from multiple preset image feature conditions;

[0332] The scores of the preset conditions corresponding to the target feature conditions are statistically analyzed to obtain the total target score;

[0333] Based on the target total score, the number of edge pixels in the second frame of the next two adjacent frames, and the difference in grayscale values ​​between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0334] Among them, the preset image feature conditions are pre-set conditions used to measure multiple image features of the next two adjacent frames to confirm whether there is an image change between the next two adjacent frames.

[0335] The target feature condition is a preset image feature condition that multiple image features satisfy in the next two adjacent frames.

[0336] The preset condition score is a pre-set score corresponding to preset image feature conditions. When the preset image feature conditions are met, the score of the condition is assigned and used to calculate the target total score to determine the degree of image change.

[0337] The target total score is the sum of the preset condition scores corresponding to the target feature conditions.

[0338] For example, multiple image features of the next two adjacent frames may include: the grayscale mean of the first frame (cropped-mean-gray1) and the grayscale mean of the second frame (cropped-mean-gray2) in the next two adjacent frames; the grayscale variance of the first frame (cropped-var-gray1) and the grayscale variance of the second frame (cropped-var-gray2) in the next two adjacent frames; the sharpness of the first frame (cropped-clear1) and the sharpness of the second frame (cropped-clear2) in the next two adjacent frames; the difference between the maximum and minimum grayscale values ​​of the first frame (during1) and the second frame (during2) in the next two adjacent frames; the number of edge pixels in the first frame (edge1) and the number of edge pixels in the second frame (edge2) in the next two adjacent frames; and the grayscale difference (diff-rate).

[0339] 1. Preset image feature conditions for the mean of a grayscale image, if they satisfy:

[0340] cropped-mean-gray2-cropped-mean-gray1>30;

[0341] Or satisfy:

[0342] cropped-mean-gray2-cropped-mean-gray1 / min(cropped-mean-gray1, cropped-mean-gray1)≥2;

[0343] Then add 1, where 1 is the preset condition score corresponding to the preset image feature condition for the grayscale image mean.

[0344] 2. Regarding the preset image feature conditions for grayscale variance, if they are satisfied:

[0345] cropped-var-gray2-cropped-var-gray1>800;

[0346] Or satisfy:

[0347] cropped-var-gray2-cropped-var-gray1 / cropped-var-gray1≥2;

[0348] Then add 1, where 1 is the preset condition score corresponding to the preset image feature conditions for grayscale variance.

[0349] 3. Regarding the preset image feature conditions for sharpness, if the following conditions are met:

[0350] cropped-clear2-cropped-clear1>500;

[0351] Or satisfy:

[0352] cropped-clear2-cropped-clear1 / cropped-clear1≥5;

[0353] Then add 1, where 1 is the preset condition score corresponding to the preset image feature conditions for clarity.

[0354] 4. Regarding the preset image feature conditions for the difference between grayscale maximum and minimum values, if the following conditions are met:

[0355] during2-during1>10;

[0356] Or it satisfies: during1 < 100;

[0357] Then add 1, where 1 is the preset condition score corresponding to the preset image feature condition for the difference between the maximum and minimum gray values.

[0358] 5. Preset image feature conditions for the number of edge pixels, if satisfied:

[0359] edge2-edge1>700;

[0360] Or it satisfies: edge1 < 100;

[0361] Then add 1, where 1 is the preset condition score corresponding to the preset image feature condition of the number of edge pixels.

[0362] 6. For preset image feature conditions regarding grayscale difference, if the following conditions are met:

[0363] If diff-rate > 0.5, then add 1, where 1 is the preset condition score corresponding to the preset image feature condition for gray-scale difference.

[0364] After judging the above 6 features, if the target total score is greater than or equal to 4, and edge2 is greater than 100 and diff-rate is greater than 0.1, it is determined that there has been a significant change between the next two adjacent frames, which is very likely to be consistent with the change from "black screen" or "the content of the first video frame" to "the second video frame".

[0365] In this situation, it's possible that the title text or other content below the first video has moved up, causing it to be judged as a significant change between the next two adjacent frames, even though the new video content hasn't actually appeared yet. Therefore, it's necessary to avoid text interference.

[0366] After performing a series of operations such as Gaussian blur, edge detection, erosion and dilation on the grayscale image of the second frame in the next two adjacent frames, contour detection is performed, and contours with small areas are filtered out (the threshold is set to 35 here). If the area of ​​a contour is less than 35, it is likely text and not actual image content. A statistical approach is used here, denoted as count. If the area of ​​a contour is greater than 1500, count is incremented by 3; otherwise, if the area of ​​a contour is greater than 700 but less than 1500, count is incremented by 2; otherwise, if the area of ​​a contour is greater than 100, count is incremented by 1.

[0367] If the final statistic is greater than or equal to 3, it means that the second frame in the next two adjacent frames does indeed contain an image. When the "non-instant opening scene" judgment branch reaches this point, it means that a "second scene frame" has appeared.

[0368] In some embodiments, considering that the next video content a user might see while swiping is a static preset cover such as "frosted glass," which does not play actual video content, to avoid interference from static covers in the instant-on scene judgment and to accurately detect the second video frame in the video frame sequence, based on the second sub-strategy, scene switching detection processing is performed on the video frame sequence to obtain the second scene frame in the video frame sequence, including:

[0369] In response to the third access scenario, if the number of edge pixels of the second frame of two adjacent frames in the video frame sequence is not less than the preset number of pixels, and the number of edge pixels of the second frame in the content entry area of ​​two adjacent frames is less than the preset number of edge pixels, then a number of consecutive video frames after the second frame in two adjacent frames are determined from the video frame sequence. The content entry area refers to the starting area in the video interface where the content to be displayed begins to replace the currently displayed content.

[0370] Content recognition processing is performed on multiple consecutive video frames to obtain the video content;

[0371] Based on the video content, the second frame in a pair of adjacent frames is determined as the second scene frame.

[0372] The currently displayed content is the content currently shown on the video interface.

[0373] Content to be displayed refers to content that has not yet been shown and is used to replace the content currently displayed in the video interface.

[0374] The content entry area refers to the starting area in the video interface where the content to be displayed begins to replace the currently displayed content.

[0375] For example, when the content to be displayed enters the video interface from bottom to top, replacing the currently displayed content, the content enters the bottom area of ​​the screen. When the content to be displayed enters the video interface from top to bottom, replacing the currently displayed content, the content enters the top area of ​​the screen. When the content to be displayed enters the video interface from left to right, replacing the currently displayed content, the content enters the left side of the screen. When the content to be displayed enters the video interface from right to left, replacing the currently displayed content, the content enters the right side of the screen.

[0376] The number of edge pixels in the content entry area of ​​the second frame in two adjacent frames is the number of edge pixels in the content entry area identified from the second frame of the two adjacent frames.

[0377] The preset edge pixel count is a pre-set number of pixels used to measure the number of edge pixels in the content entry area of ​​the second frame in two adjacent frames.

[0378] A series of consecutive video frames refers to a sequence of video frames that are consecutive after the second frame in any two adjacent frames. The number of consecutive video frames is set according to the actual situation.

[0379] For example, multiple consecutive video frames can be two consecutive frames, three consecutive frames, or the remaining video frames after the second frame in a video frame sequence.

[0380] The video content is used to indicate the type of content to be displayed in the video interface, and to determine whether the content to be displayed is a video type that can be played directly.

[0381] For example, if the number of edge pixels in the second frame of two adjacent frames in a video frame sequence is not less than a preset number of pixels, it indicates that video footage appears in the second frame of the two adjacent frames, which may belong to a scene that opens instantly. If the number of edge pixels in the content entry area of ​​the second frame of two adjacent frames is less than a preset number of edge pixels, it indicates that the image content in the content entry area of ​​the second frame may belong to a preset cover image (such as "frosted glass," "blur effect," "text watermark," etc.). To prove that it is not a preset cover image, several consecutive video frames following the second frame of the two adjacent frames are determined from the video frame sequence. Using a content processing model (such as a multimodal large model, classification model, etc.), corresponding prompts are designed and several consecutive video frames are input, allowing the content processing model to determine whether the following video content belongs to a preset cover image. If the video content does not belong to a preset cover image, the second frame of the two adjacent frames is determined as the second scene frame. When the first scene frame and the second scene frame are two adjacent video frames, it belongs to a scene that opens instantly.

[0382] Because the blurred effect of the preset cover image results in images without sharp edges, images with sharp edges can be quickly filtered based on the number of edge pixels in the content entry area of ​​the second frame out of two adjacent frames. The preset cover image is a pre-set cover image. For example, the preset cover image can be frosted glass, a blurred effect, or a text watermark.

[0383] In some embodiments, to confirm whether the instant-open scenario is met, and further detect a second video frame in the video frame sequence, in response to a third access scenario, if the number of edge pixels of the second frame in two adjacent frames in the video frame sequence is not less than a preset number of pixels, and the number of edge pixels of the second frame in the content entry area in two adjacent frames is less than a preset number of edge pixels, then determine a plurality of consecutive video frames following the second frame in two adjacent frames from the video frame sequence, including:

[0384] In response to the third access scenario, if the number of edge pixels in the second frame of two adjacent frames in the video frame sequence is not less than the preset number of pixels, the two adjacent frames are scanned based on the second preset detection area to obtain the second detection area in the two adjacent frames.

[0385] Determine multiple image features of two adjacent frames in the second detection region;

[0386] Based on multiple image features of two adjacent frames in the second detection region, the target feature conditions satisfied by the two adjacent frames are determined from multiple preset image feature conditions.

[0387] Based on the target feature conditions satisfied by two adjacent frames, determine the number of edge pixels in the content entry area of ​​the second frame in the two adjacent frames.

[0388] When the number of edge pixels in the content entry area of ​​the second frame in two adjacent frames is less than the preset number of edge pixels, multiple consecutive video frames after the second frame in two adjacent frames are determined from the video frame sequence.

[0389] The second preset detection area is a pre-set area for detecting image features. This area is used to determine whether there are image changes between two adjacent frames.

[0390] For example, when the content to be displayed enters the video interface from bottom to top, the second preset detection area is located in the n rows of pixels at the bottom of the screen, and the specific range is defined based on the height of the screen resolution. The second preset detection area can include [screen resolution height multiplied by 0.996, screen resolution height multiplied by 1], [screen resolution height multiplied by 0.975, screen resolution height multiplied by 0.98], and these areas help detect changes in the bottom area of ​​the screen, thereby determining whether there is a content update.

[0391] When the content to be displayed enters the video interface from top to bottom, the second preset detection area is located in the n rows of pixels at the top of the screen. The specific range is defined based on the height of the screen resolution. These areas help detect changes in the top area of ​​the screen, thereby determining whether there is a content update.

[0392] When the content to be displayed enters the video interface from left to right, the second preset detection area is located in n rows of pixels on the left side of the screen. The specific range is defined based on the width of the screen resolution. These areas help detect changes in the left side of the screen, thereby determining whether there is a content update.

[0393] When the content to be displayed enters the video interface from right to left, the second preset detection area is located in n rows of pixels on the right side of the screen. The specific range is defined based on the width of the screen resolution. These areas help detect changes in the right side of the screen, thereby determining whether there is a content update.

[0394] The second region to be detected is the region in two adjacent frames that corresponds to the second preset detection region.

[0395] Multiple image features of two adjacent frames in the second detection area are the visual attributes that the two adjacent frames have in the second detection area, which may include grayscale image mean, grayscale variance, sharpness, etc.

[0396] The target feature conditions satisfied by two adjacent frames are the preset image feature conditions satisfied by multiple image features of the two adjacent frames.

[0397] The number of edge pixels in the content entry area of ​​the second frame in two adjacent frames refers to the total number of edge pixels located in the second detection area in the second frame of two adjacent frames.

[0398] For example, multiple image features of two adjacent frames in the second detection region may include: the grayscale mean value (cropped-mean-gray1) of the first frame and the grayscale mean value (cropped-mean-gray2) of the second frame in the second detection region; the grayscale variance (cropped-var-gray1) of the first frame and the grayscale variance (cropped-var-gray2) of the second frame in the second detection region; and the sharpness (cropped-clear1) of the first frame and the sharpness (cropped-clear2) of the second frame in the second detection region.

[0399] 1. In scenarios where instant loading is possible, the preset image feature conditions for the grayscale image mean, if satisfied:

[0400] cropped-mean-gray2-cropped-mean-gray1>10;

[0401] Then add 2 to the calculation;

[0402] Otherwise, if the following conditions are met:

[0403] cropped-mean-gray2-cropped-mean-gray1 / cropped-mean-gray1≥1.5;

[0404] Then add 2 to the calculation;

[0405] Otherwise, if the following conditions are met:

[0406] cropped-mean-gray2-cropped-mean-gray1>1;

[0407] Then add 1 to the calculation;

[0408] Otherwise, if the following conditions are met:

[0409] cropped-mean-gray2-cropped-mean-gray1 / cropped-mean-gray1≥0.8;

[0410] Then add 1 to the calculation.

[0411] 2. In scenarios that may open instantly, the preset image feature conditions for grayscale variance, if satisfied:

[0412] |cropped-var-gray2-cropped-var-gray1|>35;

[0413] Then add 2 to the calculation;

[0414] Otherwise, if the following conditions are met:

[0415] cropped-var-gray2-cropped-var-gray1 / cropped-var-gray1≥1.5;

[0416] Then add 2 to the calculation;

[0417] Otherwise, if the following conditions are met:

[0418] |cropped-var-gray2-cropped-var-gray1|>5;

[0419] Then add 1 to the calculation;

[0420] Otherwise, if the following conditions are met:

[0421] cropped-var-gray2-cropped-var-gray1 / cropped-var-gray1≥0.8;

[0422] Then add 1 to the calculation.

[0423] 3. In scenarios where instant loading is possible, preset image feature conditions for image clarity, if satisfied:

[0424] |cropped-clear2-cropped-clear1|>20;

[0425] Then add 2 to the calculation;

[0426] Otherwise, if the following conditions are met:

[0427] cropped-clear2-cropped-clear1 / cropped-clear1≥1.5;

[0428] Then add 2 to the calculation;

[0429] Otherwise, if the following conditions are met:

[0430] |cropped-clear2-cropped-clear1|>2;

[0431] Then add 1 to the calculation;

[0432] Otherwise, if the following conditions are met:

[0433] cropped-clear2-cropped-clear1 / cropped-clear1≥0.8;

[0434] Then add 1 to the calculation;

[0435] After the above three feature judgments, if the total score obtained based on the target feature conditions satisfied by two adjacent frames is greater than or equal to 3, and the difference between the maximum and minimum grayscale values ​​of the second frame in the second detection area is greater than or equal to 10, then it can be considered that a significant change has occurred in the second detection area from the first frame to the second frame between the two adjacent frames. Based on this, a further constraint is imposed that `cropped-clear2` must be greater than or equal to 2. This condition emphasizes that a change has occurred and that there is indeed an image below. If all three conditions are met, then it can be considered that a new image has appeared below, which meets the "instant opening scene" setting.

[0436] However, the authenticity of the "instant scene opening" still needs further verification. Because the new screen only appears in the content entry area, it is difficult to determine whether the content of the next video has actually appeared, or whether a video cover with a preset cover image (such as frosted glass) effect set by the next live stream host has appeared (used to temporarily display the video cover when the next video has not finished loading, and the video cover is highly blurred).

[0437] To address this situation, first determine if the number of edge pixels in the content area of ​​the second frame is greater than or equal to 2. If this condition is met, it proves that it is not "frosted glass," because the blurring effect of "frosted glass" would result in an image without clear edges. Therefore, this method can be used to quickly filter out images with clear edges, considering them to be the actual scene of the next video, meeting the "true instant opening" requirement, at which point the judgment ends. Otherwise, further judgment is still required.

[0438] Further judgment was made using a content processing model. Corresponding prompts were designed and multiple consecutive video frames were input, allowing the content processing model to determine whether the next newly appearing content belonged to the "frosted glass" video cover or the actual content of the next video.

[0439] In some embodiments, to avoid interference from static cover images in determining the instant-open scene, a second scene frame can be further detected. Based on the video content, the second frame among two adjacent frames is determined as the second scene frame, including:

[0440] When the video content does not belong to the preset cover image, the second frame of the two adjacent frames is determined as the second scene frame;

[0441] When the video content belongs to the preset cover image, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames. The second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0442] Based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0443] When the contour information of the second frame in the next two adjacent frames meets the preset contour conditions, the second frame in the next two adjacent frames is determined as the second scene frame.

[0444] For example, if the content processing model determines that the video content is newly appearing and is real video content, then it is "true instant opening". Therefore, the second frame in two adjacent frames can be identified as the second scene frame, and the judgment can be terminated. Conversely, if the content processing model determines that it is a preset cover image ("frosted glass" effect), then it is determined as "fake instant opening".

[0445] In the above-mentioned "instant opening" judgment process, if the condition of "true instant opening" is met, the judgment is exited; otherwise, it is considered that "instant opening" has not been achieved, and the judgment branch of the second scene frame of "non-instant opening" needs to be entered.

[0446] 105. Determine the video startup time based on the first scene frame and the second scene frame.

[0447] The video startup time refers to the time interval between the first scene frame and the second scene frame.

[0448] In some embodiments, in order to calculate the video startup time based on the first scene frame and the second scene frame, determining the video startup time based on the first scene frame and the second scene frame includes:

[0449] The frame spacing is determined based on the positions of the first scene frame and the second scene frame in the video frame sequence.

[0450] Determine the duration of a single video frame based on the frame rate of the screen recording video;

[0451] The video startup time is determined based on the frame interval and the duration of a single video frame.

[0452] Frame spacing refers to the positional difference between the first scene frame and the second scene frame in a video frame sequence.

[0453] Frame spacing = Frame number of the second scene frame - Frame number of the first scene frame.

[0454] Frame rate refers to the number of image frames captured and recorded per second when recording the screen, usually expressed as frames per second (FPS).

[0455] The duration of a single video frame refers to the time occupied by each frame of an image at the frame rate of a screen-recorded video.

[0456] Duration of a single video frame = 1000 / frame rate of the screen recording video.

[0457] The time it takes to start a video is equal to the frame interval multiplied by the duration of a single video frame.

[0458] Video startup time calculation:

[0459] Where video_time is the time taken to start the video, fps is the frame rate of the screen recording video, video(0) is the position of the second scene frame in the video frame sequence, and click(0) is the position of the first scene frame in the video frame sequence.

[0460] In some embodiments, in order to calculate the video startup time based on the first scene frame and the second scene frame, determining the video startup time based on the first scene frame and the second scene frame includes:

[0461] Determine the first playback time point corresponding to the first scene frame in the screen recording video, and the second playback time point corresponding to the second scene frame in the screen recording video;

[0462] The video startup time is determined based on the first and second playback times.

[0463] The first playback time point refers to the moment when the second scene frame appears in the screen recording video.

[0464] The second playback time point is the time when the second scene frame appears in the screen recording video.

[0465] The video startup time is the time difference between the appearance of the first scene frame and the appearance of the second scene frame.

[0466] As can be seen from the above, the embodiments of this application can acquire screen recording videos, which record the process of entering the video interface from the target access point and playing the video on the video interface; perform frame segmentation processing on the screen recording video to obtain a video frame sequence; determine the target detection strategy corresponding to the target access point from multiple detection strategies; perform scene switching detection processing on the video frame sequence based on the target detection strategy to obtain the first scene frame and the second scene frame in the video frame sequence, where the first scene frame corresponds to the time point when the target access point is triggered and the second scene frame corresponds to the time point when the video to be played appears on the video interface; and determine the video start time based on the first scene frame and the second scene frame.

[0467] Therefore, this solution can perform frame-by-frame processing on the acquired screen recording video to obtain a video frame sequence. Since the screen recording video documents the process of entering the video interface from the target access point and playing the video on the video interface, a target detection strategy can be determined from multiple detection strategies based on this target access point. Thus, based on this target detection strategy, scene switching detection processing can be performed specifically on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence. Because the first scene frame corresponds to the time point when the target access point is triggered, and the second scene frame corresponds to the time point when the video to be played appears on the video interface, the video startup time can be accurately calculated based on these two scene frames. This method improves the accuracy of time calculation, helps to accurately evaluate user experience, further supports effective platform optimization, helps reduce video startup time, improves video startup efficiency, and thus enhances user satisfaction.

[0468] The method described in the above embodiments will be further described in detail below.

[0469] In this embodiment, the method of this application embodiment will be described in detail using the instant opening time performance detection applied to video platforms such as live streaming, short videos, and long videos as an example.

[0470] like Figure 2a As shown, the specific process of one detection method is as follows:

[0471] 201. Obtain the screen recording video. The screen recording video records the process of entering the video interface from the target access point and playing the video on the video interface.

[0472] 202. Perform frame segmentation on the screen recording video to obtain a video frame sequence.

[0473] 203. Determine the target detection strategy corresponding to the target access point from multiple detection strategies.

[0474] 204. Based on the target detection strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access entry is triggered, and the second scene frame corresponds to the time point when the video to be played appears on the video interface.

[0475] In some embodiments, the target detection strategy includes a first sub-strategy and a second sub-strategy. The first sub-strategy is used to detect the video frame corresponding to the triggering of the target access entry, and the second sub-strategy is used to detect the video frame corresponding to the appearance of the video interface when the video to be played appears.

[0476] Based on the object detection strategy, scene transition detection processing is performed on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence, including:

[0477] Based on the first sub-strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame in the video frame sequence.

[0478] Based on the second sub-strategy, scene switching detection processing is performed on the video frame sequence to obtain the second scene frame in the video frame sequence.

[0479] In some embodiments, based on a first sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain a first scene frame in the video frame sequence, including:

[0480] In response to the first access scenario, a grayscale pixel difference greater than a first preset pixel difference is determined based on the grayscale pixel difference between two adjacent frames in the video frame sequence.

[0481] The grayscale difference between two adjacent frames is determined based on the grayscale pixel difference greater than the first preset pixel difference and the grayscale pixel difference between two adjacent frames.

[0482] Determine multiple image features of a specific frame in a video frame sequence between two adjacent frames, where the specific frame is one of the two adjacent frames;

[0483] The first scene frame is determined from the video frame sequence based on the difference in grayscale values ​​between two adjacent frames and multiple image features of a specific frame in the two adjacent frames.

[0484] like Figure 2b As shown, in a live streaming scenario, when a user moves from one live stream screen 1 to another live stream screen 2, the process involves clicking to enter a new page, data reading, data processing, page rendering, and video stream rendering. The video startup time needs to be calculated from the moment the user starts swiping into the other live stream until live stream screen 2 appears.

[0485] like Figure 2cAs shown, the access point can be a URL address, a video cover of a content aggregation page, a referral video, or a video containing a video recommendation entry point. Screen recording videos can record the process of entering the video interface from the target access point and playing the video on the video interface. Screen recording videos can be detected, and the first scene frame and the second scene frame corresponding to different access points can be detected.

[0486] like Figure 2d As shown, in a live streaming scenario, the video startup time is detected based on four different access points.

[0487] 1.1, such as Figure 2e As shown, when the first access scenario is a scene where the video interface is accessed from a URL address, the specific steps for scene switching detection processing of the video frame sequence based on the first sub-strategy are as follows:

[0488] Condition 1:

[0489] First, the entire page will turn black, resulting in a significant change. The grayscale images of adjacent frames in the video frame sequence are subtracted to obtain the grayscale pixel difference. A first preset pixel difference value is set to 25. The proportion of grayscale pixel differences greater than this value is counted among all grayscale pixel differences to obtain the grayscale difference between adjacent frames. A preset proportion is set to 0.6; a grayscale difference greater than this proportion is considered a "strong change" between adjacent frames. If adjacent frames meet the above "strong change" criteria, then condition 1 is satisfied.

[0490] Condition 2:

[0491] As the screen transitions from a dialog box to black, the grayscale level decreases. Furthermore, if the entire screen turns black, the grayscale level not only decreases but also falls below a certain threshold. Here, the grayscale image mean threshold is set to 15. If a specific frame between two adjacent frames is taken as the second frame, and the grayscale image mean of the second frame is lower than the grayscale image mean threshold, then condition 2 is considered to be met.

[0492] Exiting the live stream (returning to the chat window) still results in the page turning completely black. Therefore, "exiting" and "entering" result in the same change. To differentiate them, we can consider the different characteristics of "entering" and "exiting":

[0493] a) Although the screen disappears upon exiting, the avatar banner remains. Therefore, the edge detection algorithm can still identify a certain number of edge pixels, thus obtaining the number of edge pixels. By counting these edge pixels, it's possible to distinguish between a completely black screen and a "false black screen" where the avatar banner still exists.

[0494] b) Sometimes the comments section remains even after the live stream video interface disappears.

[0495] Condition 3:

[0496] Based on a above, the threshold for the number of edge pixels is set to 100. If the number of edge pixels in the second frame of two adjacent frames is less than the threshold, then condition 3 is met.

[0497] Condition 4:

[0498] Based on b) above, using image-to-text conversion, we can determine whether there is still text in the comment section. If text exists, it is also considered a "false black screen". Here, the text quantity threshold is set to 15. If the text quantity is less than the text quantity threshold, it means that condition 4 is met.

[0499] If and only if all four conditions above are met in two adjacent frames, then the first frame of the two adjacent frames is regarded as the video frame corresponding to the trigger target access entry.

[0500] In some embodiments, based on a second sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain a second scene frame in the video frame sequence, including:

[0501] In response to the first access scenario, a second mean difference is determined based on the mean grayscale image of two adjacent frames in the remaining frame sequence, wherein the remaining frame sequence includes frames in the video frame sequence that are located after the first scene frame;

[0502] The second scene frame is determined from the remaining frame sequence based on the mean grayscale image of two adjacent frames and the difference between the second mean.

[0503] 1.2, such as Figure 2f As shown, when the first access scenario is a scene where the video interface is accessed from a URL address, the specific steps for scene switching detection processing of the video frame sequence based on the second sub-strategy are as follows:

[0504] As mentioned in section 1.1, clicking the URL to enter the video interface initially turns black. Therefore, the video frame corresponding to the appearance of the video to be played essentially transitions from a black screen. Consequently, the grayscale of the first frame in the remaining frame sequence will be relatively low, while the grayscale of the second frame will increase because video content appears "from nothing to something." Here, the grayscale image mean threshold is set to 30, and the second preset mean difference between adjacent frames is 10: When the grayscale image mean of the second frame in the remaining frame sequence is greater than the grayscale image mean threshold, the grayscale image mean of the first frame in the adjacent frames is less than the grayscale image mean threshold, and the second mean difference between the grayscale image mean of the second frame and the grayscale image mean of the first frame in the adjacent frames is greater than the second preset mean difference, then the second frame in the remaining frame sequence is considered the second scene frame. The second scene frame is the video frame corresponding to the appearance of the video to be played on the video interface.

[0505] In some embodiments, based on a first sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain a first scene frame in the video frame sequence, including:

[0506] In response to the second access scenario, the first mean difference is determined based on the mean grayscale image of two adjacent frames in the video frame sequence;

[0507] The first scene frame is determined from the video frame sequence based on the first mean difference, the grayscale difference between two adjacent frames, and multiple image features of a specific frame in two adjacent frames.

[0508] 2.1 When the second access scenario involves entering the video interface from the video cover image on the content aggregation page, the specific steps for scene switching detection processing of the video frame sequence based on the first sub-strategy are as follows:

[0509] The judgment conditions are the same as the four conditions for the scenario of entering the video interface from the URL address, with the addition of one more condition: Since the video cover in the content aggregation page may be switched by the user swiping the screen left and right to switch the video cover under different themes, in order to avoid such screen swiping being mistakenly judged as the first scene frame, an additional judgment condition is added: Set the first preset mean difference between the mean grayscale image of the second frame of two adjacent frames in the video frame sequence and the mean grayscale image of the first frame to 5. If the first mean difference is less than the first preset mean difference, it is a left and right swipe, not a real click to enter the room.

[0510] In some embodiments, based on a second sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain a second scene frame in the video frame sequence, including:

[0511] In response to the second access scenario, a second mean difference is determined based on the mean grayscale image of two adjacent frames in the remaining frame sequence, wherein the remaining frame sequence includes frames in the video frame sequence that are located after the first scene frame;

[0512] The second scene frame is determined from the remaining frame sequence based on the mean grayscale image of two adjacent frames and the difference between the second mean.

[0513] 2.2 When the second access scenario involves entering the video interface from the video cover image on the content aggregation page, the specific steps for scene switching detection processing of the video frame sequence based on the second sub-strategy are as follows:

[0514] The judgment conditions are exactly the same as those for live streaming sharing.

[0515] In some embodiments, based on a first sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain a first scene frame in the video frame sequence, including:

[0516] In response to a third access scenario, a grayscale pixel difference greater than a second preset pixel difference is determined based on the grayscale pixel difference between two adjacent frames in the video frame sequence in the first preset region.

[0517] The difference in grayscale pixel values ​​between two adjacent frames in the first preset region is determined based on the grayscale pixel difference value greater than the second preset pixel difference value and the grayscale pixel difference value between two adjacent frames in the first preset region.

[0518] Based on the grayscale difference between two adjacent frames in the first preset region, the additional visual element regions in the two adjacent frames are masked to obtain the updated two adjacent frames.

[0519] Pair multiple feature points between two adjacent frames after the update to obtain multiple feature point matching pairs between the two adjacent frames.

[0520] Based on the position information of multiple feature point matching pairs in two adjacent frames, an affine transformation is performed on the multiple feature point matching pairs in two adjacent frames to obtain an affine transformation matrix; based on the affine transformation matrix, the displacement component of the second frame relative to the first frame in the vertical translation direction is determined; based on the position component in the vertical direction, the inter-frame motion degree of the two adjacent frames is determined.

[0521] And based on the position information of the feature point matching pairs in two adjacent frames, determine the displacement distance of the feature point matching pairs in multiple directions; based on the displacement distance of the feature point matching pairs in multiple directions and the preset displacement distance in multiple directions, determine the target feature point matching pair from the multiple feature point matching pairs in two adjacent frames; determine the proportion information of the target feature point matching pair in the multiple feature point matching pairs in two adjacent frames; and determine the inter-frame movement degree of the two adjacent frames based on the proportion information.

[0522] When the inter-frame motion between two adjacent frames is within the range of weak inter-frame motion, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames. The second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0523] Based on the difference in grayscale values ​​between two adjacent frames in the first preset region, multiple feature points of the two adjacent frames in the second preset region are paired to obtain multiple feature point matching pairs of the two adjacent frames.

[0524] Based on the position information of multiple feature point matching pairs in the next two adjacent frames, the inter-frame motion degree of the next two adjacent frames is determined.

[0525] When the inter-frame motion of the next two adjacent frames falls within the range of weak inter-frame motion, the first frame of the two adjacent frames is determined as the first scene frame.

[0526] 3.1, such as Figure 2gAs shown, when the third access scenario involves entering the video interface from a referral video or a video containing a video recommendation entry point, the specific steps for scene switching detection processing of the video frame sequence based on the first sub-strategy are as follows:

[0527] Considering that when a user swipes up to switch to the next video, the overall image will shift upwards. Therefore, we can determine the movement trajectory of a feature point pair by matching feature points between adjacent frames. If the upward movement of the feature points is large enough, it can be considered an upward animation. However, this baseline solution still faces the following problems that need to be addressed:

[0528] a) There are additional visual elements, such as text in the comment section, which act as a distraction. New comments will push the content of the original comment section to the top, creating an "illusion of scrolling up".

[0529] b) Some videos have the illusion of "sliding up" during the shooting process (the image content gradually moves from the bottom of the screen to the top of the screen).

[0530] To solve the above problems, the overall solution is as follows:

[0531] a) Considering the interference caused by the "top-up" phenomenon of "comment text", the comment section is covered up so that no feature point matching pairs are obtained for this part, which will affect the subsequent calculation.

[0532] b) Considering the illusion of "false upward movement" inherent in the video content itself, detection is performed on the first preset area, such as the "avatar" area. This is because if the user swipes up, all content on the screen will move upward, including the "avatar" area. However, if only the video content partially "moves upward," the "avatar" area will remain unchanged. Therefore, a judgment is made first:

[0533] Calculate the grayscale pixel difference between two adjacent frames in the headshot area. Set the second preset pixel difference thresh_1 to 15 and the preset proportion thresh_2 to 0.2. If the absolute value of the grayscale pixel difference in the headshot area is greater than the proportion of thresh_1 to the total number of pixels in the headshot area is greater than thresh_2, then it is considered that a real "upward shift" may have occurred; otherwise, the judgment is exited.

[0534] c) Use a feature point detection algorithm to obtain the feature points of each of two adjacent frames, including a first feature point and a second feature point. The first feature point is the feature point of the first frame in the second preset region, and the second feature point is the feature point of the second frame in the second preset region. Use a feature point matching algorithm (nearest neighbor) to match the feature points of adjacent frames. Furthermore, filter out feature point pairs with low confidence. The filtering mechanism is as follows:

[0535] A ratio test is used to filter matches. If the distance from the first feature point to its nearest matching candidate second feature point is smaller than the distance to its second nearest matching candidate second feature point multiplied by a preset threshold thresh_3 (0.75 in this case), then the match is likely correct. This is because if the distances between the best and second best matching candidate second feature points of the first feature point are very close, it's impossible to determine whether the best matching candidate second feature point is correct. Conversely, if the distance between the best matching candidate second feature point and the second best matching candidate second feature point is much smaller than the distance between the second best matching candidate second feature point, then the best match is likely correct.

[0536] In addition, the second layer of filtering uses the Random Sample Consensus (RANSAC) algorithm for geometric verification, and the specific mechanism is as follows:

[0537] The remaining feature point pairs from b) are used to estimate the homography matrix (the homography matrix reflects the mapping relationship between the first feature point in the first frame and the second feature point of the target in the second frame in two adjacent frames). Then, the homography matrix is ​​used to calculate the error information of each feature point matching pair. If the error information is greater than the preset threshold thresh_4 (5 here), then the feature point matching pair is considered to be an "outlier" because it differs greatly from the homography matrix fitted to this batch of data. Therefore, the feature point matching pair will be filtered out.

[0538] d) Among the remaining feature point matching pairs with high confidence, by calculating the affine transformation matrix and mask matrix from the first feature point in the first frame to the second feature point in the second frame in two adjacent frames, the overall translation direction and corresponding components from the feature points in the first frame to the feature points in the second frame can be obtained. The vertical displacement component is extracted to determine whether the image has shifted upwards as a whole. Here, the thresholds for the upward displacement component are set to 5 for thresh_5 and 40 for thresh_6. If the vertical displacement component is greater than thresh_5 but less than thresh_6, it is considered as a "weak inter-frame motion range". If the vertical displacement component is greater than thresh_6, it is considered as a "strong inter-frame motion range".

[0539] e) Among the remaining feature point matching pairs with high confidence from step a), a predetermined displacement distance in multiple directions is used to indicate that the feature point matching pair has moved upwards. The percentage of target feature point matching pairs meeting this condition out of all feature point matching pairs is calculated. Here, the predetermined percentages are 0.4 for thresh_7 and 0.9 for thresh_8. If the calculated percentage is greater than thresh_7 but less than thresh_8, it is considered a "weak inter-frame motion range"; if it is greater than thresh_8, it indicates a "strong inter-frame motion range".

[0540] f) In summary, this forms the first conditional judgment: if in either d) or e), a branch is judged as "strong upward movement," then it is considered that a "strong upward movement" has indeed occurred between the two adjacent frames. Otherwise, both branches d) and e) must be judged as "weak inter-frame motion range" for it to be considered that a "weak inter-frame motion range" has indeed occurred between the two adjacent frames.

[0541] g) If a "weak inter-frame motion range" occurs between two adjacent frames, it can be considered as the first frame of "live stream single-stream click to enter the room". If a "weak inter-frame motion range" occurs between two adjacent frames, the next frame needs to be continuously detected. Only when two "weak inter-frame motion ranges" occur in three adjacent frames can it be considered as the first frame of "entering the video interface from the referral video or a video containing a video recommendation entry".

[0542] In some embodiments, based on a second sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain a second scene frame in the video frame sequence, including:

[0543] In response to a third access scenario, if the number of edge pixels in the second frame of two adjacent frames in the video frame sequence is less than the preset number of pixels, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames, and the second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0544] Based on the first preset detection area, perform region scanning processing on the next two adjacent frames to obtain the first detection area in the next two adjacent frames.

[0545] Determine multiple image features in the first detection region for the next two adjacent frames;

[0546] Based on the grayscale pixel difference between the next two adjacent frames, determine the grayscale pixel difference that is greater than the third preset pixel difference.

[0547] The grayscale difference between the next two adjacent frames is determined based on the grayscale pixel difference greater than the third preset pixel difference and the grayscale pixel difference between the next two adjacent frames.

[0548] Based on multiple image features in the first detection region of the next two adjacent frames, target feature conditions are determined from multiple preset image feature conditions.

[0549] The scores of the preset conditions corresponding to the target feature conditions are statistically analyzed to obtain the total target score;

[0550] Based on the target total score, the number of edge pixels in the second frame of the next two adjacent frames, and the difference in grayscale values ​​between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0551] When the contour information of the second frame in the next two adjacent frames meets the preset contour conditions, the second frame in the next two adjacent frames is determined as the second scene frame.

[0552] In some embodiments, based on a second sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain a second scene frame in the video frame sequence, including:

[0553] In response to the third access scenario, if the number of edge pixels in the second frame of two adjacent frames in the video frame sequence is not less than the preset number of pixels, the two adjacent frames are scanned based on the second preset detection area to obtain the second detection area in the two adjacent frames.

[0554] Determine multiple image features of two adjacent frames in the second detection region;

[0555] Based on multiple image features of two adjacent frames in the second detection region, the target feature conditions satisfied by the two adjacent frames are determined from multiple preset image feature conditions.

[0556] Based on the target feature conditions satisfied by two adjacent frames, determine the number of edge pixels in the content entry area of ​​the second frame in the two adjacent frames.

[0557] When the number of edge pixels in the content entry area of ​​the second frame in two adjacent frames is less than the preset number of edge pixels, determine multiple consecutive video frames after the second frame in two adjacent frames from the video frame sequence.

[0558] Content recognition processing is performed on multiple consecutive video frames to obtain the video content;

[0559] When the video content does not belong to the preset cover image, the second frame of the two adjacent frames is determined as the second scene frame;

[0560] When the video content belongs to the preset cover image, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames. The second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0561] Based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0562] When the contour information of the second frame in the next two adjacent frames meets the preset contour conditions, the second frame in the next two adjacent frames is determined as the second scene frame.

[0563] 3.2, such as Figure 2h As shown, when the third access scenario involves entering the video interface from a referral video or a video containing a video recommendation entry point, the specific steps for scene switching detection processing of the video frame sequence based on the second sub-strategy are as follows:

[0564] Because in the "instant opening scenario," the video frame corresponding to the triggering of the target access point and the video frame corresponding to the appearance of the video to be played on the video interface are two adjacent frames, meaning that the next video content has already appeared when the user swipes up. Therefore, this application will first determine whether it belongs to the "instant opening scenario," and will proceed according to the "instant opening scenario" judgment branch. If it does not belong to the "instant opening scenario," it will proceed according to the "non-instant opening scenario" judgment branch. The general process is shown in the following diagram:

[0565] For two adjacent frames, first perform edge detection on the video frame of the second frame. If the number of edge pixels in the second frame is less than the preset number of pixels (thresh_1, which is 400 in this case), it means that no image appears in the second frame, so it is not the video frame corresponding to the "video frame when the video to be played appears on the video interface", and the judgment ends at this time. If an image exists, determine whether it belongs to the "instant opening scene".

[0566] The second preset detection area is obtained by taking the pixels in the bottom few rows of the screen (based on the height multiplied by 0.996 to 1.0 according to the screen resolution, or by taking the height multiplied by 0.975 to 0.98). The mean, variance, and sharpness of the grayscale images of two adjacent frames in the second preset detection area are calculated (sharpness is expressed by calculating the Laplacian transform of the grayscale image and returning its variance. A larger variance usually indicates more edge and texture information in the image, while a lower variance indicates a blurry image or a lack of texture). In addition, the difference between the maximum and minimum grayscale values ​​of the second frame in the second preset detection area of ​​two adjacent frames is calculated, as well as the number of edge pixels in the second frame of two adjacent frames in the second preset detection area.

[0567] After obtaining the above characteristics, based on whether "instant opening" refers to switching from the content of the previous video to the content of the next video, or switching from a black screen at the bottom to the content of the next video, a statistical measure was used to express whether either of these two changes occurred. The statistical measure is represented by `count`, which is initialized to 0.

[0568] Regarding the feature of "grayscale image mean", let the grayscale mean of the corresponding region in the first frame and the second frame be cropped-mean-gray1 and cropped-mean-gray2, respectively. If the following conditions are met:

[0569] cropped-mean-gray2-cropped-mean-gray1>10;

[0570] Then increment the count by 2;

[0571] Otherwise, if the following conditions are met:

[0572] cropped-mean-gray2-cropped-mean-gray1 / cropped-mean-gray1≥1.5;

[0573] Then increment the count by 2;

[0574] Otherwise, if the following conditions are met:

[0575] cropped-mean-gray2-cropped-mean-gray1>1;

[0576] Then increment the count by 1;

[0577] Otherwise, if the following conditions are met:

[0578] cropped-mean-gray2-cropped-mean-gray1 / cropped-mean-gray1≥0.8;

[0579] Then increment count by 1.

[0580] Regarding the feature of "grayscale variance", let the grayscale variance of the corresponding region in the first frame and the second frame be cropped-var-gray1 and cropped-var-gray2, respectively. If the following conditions are met:

[0581] |cropped-var-gray2-cropped-var-gray1|>35;

[0582] Then increment the count by 2;

[0583] Otherwise, if the following conditions are met:

[0584] cropped-var-gray2-cropped-var-gray1 / cropped-var-gray1≥1.5;

[0585] Then increment the count by 2;

[0586] Otherwise, if the following conditions are met:

[0587] |cropped-var-gray2-cropped-var-gray1|>5;

[0588] Then increment the count by 1;

[0589] Otherwise, if the following conditions are met:

[0590] cropped-var-gray2-cropped-var-gray1 / cropped-var-gray1≥0.8;

[0591] Then increment count by 1.

[0592] Regarding the feature of "clarity", let the clarity of the corresponding area in the first frame and the second frame be cropped-clear1 and cropped-clear2, respectively. If the following conditions are met:

[0593] |cropped-clear2-cropped-clear1|>20;

[0594] Then increment the count by 2;

[0595] Otherwise, if the following conditions are met:

[0596] cropped-clear2-cropped-clear1 / cropped-clear1≥1.5;

[0597] Then increment the count by 2;

[0598] Otherwise, if the following conditions are met:

[0599] |cropped-clear2-cropped-clear1|>2;

[0600] Then increment the count by 1;

[0601] Otherwise, if the following conditions are met:

[0602] cropped-clear2-cropped-clear1 / cropped-clear1≥0.8;

[0603] Then increment count by 1.

[0604] Based on the above judgment, if the current statistical value is greater than or equal to 3, and the difference between the maximum and minimum grayscale values ​​of the second frame in the second preset detection area of ​​two adjacent frames is greater than or equal to 10, then it can be considered that a significant change has occurred in the bottom area from the first frame to the second frame. On this basis, further constraint is placed that cropped-clear2 is greater than or equal to 2. This condition emphasizes that a change has occurred and that there is indeed an image below. If all three conditions are met, then it can be considered that a new image has appeared below, which meets the "instant scene opening" setting.

[0605] However, the authenticity of the "instant scene opening" still needs further verification. Because the new image only appears in the bottom few rows of pixels on the screen, it is difficult to determine whether the content of the next video has actually appeared, or whether it is a preset cover image with a "frosted glass" effect set by the live stream host (used to temporarily display the live stream cover when the content has not finished loading, and to highly blur the live stream cover).

[0606] To address this situation, first determine if the number of edge pixels in the content entry area of ​​the second frame of two adjacent frames is greater than or equal to 2. If this condition is met, it proves that it is not "frosted glass," because the blurring effect of "frosted glass" would cause the image to lack clear edges. Therefore, this method can be used to quickly filter out images with clear edges, considering them to be the actual scene of the next video, meeting the "true instant opening" requirement, at which point the judgment ends. Otherwise, further judgment is still required.

[0607] Further judgment utilizes content processing models, such as a multimodal large model. Prompt words are designed, and multiple consecutive video frames are input. These frames determine whether the next appearing content belongs to the "frosted glass" live stream cover or the actual content of the next video. If the multimodal large model determines that the new content is real video content, then it's a "true instant opening." Therefore, the second frame of any two adjacent frames can be identified as the video frame corresponding to the appearance of the video to be played on the video interface—the second scene frame—and the judgment is terminated. Conversely, if the multimodal large model determines it's a "frosted glass" effect, then it's a "fake instant opening."

[0608] In the above-mentioned "instant opening" judgment process, if the condition of "true instant opening" is met, the judgment is exited; otherwise, it is considered that "instant opening" has not been achieved, and the judgment branch of the second scene frame of "non-instant opening" needs to be entered.

[0609] Analysis of the second scene frame appearing in "non-instant opening": For the area where new video content appears, the previous frame could be either a black screen or the content of the first video frame. Based on two adjacent frames, the next two adjacent frames are determined from the video frame sequence. The second frame of the next two adjacent frames is the same as the first frame of the next two adjacent frames. Therefore, this application scans three different first preset detection areas for the next two adjacent frames (namely [screen resolution height multiplied by 0.25, screen resolution height multiplied by 0.5], [screen resolution height multiplied by 0.52, screen resolution height multiplied by 0.78], and [screen resolution height multiplied by 0.72, screen resolution height multiplied by 0.985]) to obtain the first detection area, in order to determine whether there is a jump from "black screen" or "first video frame" to "second video frame". To make this judgment, the following parameters are still taken from the first detection area: "number of edge pixels", "mean grayscale image", "grayscale variance", "clarity", and "difference between maximum and minimum grayscale values". The grayscale difference between two adjacent frames is calculated, and the proportion of grayscale pixel difference absolute values ​​greater than the third preset pixel difference (set to 25 here) to the total grayscale pixel difference of the next two adjacent frames is calculated as the diff-rate, thus obtaining the grayscale difference.

[0610] Regarding the characteristic of "grayscale image mean", a statistical measure is used to express whether the above two changes have occurred. The statistic is represented by `count`, initialized to 0. Record the cropped-mean-gray1 and cropped-mean-gray2 of the grayscale image mean in the corresponding first detection region of two adjacent frames (first and second frames). If the following conditions are met:

[0611] cropped-mean-gray2-cropped-mean-gray1>30;

[0612] Or it satisfies:

[0613] cropped-mean-gray2-cropped-mean-gray1 / min(cropped-mean-gray1, cropped-mean-gray1)≥2;

[0614] Then increment count by 1.

[0615] Regarding the feature of "grayscale variance", record the grayscale variances of the first and second frames in the corresponding first detection region as cropped-var-gray1 and cropped-var-gray2, respectively. If the following conditions are met:

[0616] cropped-var-gray2-cropped-var-gray1>800;

[0617] Or it satisfies:

[0618] cropped-var-gray2-cropped-var-gray1 / cropped-var-gray1≥2;

[0619] Then increment count by 1.

[0620] For the feature of "clarity", record the clarity of the first and second frames in the corresponding first detection region as cropped-clear1 and cropped-clear2, respectively. If the following conditions are met:

[0621] cropped-clear2-cropped-clear1>500;

[0622] Or it satisfies:

[0623] cropped-clear2-cropped-clear1 / cropped-clear1≥5;

[0624] Then increment count by 1.

[0625] Regarding the feature of "grayscale maximum / minimum difference", record the grayscale maximum / minimum difference between the first and second frames in the corresponding first detection region as during1 and during2, respectively. If the following conditions are met:

[0626] during2-during1>10;

[0627] And it satisfies:

[0628] during1 < 100;

[0629] Then increment count by 1.

[0630] For the feature of "number of edge pixels", record the number of edge pixels in the corresponding first detection region in the first frame and the second frame of two adjacent frames as edge1 and edge2. If the following conditions are met:

[0631] edge2-edge1>700;

[0632] And it satisfies:

[0633] edge1 < 100;

[0634] Then increment count by 2.

[0635] For diff_rate, if the following conditions are met:

[0636] diff-rate > 0.5;

[0637] Then increment count by 1.

[0638] After judging the above six features, if the statistic is greater than or equal to 4, edge2 is greater than 100, and diff_rate is greater than 0.1, it is considered that the first detection area has changed significantly, so it is very likely to match the change from "black screen" or "first video screen content" to "second video screen".

[0639] In this situation, the title text or other content below the first video might have shifted upwards, causing it to be judged as a significant change in that area, even though the new video content hasn't actually appeared yet. Therefore, it's necessary to avoid text interference.

[0640] Therefore, after performing a series of operations such as Gaussian blur, edge detection, erosion and dilation on the grayscale image of the second frame corresponding to the first detection area, which is two adjacent frames, contour detection is performed, and contours with small areas are filtered out (the threshold is set to 35 here). If the area of ​​the contour is less than 35, it is likely text and not actual image content. The concept of statistical measures is still used here. Let the statistical measure be count. If the area of ​​the contour is greater than 1500, count is incremented by 3; otherwise, if the area of ​​the contour is greater than 700 but less than 1500, count is incremented by 2; otherwise, if the area of ​​the contour is greater than 100, count is incremented by 1.

[0641] If the final statistic is greater than or equal to 3, it means that the second frame of the next two adjacent frames does indeed contain an image in the corresponding first detection area. When the "non-instant opening" judgment branch reaches this point, it means that the first detection area has a "second scene frame". Because this application scans three areas sequentially, as long as one detection area is judged as a "second scene frame", the remaining areas do not need to be scanned.

[0642] 205. Determine the frame spacing based on the positions of the first scene frame and the second scene frame in the video frame sequence.

[0643] 206. Determine the video startup time based on the frame interval and the frame rate of the screen recording video.

[0644] This application can be applied to test the instant-on performance of live streaming, short videos, and long videos. By analyzing screen recordings, the video loading time for users can be obtained, along with the video playback process within the video interface. Currently, this application can be tested on different video platforms. On one hand, this application can accurately measure the video loading time of each video platform, not only correcting the accuracy of statistical data but also promoting product optimization. On the other hand, due to its versatility, it allows for comparison of performance differences across different video platforms, enabling users to understand their own strengths and weaknesses, learn from each other, and further promote product optimization. Figure 2c This is a schematic diagram of the product flow of this application. The detection process for the four different access scenarios is consistent, but the identification methods for "the video frame corresponding to the triggering of the access entry" and "the video frame corresponding to the appearance of the video to be played on the video interface" are different.

[0645] This application proposes a video start-up time detection method based on grayscale image analysis. This method can statistically analyze the video start-up time for different access scenarios, offering advantages such as non-intrusiveness, strong versatility, applicability to various video platforms, and ease of operation. The characteristic manifestation after triggering the access point is the appearance of a video frame; this serves as the starting point and the ending point for measuring video start-up time. Combined with the time point of the initial click on the access point, the video start-up time can be measured. First, video clips containing four access scenarios triggering access points are recorded from a video platform. A screen capture device is used to capture video frames with uniform frame intervals. Therefore, the video start-up time can be calculated by analyzing the frame rate and frame interval by detecting the video frame corresponding to the triggering of the access point and the video frame corresponding to the appearance of the video to be played. The key lies in detecting the video frame corresponding to the triggering of the access point and the video frame corresponding to the appearance of the video to be played. By analyzing the changing characteristics of the initial click on the access point and the changing characteristics of the appearing video frame in the video for the four access scenarios, combined with computer vision algorithms, the first and second scene frames can be identified, thereby calculating the video start-up time.

[0646] In summary, this application presents a non-intrusive, highly versatile, and easy-to-operate solution for calculating video startup time across various video platforms. It only requires analyzing screen recordings to calculate startup time. Firstly, it is non-intrusive because it only requires obtaining the screen recording; no code-level tracking is needed to calculate time points. Secondly, its versatility stems from the fact that this application analyzes video startup time from a video perspective, independent of the platform. Therefore, it can be used on various video platforms and for comparative analysis of video startup time performance across different platforms. Finally, this application is a lightweight, low-cost, simple, and fully automated method that only requires inputting a screen recording to obtain the video startup time.

[0647] To better implement the above methods, this application also provides a detection device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0648] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the detection device as specifically integrated into an electronic device.

[0649] For example, such as Figure 3 As shown, the detection device may include an acquisition unit 301, a framing unit 302, a strategy determination unit 303, a detection unit 304, and a time consumption determination unit 305, as follows:

[0650] (I) Acquisition Unit 301.

[0651] The acquisition unit 301 is used to acquire the screen recording video, which records the process of entering the video interface from the target access point and playing the video on the video interface.

[0652] (ii) Frame segmentation unit 302.

[0653] The framing unit 302 is used to perform frame segmentation processing on the screen recording video to obtain a video frame sequence.

[0654] (III) Strategy Determination Unit 303.

[0655] The strategy determination unit 303 is used to determine the target detection strategy corresponding to the target access entry from multiple detection strategies.

[0656] (iv) Detection unit 304.

[0657] The detection unit 304 is used to perform scene switching detection processing on the video frame sequence based on the target detection strategy to obtain the first scene frame and the second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access entry is triggered, and the second scene frame corresponds to the time point when the video to be played appears on the video interface.

[0658] In some embodiments, the target detection strategy includes a first sub-strategy and a second sub-strategy. The first sub-strategy is used to detect the video frame corresponding to the triggering of the target access entry, and the second sub-strategy is used to detect the video frame corresponding to the appearance of the video interface when the video to be played appears.

[0659] The detection unit includes a first subunit and a second subunit:

[0660] The first subunit is used to perform scene switching detection processing on the video frame sequence based on the first sub-strategy to obtain the first scene frame in the video frame sequence;

[0661] The second subunit is used to perform scene switching detection processing on the video frame sequence based on the second sub-strategy to obtain the second scene frame in the video frame sequence.

[0662] In some embodiments, the first subunit is used for:

[0663] In response to the first access scenario, the grayscale difference between two adjacent frames is obtained based on the grayscale pixel difference between two adjacent frames in the video frame sequence.

[0664] Determine multiple image features of a specific frame in a video frame sequence between two adjacent frames, where the specific frame is one of the two adjacent frames;

[0665] The first scene frame is determined from the video frame sequence based on the difference in grayscale values ​​between two adjacent frames and multiple image features of a specific frame in the two adjacent frames.

[0666] In some embodiments, based on the first sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain the first scene frame in the video frame sequence, which further includes:

[0667] In response to the second access scenario, the first mean difference is determined based on the mean grayscale image of two adjacent frames in the video frame sequence;

[0668] The first scene frame is determined from the video frame sequence based on the first mean difference, the grayscale difference between two adjacent frames, and multiple image features of a specific frame in two adjacent frames.

[0669] In some embodiments, the second subunit is used for:

[0670] Based on the second sub-strategy, scene transition detection processing is performed on the video frame sequence to obtain the second scene frame in the video frame sequence, including:

[0671] The second mean difference is determined based on the mean grayscale image of two adjacent frames in the remaining frame sequence. The remaining frame sequence includes frames in the video frame sequence that are located after the first scene frame.

[0672] The second scene frame is determined from the remaining frame sequence based on the mean grayscale image of two adjacent frames and the difference between the second mean.

[0673] In some embodiments, the first subunit is used for:

[0674] In response to a third access scenario, the difference in grayscale values ​​between two adjacent frames in the first preset region is determined based on the difference in grayscale pixels between two adjacent frames in the video frame sequence in the first preset region.

[0675] Based on the difference in grayscale values ​​between two adjacent frames in the first preset region, multiple feature points of two adjacent frames in the second preset region are paired to obtain multiple feature point matching pairs between two adjacent frames.

[0676] The degree of inter-frame motion between two adjacent frames is determined based on the position information of multiple feature point matching pairs between two adjacent frames.

[0677] The first scene frame is determined from the video frame sequence based on the degree of motion between frames.

[0678] In some embodiments, pairing multiple feature points of two adjacent frames in a second preset region to obtain multiple feature point matching pairs of two adjacent frames includes:

[0679] The additional visual element regions in two adjacent frames are masked to obtain the updated two adjacent frames.

[0680] Multiple feature points of two adjacent frames are paired after the update to obtain multiple feature point matching pairs of two adjacent frames.

[0681] In some embodiments, determining the inter-frame motion degree between two adjacent frames based on the position information of multiple feature point matching pairs between two adjacent frames includes:

[0682] Based on the position information of multiple feature point matching pairs in two adjacent frames, determine the vertical displacement component of the second frame relative to the first frame in the two adjacent frames.

[0683] The degree of motion between adjacent frames is determined based on the position component in the vertical direction.

[0684] In some embodiments, determining the inter-frame motion degree between two adjacent frames based on the position information of multiple feature point matching pairs between two adjacent frames includes:

[0685] Based on the position information of the feature point matching pairs in two adjacent frames, determine the displacement distance of the feature point matching pairs in multiple directions;

[0686] Based on the displacement distance of the feature point matching pair in multiple directions and the preset displacement distance in multiple directions, the target feature point matching pair is determined from multiple feature point matching pairs in two adjacent frames.

[0687] Determine the proportion of target feature point matching pairs in multiple feature point matching pairs in two adjacent frames;

[0688] Based on the proportion information, determine the degree of inter-frame movement between two adjacent frames.

[0689] In some embodiments, determining a first scene frame from a video frame sequence based on the degree of inter-frame motion includes:

[0690] When the inter-frame motion between two adjacent frames is within the range of weak inter-frame motion, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames. The second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0691] Based on the difference in grayscale values ​​between two adjacent frames in the first preset region, multiple feature points of the two adjacent frames in the second preset region are paired to obtain multiple feature point matching pairs of the two adjacent frames.

[0692] Based on the position information of multiple feature point matching pairs in the next two adjacent frames, the inter-frame motion degree of the next two adjacent frames is determined.

[0693] When the inter-frame motion of the next two adjacent frames falls within the range of weak inter-frame motion, the first frame of the two adjacent frames is determined as the first scene frame.

[0694] In some embodiments, the second subunit is used for:

[0695] In response to the third access scenario, if the number of edge pixels in the second frame of two adjacent frames in the video frame sequence is less than the preset number of pixels, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames, and the second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0696] Based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0697] When the contour information of the second frame in the next two adjacent frames meets the preset contour conditions, the second frame in the next two adjacent frames is determined as the second scene frame.

[0698] In some embodiments, based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain contour information of the second frame of the next two adjacent frames, including:

[0699] Based on multiple image features of the next two adjacent frames, the target feature conditions are determined from multiple preset image feature conditions;

[0700] The scores of the preset conditions corresponding to the target feature conditions are statistically analyzed to obtain the total target score;

[0701] Based on the target total score, the number of edge pixels in the second frame of the next two adjacent frames, and the difference in grayscale values ​​between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0702] In some embodiments, the second subunit is used for:

[0703] In response to the third access scenario, if the number of edge pixels of the second frame of two adjacent frames in the video frame sequence is not less than the preset number of pixels, and the number of edge pixels of the second frame in the content entry area of ​​two adjacent frames is less than the preset number of edge pixels, then a number of consecutive video frames after the second frame in two adjacent frames are determined from the video frame sequence. The content entry area refers to the starting area in the video interface where the content to be displayed begins to replace the currently displayed content.

[0704] Content recognition processing is performed on multiple consecutive video frames to obtain the video content;

[0705] Based on the video content, the second frame in a pair of adjacent frames is determined as the second scene frame.

[0706] In some embodiments, determining the second frame of two adjacent frames as the second scene frame based on video content includes:

[0707] When the video content does not belong to the preset cover image, the second frame of the two adjacent frames is determined as the second scene frame;

[0708] When the video content belongs to the preset cover image, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames. The second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames.

[0709] Based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

[0710] When the contour information of the second frame in the next two adjacent frames meets the preset contour conditions, the second frame in the next two adjacent frames is determined as the second scene frame.

[0711] (V) Time Consumption Determination Unit 305.

[0712] The time consumption determination unit 305 is used to determine the video start time based on the first scene frame and the second scene frame.

[0713] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0714] As can be seen from the above, the detection device in this embodiment acquires a screen recording video by an acquisition unit. The screen recording video records the process of entering the video interface from the target access point and playing the video on the video interface. The screen recording video is segmented into frames by a frame segmentation unit to obtain a video frame sequence. The strategy determination unit determines the target detection strategy corresponding to the target access point from multiple detection strategies. Based on the target detection strategy, the detection unit performs scene switching detection processing on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access point is triggered, and the second scene frame corresponds to the time point when the video to be played appears on the video interface. The time consumption determination unit determines the video start time based on the first scene frame and the second scene frame.

[0715] Therefore, the embodiments of this application can improve the accuracy of video startup time detection, help to accurately evaluate user experience, further support the platform for effective optimization, reduce video startup time, improve video startup efficiency, and thus enhance user satisfaction.

[0716] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0717] In some embodiments, the detection device may also be integrated into multiple electronic devices, such as multiple servers, with the detection method of this application being implemented by multiple servers.

[0718] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 4 As shown, it illustrates a schematic diagram of the server structure involved in an embodiment of this application. Specifically:

[0719] The server may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 4 The server architecture shown does not constitute a limitation on the server and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Wherein:

[0720] The processor 401 is the control center of the server, connecting various parts of the server via various interfaces and lines. It performs various server functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 401.

[0721] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0722] The server also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0723] The server may also include an input module 404, which can be used to receive input numeric or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0724] The server may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, through which the server can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0725] Although not shown, the server may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the server loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0726] The screen recording video records the process of entering the video interface from the target access point and playing the video on the video interface.

[0727] The screen recording video is segmented into frames to obtain a video frame sequence;

[0728] Determine the target detection strategy corresponding to the target access point from multiple detection strategies;

[0729] Based on the target detection strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access entry is triggered, and the second scene frame corresponds to the time point when the video interface shows the video to be played.

[0730] The video startup time is determined based on the first scene frame and the second scene frame.

[0731] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0732] As can be seen from the above, this application can improve the accuracy of video startup time detection, help to accurately assess user experience, further support the platform for effective optimization, reduce video startup time, improve video startup efficiency, and thus enhance user satisfaction.

[0733] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0734] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the detection methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0735] The screen recording video records the process of entering the video interface from the target access point and playing the video on the video interface.

[0736] The screen recording video is segmented into frames to obtain a video frame sequence;

[0737] Determine the target detection strategy corresponding to the target access point from multiple detection strategies;

[0738] Based on the target detection strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access entry is triggered, and the second scene frame corresponds to the time point when the video interface shows the video to be played.

[0739] The video startup time is determined based on the first scene frame and the second scene frame.

[0740] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0741] According to one aspect of this application, a computer program product or computer program is provided, comprising a computer program or instructions that, when executed by a processor, implement the steps of the methods provided in the various optional implementations of the detection aspect provided in the above embodiments. The computer program / instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instructions from the computer-readable storage medium and executes the computer program / instructions, causing the electronic device to perform the methods provided in the various optional implementations of the detection aspect provided in the above embodiments.

[0742] Since the instructions stored in the storage medium can execute the steps of any of the detection methods provided in the embodiments of this application, the beneficial effects that any of the detection methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0743] The foregoing has provided a detailed description of a detection method, apparatus, electronic device, storage medium, and program product provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A detection method, characterized in that, include: Acquire screen recording video, which records the process of entering the video interface from the target access point and playing the video on the video interface; The screen recording video is segmented into frames to obtain a video frame sequence; Determine the target detection strategy corresponding to the target access point from multiple detection strategies; Based on the target detection strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame and the second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access entry is triggered, and the second scene frame corresponds to the time point when the video interface shows the video to be played. The video startup time is determined based on the first scene frame and the second scene frame.

2. The detection method as described in claim 1, characterized in that, The target detection strategy includes a first sub-strategy and a second sub-strategy. The first sub-strategy is used to detect the video frame corresponding to the triggering of the target access entry, and the second sub-strategy is used to detect the video frame corresponding to the appearance of a video to be played on the video interface. The step of performing scene transition detection processing on the video frame sequence based on the target detection strategy to obtain the first scene frame and the second scene frame in the video frame sequence includes: Based on the first sub-strategy, scene switching detection processing is performed on the video frame sequence to obtain the first scene frame in the video frame sequence; Based on the second sub-strategy, scene switching detection processing is performed on the video frame sequence to obtain the second scene frame in the video frame sequence.

3. The detection method as described in claim 2, characterized in that, The step of performing scene switching detection processing on the video frame sequence based on the first sub-strategy to obtain the first scene frame in the video frame sequence includes: In response to the first access scenario, the grayscale difference between two adjacent frames is obtained based on the grayscale pixel difference between two adjacent frames in the video frame sequence. Determine multiple image features of a specific frame in two adjacent frames in the video frame sequence, wherein the specific frame is one of the two adjacent frames; The first scene frame is determined from the video frame sequence based on the grayscale difference between two adjacent frames and multiple image features of a specific frame in the two adjacent frames.

4. The detection method as described in claim 3, characterized in that, The step of performing scene switching detection processing on the video frame sequence based on the first sub-strategy to obtain the first scene frame in the video frame sequence further includes: In response to the second access scenario, a first mean difference is determined based on the mean grayscale image of two adjacent frames in the video frame sequence; The first scene frame is determined from the video frame sequence based on the first mean difference, the grayscale difference between the two adjacent frames, and multiple image features of a specific frame in the two adjacent frames.

5. The detection method as described in claim 3 or 4, characterized in that, The step of performing scene switching detection processing on the video frame sequence based on the second sub-strategy to obtain the second scene frame in the video frame sequence includes: The second mean difference is determined based on the mean grayscale image of two adjacent frames in the remaining frame sequence, wherein the remaining frame sequence includes frames in the video frame sequence that are located after the first scene frame; The second scene frame is determined from the remaining frame sequence based on the mean grayscale image of two adjacent frames in the remaining frame sequence and the difference between the two mean values.

6. The detection method as described in claim 2, characterized in that, The step of performing scene switching detection processing on the video frame sequence based on the first sub-strategy to obtain the first scene frame in the video frame sequence includes: In response to a third access scenario, the grayscale difference between two adjacent frames in the first preset region is determined based on the grayscale pixel difference between two adjacent frames in the video frame sequence in the first preset region. Based on the difference in grayscale values ​​between two adjacent frames in the first preset region, multiple feature points of the two adjacent frames in the second preset region are paired to obtain multiple feature point matching pairs of the two adjacent frames. Based on the position information of multiple feature point matching pairs between two adjacent frames, the inter-frame motion degree of the two adjacent frames is determined. The first scene frame is determined from the video frame sequence based on the degree of inter-frame motion.

7. The detection method as described in claim 6, characterized in that, Determining the first scene frame from the video frame sequence based on the inter-frame motion degree includes: When the inter-frame motion of the two adjacent frames is within the range of weak inter-frame motion, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames, and the second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames. Based on the difference in grayscale values ​​between the next two adjacent frames in the first preset region, multiple feature points of the next two adjacent frames in the second preset region are paired to obtain multiple feature point matching pairs of the next two adjacent frames. Based on the position information of multiple feature point matching pairs in the next two adjacent frames, the inter-frame motion degree of the next two adjacent frames is determined. When the inter-frame motion of the next two adjacent frames falls within the range of weak inter-frame motion, the first frame of the two adjacent frames is determined as the first scene frame.

8. The detection method according to any one of claims 2, characterized in that, The step of performing scene switching detection processing on the video frame sequence based on the second sub-strategy to obtain the second scene frame in the video frame sequence includes: In response to a third access scenario, if the number of edge pixels in the second frame of two adjacent frames in the video frame sequence is less than a preset number of pixels, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames, wherein the second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames. Based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames. When the contour information of the second frame in the next two adjacent frames meets the preset contour conditions, the second frame in the next two adjacent frames is determined as the second scene frame.

9. The detection method as described in claim 8, characterized in that, The step of performing contour detection processing on the second frame of the next two adjacent frames based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames includes: Based on multiple image features of the next two adjacent frames, target feature conditions are determined from multiple preset image feature conditions; The preset condition scores corresponding to the target feature conditions are statistically analyzed to obtain the total target score. Based on the target total score, the number of edge pixels in the second frame of the next two adjacent frames, and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames.

10. The detection method as described in claim 2, characterized in that, The step of performing scene switching detection processing on the video frame sequence based on the second sub-strategy to obtain the second scene frame in the video frame sequence includes: In response to a third access scenario, if the number of edge pixels of the second frame of two adjacent frames in the video frame sequence is not less than a preset number of pixels, and the number of edge pixels of the second frame in the content entry area of ​​the two adjacent frames is less than the preset number of edge pixels, then a number of consecutive video frames after the second frame in the two adjacent frames are determined from the video frame sequence. The content entry area refers to the starting area in the video interface where the content to be displayed begins to replace the currently displayed content. Content recognition processing is performed on the consecutive video frames to obtain the video content; Based on the video content, the second frame among the two adjacent frames is determined as the second scene frame.

11. The detection method as described in claim 10, characterized in that, The step of determining the second frame of two adjacent frames as the second scene frame based on the video content includes: When the video content does not belong to the preset cover image, the second frame of the two adjacent frames is determined as the second scene frame; When the video content belongs to a preset cover image, the next two adjacent frames are determined from the video frame sequence based on the two adjacent frames, and the second frame of the two adjacent frames is the same as the first frame of the next two adjacent frames; Based on multiple image features of the next two adjacent frames and the grayscale difference between the next two adjacent frames, contour detection processing is performed on the second frame of the next two adjacent frames to obtain the contour information of the second frame of the next two adjacent frames. When the contour information of the second frame in the next two adjacent frames meets the preset contour conditions, the second frame in the next two adjacent frames is determined as the second scene frame.

12. A detection device, characterized in that, include: The acquisition unit is used to acquire screen recording video, which records the process of entering the video interface from the target access point and playing the video on the video interface; The frame segmentation unit is used to segment the screen recording video into frames to obtain a video frame sequence. A strategy determination unit is used to determine, from multiple detection strategies, the target detection strategy corresponding to the target access point; The detection unit is used to perform scene switching detection processing on the video frame sequence based on the target detection strategy to obtain a first scene frame and a second scene frame in the video frame sequence. The first scene frame corresponds to the time point when the target access entry is triggered, and the second scene frame corresponds to the time point when the video interface shows the video to be played. The time consumption determination unit is used to determine the video start-up time based on the first scene frame and the second scene frame.

13. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps of the detection method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the detection method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps of the detection method according to any one of claims 1 to 11.

Images