Page detection method and device for target application, and electronic device

Abstract

Embodiments of the present application provide a page detection method and device for a target application and an electronic device, wherein the method comprises: obtaining a target video, and then performing frame processing on the target video to determine a page image sequence, so that static component features can be extracted from the page image sequence, and time sequence analysis is performed on the page image sequence to determine dynamic interaction features, so that the page detection result can be determined by comprehensively considering the static component features and the dynamic interaction features, and the accuracy of the detection result is effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method and apparatus for detecting pages of a target application, and an electronic device. Background Technology

[0002] With the development of computer technology, application interfaces have become increasingly sophisticated. These interfaces not only contain numerous static page elements (such as buttons, text, and icons) but also a wealth of dynamic content and interactive effects. To ensure a positive user experience and visual fidelity after application release, interface testing has become an indispensable step.

[0003] However, in related technologies, interface detection schemes mainly rely on static screenshot comparison technology, that is, static mode based on screenshot comparison, which captures a single frame image in a static state for pixel-level or feature matching. The detection results obtained by this mode are relatively inaccurate.

[0004] Therefore, accurately determining the page detection results has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for detecting pages of a target application, in order to at least solve the problem in the related art where detection based on screenshot comparison in a static mode results in low accuracy of page detection results.

[0006] According to one embodiment of this application, a page detection method for a target application is provided, comprising: acquiring a target video to be processed; performing frame segmentation processing on the target video, and determining a page image sequence based on the frame segmentation results; extracting static component features from the page image sequence, and performing temporal analysis on the page image sequence based on the interaction state corresponding to the page image sequence to determine dynamic interaction features; performing page detection on the target application based on the static component features and the dynamic interaction features, and determining the page detection result of the target application.

[0007] According to another embodiment of this application, a page detection device for a target application is provided, comprising: an acquisition module for acquiring a target video to be processed; a first determination module for performing frame segmentation processing on the target video and determining a page image sequence based on the frame segmentation results; a second determination module for extracting static component features from the page image sequence and performing time-series analysis on the page image sequence based on the interaction state corresponding to the page image sequence to determine dynamic interaction features; and a detection module for performing page detection on the target application based on the static component features and the dynamic interaction features, and determining the page detection result of the target application.

[0008] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0009] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0010] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0011] According to this application, by acquiring a target video and then performing frame-by-frame processing on the target video to determine the page image sequence, static component features can be extracted from the page image sequence, and dynamic interaction features can be determined by performing temporal analysis on the page image sequence. Thus, the page detection results can be determined by combining static component features and dynamic interaction features, effectively improving the accuracy of the detection results. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0014] Figure 1 This is a hardware structure block diagram of a computer device for a page detection method of a target application according to an embodiment of this application;

[0015] Figure 2 This is one of the flowcharts of a page detection method for a target application according to an embodiment of this application;

[0016] Figure 3 This is a second flowchart of a page detection method for a target application according to an embodiment of this application;

[0017] Figure 4 This is the third flowchart of a page detection method for a target application according to an embodiment of this application;

[0018] Figure 5This is the fourth flowchart of a page detection method for a target application according to an embodiment of this application;

[0019] Figure 6 This is the fifth flowchart of a page detection method for a target application according to an embodiment of this application;

[0020] Figure 7 This is a structural block diagram of a page detection device for a target application according to an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0023] The methods and embodiments provided in this application can be executed in a computer device or similar computing device. Taking running on a computer device as an example, Figure 1 This is a hardware structure block diagram of a computer device for a page detection method of a target application according to an embodiment of this application. For example... Figure 1 As shown, a computer device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor or programmable logic device) and a memory 104 for storing data are also shown. The computer device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer device described above. For example, the computer device may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0024] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the page detection method of the target application in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to computer devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer equipment. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0026] This embodiment provides a page detection method for a target application, applied to the aforementioned computer device. Figure 2 This is a flowchart of a page detection method for a target application according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0027] Step S202: Obtain the target video to be processed.

[0028] Optionally, the target video is a video used to record page changes during the operation of the target application. When determining the target video, operation commands can be sent to the target application through an automated testing framework or mobile testing tools to trigger jumps, click interactions, or data loading behaviors. A screen recording service is simultaneously started in the background to capture screen pixel data at the set resolution and frame rate, generating the target video. The target video is a time-series data carrier used to completely record the dynamic evolution of the target application from its initial state to its final state. Specifically, the video content of the target video can include: loading animations, transition effects, pop-up appearance and disappearance, list scrolling and refreshing, and all visual changes such as button press states and loading status indicators.

[0029] Step S204: Perform frame segmentation on the target video and determine the page image sequence based on the frame segmentation results.

[0030] In an exemplary embodiment, the target video is segmented into frames, and the page image sequence is determined based on the segmentation results. This includes: segmenting the target video into frames and extracting a candidate frame sequence from the target video; calculating the similarity between adjacent candidate frames in the candidate frame sequence to obtain a similarity sequence; performing initial screening on the candidate frame sequence based on the similarity sequence to determine an initial keyframe sequence; performing deduplication on the initial keyframe sequence and determining the page image sequence based on the deduplication results.

[0031] Optionally, when performing frame segmentation on the target video, a preset frame segmentation strategy can be matched according to the type (such as homepage, details page, list page). For example, for live streams, a keyframe extraction strategy is used, focusing on capturing significant changes in the image; while for settings pages, a stable frame sampling strategy is used, focusing on capturing state transition nodes. Based on the frame segmentation strategy, candidate frame sequences containing potential state changes are initially screened from the target video to eliminate a large number of redundant static or transitional frames. The candidate frame sequence is traversed, and the visual similarity between adjacent candidate frames is calculated pairwise using structural similarity index or histogram comparison algorithms to generate a similarity sequence. The similarity sequence reflects the degree of drastic change in content over time: frames with high similarity indicate static or minor adjustments in the image, while frames with abrupt changes in similarity correspond to key state transition points such as transitions, pop-up appearances, or data refreshes.

[0032] Specifically, for the candidate frame sequence, the similarity between adjacent frames can be calculated, such as structural similarity (SSIM). SSIM can more accurately reflect differences in human visual perception, ignoring subtle rendering noise such as anti-aliasing. This generates a similarity sequence, with values ​​closer to 1 indicating a more static image. Next, based on the similarity sequence, mutation and stability detection are performed on the candidate frame sequence to extract keyframes, forming an initial keyframe sequence. Finally, since mutation or stability detection may generate redundant frames, the similarity between adjacent frames is calculated by traversing the initial keyframe sequence. If the similarity is higher than a deduplication threshold (e.g., 0.98), it is considered content duplication. The deduplicated frame sequence is the final page image sequence. This sequence removes dynamic interference and retains key logical nodes of UI interaction, providing a high signal-to-noise ratio data foundation for subsequent state machine alignment and comparison.

[0033] In an exemplary embodiment, preliminary screening of candidate frame sequences based on a similarity sequence to determine an initial keyframe sequence includes: comparing any similarity in the similarity sequence with a preset mutation similarity threshold to obtain a first comparison result; if the first comparison result indicates that any similarity is less than the mutation similarity threshold, determining a change interval from the similarity sequence, comparing the similarity difference between adjacent similarities in the change interval with a preset stable similarity threshold to obtain a second comparison result; and performing preliminary screening of candidate frame sequences based on the first comparison result and the second comparison result to determine the initial keyframe sequence.

[0034] Understandably, the process begins with mutation detection, comparing each similarity value in the similarity sequence with a preset mutation similarity threshold (e.g., 0.85). If a similarity value is less than the threshold, it indicates a significant change between adjacent frames (e.g., transitions, pop-ups), and this location and its surrounding area are marked as a change interval, representing the critical period for dynamic interaction. Next, stability analysis is performed within the interval. For each defined change interval, the difference between adjacent similarities is calculated and compared with a preset stability similarity threshold (e.g., 0.05). If the difference is less than the stability threshold, the similarity remains relatively stable within the interval, indicating the image is stabilizing; a larger difference indicates the image is still undergoing drastic changes. Finally, combining the first and second comparison results, if a frame is at the starting point of a sudden drop in similarity, it is identified as a mutation initiation keyframe, used to capture the moment the interaction begins. If a frame is at the end of the change interval, and the similarity difference between subsequent frames is less than the stability threshold, it is identified as a stability keyframe, representing the final static state after the interaction is completed. Using the above logic, keyframes representing the start of an action and the end of a state are accurately extracted from the candidate frame sequence, forming an initial keyframe sequence. This sequence captures both the turning points of dynamic interactions and preserves the static state of the interaction results, providing a high-quality data foundation for subsequent deduplication and state machine alignment, and effectively avoiding false detections or missed detections caused by severe screen jitter.

[0035] Step S206: Extract static component features from the page image sequence, and perform time-series analysis on the page image sequence based on the interaction state corresponding to the page image sequence to determine dynamic interaction features;

[0036] In an exemplary embodiment, component identification is performed on the component region, and static component features are determined based on the identification results. This includes: segmenting each page image in the page image sequence to determine the dynamic region corresponding to each page image; generating a binary mask image based on each page image and the dynamic region corresponding to each page image; determining the component region from each binary mask image, and extracting the text content of the component region and the text box containing the text content; and extracting features from the text content and the text box according to at least one component feature dimension to determine the static component features.

[0037] For example, each page image in the page image sequence is segmented to determine the dynamic region corresponding to each page image; a binary mask image is generated based on each page image and the dynamic region corresponding to each page image; a component region is determined from each binary mask image, and component recognition is performed on the component region; static component features are determined based on the recognition results.

[0038] Specifically, segmentation and inference can be performed on each frame of the image sequence of the page to be tested. For example, pre-trained convolutional neural networks (such as Mask R-CNN or U-Net models) can be used to perform pixel-level classification of images, accurately identifying unstable dynamic areas such as video playback areas, live streams, and automatic carousels, as well as interference items such as system status bars and floating balls. The identified dynamic interference areas and interference items are marked as "0" (black, i.e., the masked area), and the remaining static page element areas (such as buttons, text boxes, navigation bars, icons, etc.) are marked as "1" (white, i.e., the valid area), generating a binary mask image. The binary mask image achieves complete isolation between dynamic content and temporal interference, ensuring that subsequent comparisons are only for fixed layout structures.

[0039] Furthermore, for the effective regions (marked as 1) in the binary mask image, the static component regions are determined through contour detection or connected component analysis. For text-type components, optical character recognition is used to extract the text content within the component, and the smallest bounding rectangle (text box) surrounding the text is accurately calculated to obtain its coordinate position and size information. Finally, at least one component feature dimension (such as layout geometry, color attributes, and content semantics) is selected to perform feature encoding on the extracted text content and text box: Geometric dimension: extracting layout features such as the relative position, aspect ratio, and spacing from the parent container of the text box; Color dimension: calculating the average color value and contrast of the text and background; Semantic dimension: vectorizing the text content to represent its semantic information. Combining the above multi-dimensional features, a structured static component feature is generated. This feature not only includes visual appearance but also contains position and semantic logic, providing a high-dimensional quantitative basis for subsequent accurate matching, cross-validation, and false alarm filtering with the benchmark design, ensuring that the detection system can distinguish between real layout errors and dynamic content interference.

[0040] Optionally, based on the text content and text boxes, the vertical distance between adjacent text lines, the horizontal spacing between adjacent text blocks within the same line, and the margin between the text block and the text box are calculated. The vertical distance, horizontal spacing, and margin are then converted into a typography feature vector, and the static component features are determined based on this vector. It should be noted that micro-typography geometric calculations are performed based on the extracted text content and text boxes. This includes calculating the vertical distance (line height) between the baselines of adjacent text lines to detect abnormal line spacing; calculating the horizontal spacing (character spacing) between adjacent characters or word blocks within the same line to identify crowded or excessively scattered characters; and calculating the minimum distance (margin) between the edge of the text content and the boundary of the text box to determine the centering or alignment of the text within the container. These geometric parameters accurately reflect whether the design's typography constraints are strictly followed, and are key indicators for detecting layout errors rather than simple pixel differences. The calculated multi-dimensional geometric parameters, such as vertical distance, horizontal spacing, and margin, are normalized and converted into a high-dimensional typography feature vector. This high-dimensional typography feature vector not only contains specific numerical measurements but also implicitly contains the topological relationships within the component. Based on this vector, the final static component features are determined and compared with the standard typography feature vector. Through this deep comparison based on geometric relationships, it is possible to effectively distinguish between subtle visual jitters caused by differences in device font rendering and genuine typography logic errors (such as line break errors and alignment deviations), thereby achieving high-precision static page fidelity detection and significantly reducing the false alarm rate.

[0041] In an exemplary embodiment, performing temporal analysis on the page image sequence based on the interaction states corresponding to the page image sequence to determine dynamic interaction features includes: performing semantic recognition on each page image in the page image sequence to determine the interaction state corresponding to each page image and generating an interaction state sequence; performing temporal analysis on the interaction state sequence to calculate the duration of each interaction state and the transition duration between adjacent interaction states; and constructing dynamic interaction features based on the duration, transition duration, and interaction state sequence.

[0042] Understandably, semantic analysis is performed on each page image in the page image sequence to extract visual features such as interface layout, control state (e.g., button color, loading icon), and text content. Based on these visual features, the corresponding interaction state (e.g., idle, loading, click) for each page image is determined. These labels are arranged chronologically to generate an interaction state sequence, thus transforming the unstructured video stream into a structured state transition record. Secondly, the interaction state sequence undergoes time-dimensional calculations: the number of consecutive frames of the same interaction state is counted, and the actual duration of that interaction state is calculated based on the frame rate. For example, the duration of the loading state is calculated to detect whether the animation is choppy or too fast. Transition duration is calculated: the switching moments between adjacent interaction states are identified, and the time consumed to jump from one interaction state to the next is calculated. For example, the response latency from click to loading is calculated to assess the smoothness of page interaction.

[0043] Furthermore, the above calculation results are integrated to construct dynamic interaction features that include interaction state sequences, durations, and transition times. These dynamic interaction features not only record what happened (interaction state sequences) but also quantify when and how quickly it occurred. This lays the foundation for subsequent accurate identification of dynamic interaction anomalies such as missing states, incorrect transition sequences, and response delays.

[0044] Step S208: Perform page detection on the target application based on static component features and dynamic interaction features, and determine the page detection results of the target application.

[0045] In an exemplary embodiment, page detection of a target application is performed based on static component features and dynamic interaction features to determine the page detection result of the target application. This includes: matching static component features with preset benchmark component features to obtain a static matching result, and comparing dynamic interaction features with a set state machine model to obtain a dynamic matching result; determining a first weight for the static matching result and a second weight for the dynamic matching result; and weighting the static matching result, the first weight, the dynamic matching result, and the second weight to determine the page detection result of the target application based on the weighted result.

[0046] For example, a dual parallel detection is performed to comprehensively evaluate the page fidelity. In the static dimension, extracted static component features (such as coordinates, size, color, and layout vectors) are matched with the baseline component features in the target baseline design generated by 3D automatic addressing. Pixel differences, structural offsets, and color deviations are calculated to generate a quantitative score, i.e., the static matching result. In the dynamic dimension, dynamic interaction features extracted from screen recording keyframes (such as state transition order, animation duration, and interaction feedback timing) are compared with a preset page state machine model to detect missing states, timing errors, or animation stutters, generating a dynamic score, i.e., the dynamic matching result.

[0047] The weights are dynamically determined based on the importance of the detection dimensions. For example, the accuracy of static layout is the foundation of page reconstruction, so static matching results are given the highest weight (e.g., 40%); while the smoothness and logical correctness of dynamic interactions directly affect user experience, so dynamic matching results are given the second highest weight (e.g., 60%). The weights can be adaptively adjusted according to the specific type (e.g., homepage focuses on dynamics, settings page focuses on statics) or business priority to ensure that the detection focus is consistent with business needs.

[0048] The final detection result is generated based on a weighted fusion algorithm. The static matching result is multiplied by the first weight, and the dynamic matching result is multiplied by the second weight; the sum of these two is the comprehensive score. If the comprehensive score is lower than a preset threshold, or if the score of any dimension is lower than the independent rejection threshold, a page bug is identified, and an alarm is triggered. Simultaneously, the system outputs a detailed attribution report, indicating whether the issue is a static layout deviation or a dynamic interaction anomaly. This enables comprehensive, quantitative, closed-loop control of the target application's quality, ensuring that page fidelity meets design specifications.

[0049] In an exemplary embodiment, comparing dynamic interaction features with a set state machine model to obtain a dynamic matching result includes: aligning the interaction state sequence indicated by the dynamic interaction features with the baseline state sequence in the state machine model in terms of timing, and determining the actual start timestamp and actual end timestamp of each interaction state in the interaction state sequence; calculating the actual duration of each interaction state and the actual transition duration between adjacent interaction states based on the actual start timestamp and actual end timestamp; comparing the actual duration with the duration indicated by the dynamic interaction features, and comparing the actual transition duration with the transition duration indicated by the dynamic interaction features to obtain a dynamic matching result.

[0050] It should be noted that the interaction state sequence extracted from the page image sequence is mapped and aligned with the baseline state sequence defined in the state machine model. By matching the state identifiers of the interaction states, the precise position of each state node in the interaction state sequence on the video timeline is determined, thereby obtaining the actual start and end timestamps of each interaction state.

[0051] Secondly, using the calibrated timestamps, the time difference between the start and end timestamps of the same state is calculated, i.e., the total duration of the interactive state on the screen. The time difference between the end timestamp of the current state and the start timestamp of the next state is also calculated, reflecting the response speed of the interactive state transition. Finally, the calculated actual duration and actual transition duration are compared with the duration recorded in the dynamic interaction features: if the difference between the actual duration and the duration indicated by the dynamic interaction features exceeds a preset tolerance range, it can be determined that the state stay is abnormal (such as animation stuttering or flashing). If the difference between the actual transition duration and the transition duration indicated by the dynamic interaction features exceeds a preset tolerance range, it can be determined that the interaction response is delayed or skipped. Combining the above comparison results, the dynamic matching result can be accurately determined, providing a valid basis for determining the page detection result.

[0052] Through the above steps S202-S208, the target video is acquired, and the page image sequence is determined based on the type. In this way, static component features can be extracted from the page image sequence, and time-series analysis of the page image sequence can be performed to determine dynamic interaction features. Thus, the page detection result can be determined by combining the static component features and the dynamic interaction features, effectively improving the accuracy of the detection result.

[0053] Optionally, before determining the page detection results of the target application based on static component features and dynamic interaction features, the page detection method for the target application further includes: obtaining the application version number of the target application, the device resolution parameters of the device where the target application is located, and the routing identifier of the target video; concatenating the application version number, device resolution parameters, and routing identifier to generate an index key; and querying the feature library based on the index key to determine the baseline component features. In other words, before determining the page detection results, a baseline positioning and acquisition process can be executed to ensure the accuracy and adaptability of the comparison object. Specifically, three core dimensions of information of the current test environment are collected in real time: first, the application version number, used to lock the design draft for a specific iteration cycle; second, the device resolution parameters, including screen pixel density (DPI) and physical resolution, used to adapt to the display differences of different terminals; and third, the routing identifier, i.e., the unique path of the page within the application, used to accurately locate specific functional pages.

[0054] Subsequently, the data from the three dimensions are concatenated according to preset rules to generate a three-dimensional index key. This index key serves as a globally unique query credential, eliminating the risk of version mismatch or device incompatibility that may occur when manually selecting a baseline image. This index key is then used for high-speed retrieval within a pre-built feature library. The feature library stores preprocessed standard page component feature data for each version and device type, including baseline coordinates, standard color values, standardized layout parameters, and expected state machine models.

[0055] If the index key matches successfully, the corresponding baseline component features are directly extracted. These features have undergone isomorphic mapping correction for this specific device and version, ensuring they are in the same coordinate system and granularity as the screenshot under test. If no exact match is found for the index key, the version list can be backtracked to the most recent compatible version, or an approximate baseline can be dynamically generated based on a proportional algorithm. This automated addressing mechanism achieves baseline management with zero manual configuration, providing accurate and reliable reference benchmarks for subsequent static component feature matching and dynamic interactive feature comparison. It fundamentally solves the baseline chaos problem under parallel iteration of multiple versions, ensuring the scientific rigor and authority of the detection results.

[0056] To better understand the process of the page detection method for the target application described above, the implementation flow of the page detection method for the target application will be further described below in conjunction with optional embodiments, but this is not intended to limit the technical solution of the embodiments of this application.

[0057] refer to Figure 3 As shown, this application discloses a schematic diagram of the overall process of a page detection method for a target application. The following is a detailed explanation. Figure 3 Detailed explanation: Figure 3 It includes a five-layer linkage architecture: building a design library, data collection, mask preprocessing, multi-dimensional fusion detection, and intelligent attribution access control.

[0058] To build a design draft library, this application establishes a three-dimensional addressing mechanism to automatically derive the baseline based on the current testing environment, achieving zero manual configuration. Specifically, a tree-like directory can be established: business line (e.g., membership) → terminal type (iOS / Android) → page route. Each leaf node carries: the original design draft image, annotation parameters (color value, spacing, font), and a baseline threshold template (e.g., allowing ±2px offset). A version evolution chain (time dimension) records the design draft change history. For example, the order page button in version V5.0 was at the bottom; in version V5.1, it was moved to the top. The system receives the version number of the app under test (e.g., V5.1) and automatically searches the chain for the corresponding version's baseline image, rather than simply retrieving the latest version. A proportional mapping template is established to address the differences in resolution for the same page. For example, a button width of 100pt on a mobile phone maps to 154pt on a TV (600pt), resolving pixel-level comparison failures caused by larger screens.

[0059] refer to Figure 4The diagram shows a flowchart of the data acquisition process. This process involves recording a screen video stream (the target video) to obtain the data to be analyzed. The video stream can cover the complete path from entering the page, clicking, to page navigation. Alternatively, the UI can be automated to control the app to navigate to the target page, wait for rendering to stabilize (no network requests, no animations), and capture a high-resolution screenshot. Further, the video stream is segmented into frames, and structural similarity (SSIM) is calculated frame by frame. When a sudden SSIM change (e.g., page transition) or stabilization (e.g., a pop-up window fully appears) is detected between consecutive frames, it is marked as a keyframe, and intermediate transition frames are discarded, resulting in a keyframe sequence. Finally, the keyframe sequence is labeled as {Frame_001: Idle, Frame_045: Clicked, Frame_060: Loading}. The keyframe sequence is compared with a predefined state machine model (Idle:0, Clicked:1, Loading:2) to check for consistency in timing (e.g., whether the Clicked state is missing in actual testing, or whether the Loading timeout occurs), thus determining dynamic interaction characteristics.

[0060] Further, refer to Figure 5 The diagram shows a flowchart of the page detection process provided in this application embodiment: For the original screenshot, processing such as segmentation, mask generation, and effective field cropping is performed to identify and eliminate dynamic areas of non-UI components, comparing only the static layout. Specifically, a pre-trained CNN (Convolutional Neural Network) can be used to infer the screenshot, identifying the "video playback area," "live stream area," "carousel area," "system status bar," and "floating ball," generating a black and white mask image of the same size as the original image. Black (masked area): dynamic content area (video, carousel). White (effective area): static UI area (buttons, text, background, navigation bar), with subsequent comparisons performed only within the white area. Through feature point matching, the coordinates of controls within the white area are mapped to the standard coordinate system of the design draft, eliminating coordinate drift caused by device resolution differences.

[0061] Further, refer to Figure 6As shown, another embodiment of this application provides a flowchart of the page detection process: The corresponding detection results can be determined from both dynamic and static dimensions. Weighted results are then applied to obtain a weighted result, which is used to filter false positives and determine the final page detection result. Specifically, when conducting detection from the static dimension, the following can be included: 1. Pixels and Structure: Using SSIM+pixel difference, detect missing controls, stretching, and abnormal whitespace. If only this dimension triggers an alarm, but the color / layout is normal, it is determined to be a rendering anti-aliasing difference, and noise reduction filtering is applied. 2. Color Consistency Detection: For video applications, the focus is on detecting brand color deviation. If a color alarm is triggered but the pixel structure is completely consistent, it is determined to be a "device color temperature difference," and corrected using screen calibration parameters. 3. Layout Constraints: Calculate text line height, character spacing, and control margins. If a layout alarm is triggered but the pixel position is correct, it is determined to be a font library rendering difference, and marked as low to medium risk. 4. Extreme Adaptation: Inject dark mode, font enlargement, and weak network simulation parameters to detect layout errors. This is only triggered in extreme modes and does not affect the normal mode score. When conducting dynamic-level detection, it can include: interaction timing detection, comparing the state transition speed and duration of keyframes in screen recording. If the static layout is perfect but the interactive state is missing, it is judged as a serious bug (such as no response when clicking), and the alarm fusion judgment mechanism is directly upgraded. In weighting, interaction timing can be given the highest weight (40%), followed by pixel structure (30%), etc. The overall score = Σ(score of each dimension × weight of each dimension). Only when the overall score is lower than the threshold and at least two dimensions alarm simultaneously is it judged as a real page problem.

[0062] Finally, intelligent attribution processing is implemented. For example, consider the following phenomenon: a button is offset by 3px. Attribution: The control's position remained unchanged in the V5.1 design draft, and it worked correctly on Android, only on iOS. Conclusion: The issue was determined to be a layout constraint error in the iOS development, not a design change. Furthermore, quantitative reports can be generated, outputting JSON / HTML reports. Additionally, if missing interactive states or brand color deviations are detected, merging into the main code is directly blocked. Finally, if developers provide evidence that the design draft has been changed but not yet included in the database, after approval, the new screenshots are automatically updated to the new baseline image, completing the closed loop.

[0063] Based on the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0064] This embodiment also provides a page detection device for a target application, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0065] Figure 7 This is a structural block diagram of a page detection device for a target application according to an embodiment of this application, such as... Figure 7 As shown, the device includes:

[0066] Module 72 is used to acquire the target video to be processed;

[0067] The first determining module 74 is used to perform frame segmentation processing on the target video and determine the page image sequence based on the frame segmentation result;

[0068] The second determining module 76 extracts static component features from the page image sequence and performs time-series analysis on the page image sequence based on the interaction state corresponding to the page image sequence to determine dynamic interaction features.

[0069] The detection module 78 is used to perform page detection on the target application based on the static component features and the dynamic interaction features, and to determine the page detection result of the target application.

[0070] According to the above-mentioned device, by acquiring the target video, the page image sequence is determined based on the type. In this way, static component features can be extracted from the page image sequence, and dynamic interaction features can be determined by performing time-series analysis on the page image sequence. Thus, the page detection result can be determined by combining the static component features and the dynamic interaction features, which effectively improves the accuracy of the detection result.

[0071] In an exemplary embodiment, the first determining module 74 is further configured to perform frame segmentation processing on the target video and extract a candidate frame sequence from the target video: calculate the similarity between adjacent candidate frames in the candidate frame sequence to obtain a similarity sequence; perform initial screening on the candidate frame sequence based on the similarity sequence to determine an initial key frame sequence; perform deduplication processing on the initial key frame sequence, and determine a page image sequence based on the deduplication result.

[0072] In an exemplary embodiment, the first determining module 74 is further configured to compare any similarity in the similarity sequence with a preset mutation similarity threshold to obtain a first comparison result; if the first comparison result indicates that any similarity is less than the mutation similarity threshold, determine a change interval from the similarity sequence, compare the similarity difference between adjacent similarities in the change interval with a preset stable similarity threshold to obtain a second comparison result; and perform preliminary screening of the candidate frame sequence based on the first comparison result and the second comparison result to determine an initial keyframe sequence.

[0073] In an exemplary embodiment, the second determining module 76 is further configured to perform segmentation processing on each page image in the page image sequence to determine the dynamic region corresponding to each page image; generate a binary mask image based on each page image and the dynamic region corresponding to each page image; determine the component region from each binary mask image, perform component recognition on the component region, and determine the static component features based on the recognition result.

[0074] In an exemplary embodiment, the second determining module 76 is further configured to perform segmentation processing on each page image in the page image sequence to determine the dynamic region corresponding to each page image; generate a binary mask image based on each page image and the dynamic region corresponding to each page image; determine a component region from each binary mask image, and extract the text content of the component region and the text box where the text content is located; and perform feature extraction on the text content and the text box according to at least one component feature dimension to determine static component features.

[0075] In an exemplary embodiment, the second determining module 76 is further configured to perform semantic recognition on each page image in the page image sequence, determine the interaction state corresponding to each page image, and generate an interaction state sequence; perform temporal analysis on the interaction state sequence, calculate the duration of each interaction state and the transition duration between adjacent interaction states; and construct dynamic interaction features based on the duration, the transition duration, and the interaction state sequence.

[0076] In an exemplary embodiment, the detection module 78 is further configured to match the static component features with preset benchmark component features to obtain a static matching result, and compare the dynamic interaction features with a set state machine model to obtain a dynamic matching result; determine a first weight of the static matching result, and determine a second weight of the dynamic matching result; and, based on the static matching result, the first weight, the dynamic matching result, and the second weight, calculate the page detection result of the target application according to the weighted result.

[0077] In an exemplary embodiment, the detection module 78 is further configured to: time-align the interaction state sequence indicated by the dynamic interaction feature with the baseline state sequence in the state machine model; determine the actual start timestamp and actual end timestamp of each interaction state in the interaction state sequence; calculate the actual duration of each interaction state and the actual transition duration between adjacent interaction states based on the actual start timestamp and actual end timestamp; compare the actual duration with the duration indicated by the dynamic interaction feature, and compare the actual transition duration with the transition duration indicated by the dynamic interaction feature to obtain a dynamic matching result.

[0078] In one exemplary embodiment, the apparatus further includes a third determining module; the third determining module is configured to obtain the application version number of the target application, the device resolution parameter of the device where the target application is located, and the routing identifier of the target video; concatenate the application version number, the device resolution parameter, and the routing identifier to generate an index key; and query a feature library based on the index key to determine the features of the baseline component.

[0079] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0080] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0081] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0082] S1, Obtain the target video to be processed;

[0083] S2, perform frame segmentation on the target video, and determine the page image sequence based on the frame segmentation results;

[0084] S3, extract static component features from the page image sequence, and perform time-series analysis on the page image sequence based on the interaction state corresponding to the page image sequence to determine dynamic interaction features;

[0085] S4, perform page detection on the target application based on the static component features and the dynamic interaction features, and determine the page detection result of the target application.

[0086] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0087] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0088] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0089] Optionally, in this embodiment, the processor may be configured to perform the following steps according to a computer program:

[0090] S1, Obtain the target video to be processed;

[0091] S2, perform frame segmentation on the target video, and determine the page image sequence based on the frame segmentation results;

[0092] S3, extract static component features from the page image sequence, and perform time-series analysis on the page image sequence based on the interaction state corresponding to the page image sequence to determine dynamic interaction features;

[0093] S4, perform page detection on the target application based on the static component features and the dynamic interaction features, and determine the page detection result of the target application.

[0094] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0095] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0096] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0097] Optionally, in this embodiment, the processor may be configured to perform the following steps according to a computer program:

[0098] S1, Obtain the target video to be processed;

[0099] S2, perform frame segmentation on the target video, and determine the page image sequence based on the frame segmentation results;

[0100] S3, extract static component features from the page image sequence, and perform time-series analysis on the page image sequence based on the interaction state corresponding to the page image sequence to determine dynamic interaction features;

[0101] S4, perform page detection on the target application based on the static component features and the dynamic interaction features, and determine the page detection result of the target application.

[0102] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0103] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0104] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting pages of a target application, characterized in that, include: Obtain the target video to be processed; The target video is segmented into frames, and the page image sequence is determined based on the segmentation results; Static component features are extracted from the page image sequence, and time-series analysis is performed on the page image sequence based on the interaction state corresponding to the page image sequence to determine dynamic interaction features; Based on the static component features and the dynamic interaction features, page detection is performed on the target application to determine the page detection result of the target application.

2. The method according to claim 1, characterized in that, The target video is segmented into frames, and the page image sequence is determined based on the segmentation results, including: The target video is segmented into frames to extract candidate frame sequences. Calculate the similarity between adjacent candidate frames in the candidate frame sequence to obtain a similarity sequence; The candidate frame sequence is initially screened based on the similarity sequence to determine the initial keyframe sequence. The initial keyframe sequence is deduplicated, and the page image sequence is determined based on the deduplication result.

3. The method according to claim 2, characterized in that, The candidate frame sequence is initially screened based on the similarity sequence to determine the initial keyframe sequence, including: Compare any similarity in the similarity sequence with a preset mutation similarity threshold to obtain a first comparison result; If the first comparison result indicates that any similarity is less than the mutation similarity threshold, a change interval is determined from the similarity sequence, and the similarity difference between adjacent similarities in the change interval is compared with a preset stable similarity threshold to obtain a second comparison result. The candidate frame sequence is initially screened based on the first comparison result and the second comparison result to determine the initial key frame sequence.

4. The method according to claim 1, characterized in that, Extracting static component features from the page image sequence includes: The page images in the page image sequence are segmented to determine the dynamic region corresponding to each page image. A binary mask image is generated based on each of the page images and the dynamic regions corresponding to each of the page images. Component regions are determined from each of the binary mask images, and the text content of the component regions and the text boxes containing the text content are extracted. The text content and the text box are subjected to feature extraction according to at least one component feature dimension to determine the static component features.

5. The method according to claim 1, characterized in that, Based on the interaction states corresponding to the page image sequence, a time-series analysis is performed on the page image sequence to determine dynamic interaction features, including: Semantic recognition is performed on each page image in the page image sequence to determine the interaction state corresponding to each page image and generate an interaction state sequence. Perform time-series analysis on the interaction state sequence to calculate the duration of each interaction state and the transition duration between adjacent interaction states; Dynamic interaction features are constructed based on the duration, the flow duration, and the interaction state sequence.

6. The method according to claim 1, characterized in that, Based on the static component features and the dynamic interaction features, page detection is performed on the target application to determine the page detection result of the target application, including: The static component features are matched with preset benchmark component features to obtain static matching results, and the dynamic interaction features are compared with a set state machine model to obtain dynamic matching results. Determine a first weight for the static matching result and a second weight for the dynamic matching result; Based on the static matching result, the first weight, the dynamic matching result, and the second weight, the page detection result of the target application is determined according to the weighted result.

7. The method according to claim 1, characterized in that, The dynamic interaction features are compared with the set state machine model to obtain dynamic matching results, including: The interaction state sequence indicated by the dynamic interaction feature is time-aligned with the baseline state sequence in the state machine model to determine the actual start timestamp and actual end timestamp of each interaction state in the interaction state sequence. Based on the actual start timestamp and the actual end timestamp, calculate the actual duration of each interaction state and the actual transition time between adjacent interaction states. The actual duration is compared with the duration indicated by the dynamic interaction feature, and the actual flow duration is compared with the flow duration indicated by the dynamic interaction feature to obtain a dynamic matching result.

8. The method according to claim 1, characterized in that, Before determining the page detection result of the target application based on the static component features and the dynamic interaction features, the method further includes: Obtain the application version number of the target application, the device resolution parameters of the device where the target application is located, and the routing identifier of the target video; Concatenate the application version number, the device resolution parameter, and the route identifier to generate an index key; The feature library is queried based on the index key to determine the features of the benchmark component.

9. A page detection device for a target application, characterized in that, include: The acquisition module is used to acquire the target video to be processed; The first determining module is used to perform frame segmentation processing on the target video and determine the page image sequence based on the frame segmentation results; The second determining module is used to extract static component features from the page image sequence and perform time-series analysis on the page image sequence based on the interaction state corresponding to the page image sequence to determine dynamic interaction features. The detection module is used to perform page detection on the target application based on the static component features and the dynamic interaction features, and to determine the page detection result of the target application.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to perform the method according to any one of claims 1 to 8.

Images