UI operation feedback duration measuring and calculating method and system based on screen recording analysis

Abstract

The invention relates to the technical field of mobile application testing, in particular to a UI operation feedback duration measuring and calculating method and system based on screen recording analysis. The method comprises the following steps: synchronously acquiring a screen recorded video and a touch event stream, establishing an alignment relationship between a touch timestamp and a video time axis, determining a submission moment of one operation according to a touch event, and mapping the submission moment as an operation starting time point; then, continuous frames after the starting time point are blocked, the inter-frame variation of the blocks is calculated, and a causal region is obtained through screening and iterative expansion according to positive lag correlation by taking the touch blocks as seeds; performing periodic change elimination on the causal region to form an effective region; and constructing an activity intensity sequence based on the effective region, determining a feedback end time point by combining activity intensity convergence, no expansion of the causal region and motion amount inhibition, and calculating UI operation feedback duration. According to the scheme, application burying points are not needed, and the method is suitable for experience index measurement and calculation of multi-device and multi-application complex scenes.

Description

Technical Field

[0001] This invention relates to the field of mobile application testing technology, specifically to a method and system for calculating UI operation feedback duration based on screen recording analysis. Background Technology

[0002] With the widespread adoption of smartphones and mobile internet, the interactive forms of mobile applications have evolved from early static pages and simple navigation to complex interface systems incorporating asynchronous loading, dynamic rendering, skeleton screens, and animated transitions. Testing methods for user experience metrics have also evolved from manual observation and timing to automated data collection and analysis. Early stages typically relied on manual operation and stopwatch recording, which was inefficient and inconsistent. Subsequently, end-to-end testing based on automated scripts emerged, attempting to quantify response and loading times through methods such as log tracking, API latency statistics, or page object tree crawling. In recent years, the diversification of mobile rendering frameworks, the coexistence of native and hybrid development, and the widespread application of WebView, cross-platform frameworks, and game engines have led to a shift in experience metric measurement from single-technology-link statistics to a focus on multi-source data fusion and the retention of visual evidence.

[0003] However, existing technologies still have significant shortcomings in measuring the duration of UI operation feedback. Log-based or event tracking methods often require intrusive modifications to the application, making them difficult to reuse quickly across multiple applications and versions. Furthermore, log timestamps do not always align with the user's visible feedback, failing to objectively reflect the real user experience. Methods relying on control trees or dumps lack observability in custom controls, complex animations, cross-platform rendering, and WebView scenarios, frequently resulting in missing or misjudged key states. While manual frame-by-frame analysis is relatively accurate, it is extremely labor-intensive and difficult to review, hindering large-scale testing. Even with screen recording comparisons, the lack of precise alignment with touch operations, stable extraction of causal regions, and mechanisms to suppress periodic animations and localized continuous motion can lead to unstable identification of the operation's starting point and feedback endpoint, resulting in significant measurement deviations. Therefore, a UI operation feedback duration measurement method and system based on screen recording analysis is needed to address these issues. Summary of the Invention

[0004] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a method and system for calculating UI operation feedback duration based on screen recording analysis, thus solving the aforementioned problems.

[0005] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: a method for calculating UI operation feedback duration based on screen recording analysis, comprising: Step S1: Collect screen recording video and touch event stream of the user's operation process. The touch event stream includes touch event type, touch coordinates and touch timestamp. Step S2: Establish the alignment relationship between the touch timestamp and the screen recording video timeline; Step S3: Determine the submission time of a user operation based on the touch event stream, and map the submission time to the operation start time point through the alignment relationship; Step S4: After the start time of the operation, the continuous frames of the screen recording video are divided into blocks and the inter-frame change of the blocks is calculated. The block corresponding to the touch coordinate is used as the seed block. The causal related block set is obtained by filtering according to the positive hysteresis correlation between the inter-frame change of each block and the inter-frame change of the seed block. The causal related block set is iteratively expanded to obtain the causal region corresponding to the current user operation. Step S5: Perform periodic change elimination on the causal region to obtain the effective region for completing the determination; Step S6: Based on the inter-frame change of the blocks within the effective region, obtain the sequence of activity intensity over time. When the activity intensity and its rate of change meet the convergence condition within a preset duration window, and the causal region no longer expands and the motion within the effective region meets the suppression condition, determine the feedback end time point. Step S7: Calculate the UI operation feedback duration based on the feedback end time and the operation start time.

[0006] Furthermore, step S2 specifically includes: At the start of screen recording, the system monotonic clock time and the timestamp of the first video frame are recorded. An initial time mapping relationship is established based on the system monotonic clock time and the timestamp of the first video frame. During the recording process, the system monotonic clock time and the corresponding video frame timestamp are obtained at preset intervals. The initial time mapping relationship is linearly fitted and corrected to compensate for time offset and time drift.

[0007] Furthermore, the submission time of a user operation is determined as follows: When the touch event type is a click operation, the touch timestamp corresponding to the touch release event is determined as the submission time; When the touch event type is a swipe or pull-to-refresh operation, the touch timestamp corresponding to the touch lift event is determined as the submission time, provided that the touch movement trajectory meets the displacement threshold. When the touch event type is a long press operation, if the duration of the touch press reaches a preset duration threshold, the touch timestamp that reaches the duration threshold is determined as the submission time.

[0008] Furthermore, the inter-frame variation of the blocks is calculated as follows: The continuous frames of the screen-recorded video are converted to grayscale, and each frame is divided into image blocks of a preset size; The pixel grayscale difference between corresponding blocks in two adjacent frames is accumulated and used as the inter-frame change of the corresponding block.

[0009] Furthermore, the causal correlation block set is obtained by filtering according to positive lag correlation, specifically as follows: The inter-frame changes of each block are formed into a change sequence over time. The Pearson correlation coefficient between the change sequence and the change sequence of the seed block is calculated within a preset lag range. The maximum correlation value within the lag range is taken as the correlation of the block. When the maximum correlation value meets a preset threshold and the corresponding lag is greater than zero, the block is determined as a causal related block.

[0010] Furthermore, the causal correlation block set is iteratively expanded, specifically as follows: The mean of the inter-frame change sequence of each block in the causal correlation block set is used as the reference change sequence, wherein the candidate blocks are other blocks besides the causal correlation block set. The causal correlation block set is updated based on the positive hysteresis correlation between the reference change sequence and the inter-frame change sequence of the candidate blocks, until the causal correlation block set no longer adds blocks in at least one consecutive iteration.

[0011] Furthermore, the periodic change elimination is performed on the causal region, specifically as follows: The inter-frame change sequence of each block within the causal region is periodically determined. The periodic determination includes calculating the peak value at non-zero lag using autocorrelation or obtaining the main frequency peak value using frequency domain analysis. When the peak value is greater than a preset period threshold and the duration reaches a preset duration, the corresponding block is determined as a periodically changing block, and the periodically changing block is removed from the causal region to obtain the effective region.

[0012] Furthermore, the sequence of activity intensity over time is the cumulative value of the inter-frame change of each block within the effective region at the same time, and the convergence condition includes the activity intensity being less than an adaptive threshold within a preset duration window and the absolute value of the rate of change of the activity intensity being less than a preset rate of change threshold, wherein the adaptive threshold is a linear combination of the median of the activity intensity within the preset duration window and a dispersion measure, and the dispersion measure includes the median of absolute deviation.

[0013] Furthermore, within the effective area, the mean optical flow modulus of adjacent frames or the directional consistency of the inter-frame pixel difference is calculated. When the mean optical flow modulus is less than a preset motion threshold, the motion amount is determined to meet the suppression condition; or when the directional consistency is less than a preset consistency threshold, the motion amount is determined to meet the suppression condition.

[0014] This invention also provides a UI operation feedback duration calculation system based on screen recording analysis, including: The screen recording and touch capture module is used to capture screen recording videos and touch event streams during user operation. The touch event stream includes touch event type, touch coordinates, and touch timestamp. The time alignment module is used to establish the alignment relationship between the touch timestamp and the screen recording video timeline; The operation start determination module is used to determine the submission time of a user operation based on the touch event stream, and to map the submission time to the operation start time point through the alignment relationship; The block change calculation module is used to divide the continuous frames of the screen recording video into blocks and calculate the inter-frame change after the operation start time point. The block corresponding to the touch coordinate is used as the seed block. The causal related block set is obtained by filtering the positive hysteresis correlation between the inter-frame change of each block and the inter-frame change of the seed block. The causal related block set is iteratively expanded to obtain the causal region corresponding to the current user operation. The causal region construction module is used to perform periodic change elimination on the causal region to obtain a valid region for completing the judgment. The effective region generation module is used to obtain the sequence of activity intensity over time based on the inter-frame change of the blocks within the effective region. When the activity intensity and its rate of change meet the convergence condition within a preset duration window, and the causal region no longer expands and the motion within the effective region meets the suppression condition, the feedback end time point is determined. The feedback end confirmation and duration calculation module is used to calculate the UI operation feedback duration based on the feedback end time and the operation start time.

[0015] (III) Beneficial Effects Compared with existing technologies, this invention provides a method and system for calculating UI operation feedback duration based on screen recording analysis, which has the following beneficial effects: 1. A method and system for measuring UI operation feedback duration based on screen recording analysis. By synchronously collecting screen recording video and touch event streams, and establishing an alignment relationship between touch timestamps and video timelines, the submission time of a user operation is accurately mapped to the start time of the operation. This avoids the invasive modifications, difficulties in cross-application reuse, and inconsistencies between the time point and the user's visible feedback caused by relying solely on logs, event tracking, or control trees. It ensures that the start time point has a unified system-level source and stable repeatability, improving the consistency and objectivity of the measurement in multi-device and multi-application environments.

[0016] 2. A method and system for measuring UI operation feedback duration based on screen recording analysis. This method divides video frames after the start time of the operation into blocks and calculates the changes between the blocks. Based on positive hysteresis correlation, it filters and iteratively expands the seed blocks corresponding to the touch coordinates to obtain causal regions. Within the causal regions, it performs periodic change elimination to obtain effective regions. At the same time, it combines activity intensity convergence and motion suppression to determine the feedback end time. This overcomes the problems of traditional screen recording comparison or full-screen stability judgment being susceptible to interference and unstable endpoint recognition in complex rendering, asynchronous loading, skeleton screen animation, and local continuous motion scenarios. It can eliminate the influence of periodic animation and irrelevant area changes on the measurement results, making the feedback endpoint closer to the time when the user can see the completion of the business feedback, and significantly improving the accuracy and verifiability of UI operation feedback duration measurement. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the UI operation feedback duration calculation method based on screen recording analysis provided by the present invention. Figure 2 This is a schematic diagram of the UI operation feedback duration calculation system based on screen recording analysis provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Please see Figure 1 , Figure 1 This is a flowchart illustrating the UI operation feedback duration calculation method based on screen recording analysis provided by the present invention; the UI operation feedback duration calculation method based on screen recording analysis includes: Step S1: Collect screen recording video and touch event stream of the user's operation process. The touch event stream includes touch event type, touch coordinates and touch timestamp. Step S2: Establish the alignment relationship between the touch timestamp and the screen recording video timeline; Step S3: Determine the submission time of a user operation based on the touch event stream, and map the submission time to the operation start time point through the alignment relationship; Step S4: After the start time of the operation, the continuous frames of the screen recording video are divided into blocks and the inter-frame change of the blocks is calculated. The block corresponding to the touch coordinate is used as the seed block. The causal related block set is obtained by filtering according to the positive hysteresis correlation between the inter-frame change of each block and the inter-frame change of the seed block. The causal related block set is iteratively expanded to obtain the causal region corresponding to the current user operation. Step S5: Perform periodic change elimination on the causal region to obtain the effective region for completing the determination; Step S6: Based on the inter-frame change of the blocks within the effective region, obtain the sequence of activity intensity over time. When the activity intensity and its rate of change meet the convergence condition within a preset duration window, and the causal region no longer expands and the motion within the effective region meets the suppression condition, determine the feedback end time point. Step S7: Calculate the UI operation feedback duration based on the feedback end time and the operation start time.

[0021] In this embodiment, in order to realize the UI operation feedback duration calculation based on screen recording analysis, screen recording video and touch event stream are first collected synchronously during the process of the user completing a business operation. The touch event stream records at least the touch event type, touch coordinates and touch timestamp. The screen recording video is used to reflect the interface change process visible to the user. The touch event stream is used to provide the operation timing basis that is directly related to the user operation and does not depend on the application's internal embedding points. Subsequently, an alignment relationship between the touch timestamp and the video timeline is established, enabling touch events to be mapped to the time position of video frames. This maps the submission time of a user operation to the start time of the operation, avoiding the deviation caused by inferring the starting point solely from screen changes or control trees. After the start time of the operation, the continuous video frames are grayscaled and segmented, and the inter-frame changes of each segment are calculated. This transforms the interface changes from continuous images into quantifiable temporal change signals. The segment corresponding to the touch coordinates is used as the seed segment. The set of causally related segments is selected based on the positive hysteresis correlation between the changes of each segment and the changes of the seed segment. The positive hysteresis is used to constrain the retention of only the areas that respond after the user operation, thereby reducing the interference of irrelevant changes such as status bars, floating controls, or background animations. Based on this, the causal-related block set is iteratively expanded, and blocks that maintain a causal relationship with the identified response area in terms of time change pattern are gradually included to obtain the causal area corresponding to this user operation, so as to depict the real feedback area such as page transition, loading and content filling triggered by this operation. Further, the causal region is processed to eliminate periodic changes, removing continuously changing blocks such as carousel ads and animated images, forming an effective region for completing the judgment, making the endpoint judgment more stable; Then, based on the inter-frame changes in the effective region, a sequence of activity intensity over time is constructed to characterize the strength of the interface feedback in the time dimension. When the activity intensity and its rate of change converge within the preset duration window, and the causal region no longer expands and the motion in the effective region meets the suppression condition, the feedback end time point is determined. Thus, even in complex scenarios such as asynchronous loading, gradual animation, and phased loading of images and text, the endpoint that is close to the user's visible experience completion state can still be obtained. The UI operation feedback time is ultimately calculated by the difference between the feedback end time and the operation start time, enabling automated and verifiable measurement of the time from user operation to business feedback across multiple devices, applications, and scenarios.

[0022] Furthermore, step S2 specifically includes: At the start of screen recording, the system monotonic clock time and the timestamp of the first video frame are recorded. An initial time mapping relationship is established based on the system monotonic clock time and the timestamp of the first video frame. During the recording process, the system monotonic clock time and the corresponding video frame timestamp are obtained at preset intervals. The initial time mapping relationship is linearly fitted and corrected to compensate for time offset and time drift.

[0023] In this embodiment, to ensure that the touch event stream and the screen recording video can accurately correspond under the same time base, thereby supporting the accurate positioning of the start time of subsequent operations, step S2 establishes an alignment relationship by combining the system monotonic clock and the video timestamp: At the start of screen recording, the system monotonic clock time and the timestamp of the first frame of the video are recorded once each. The two are used to form an initial time mapping relationship so that the timestamp of the touch event can be converted into the moment on the video timeline. Considering that different devices may have factors such as encoding buffering, thread scheduling, and clock drift during the recording process, which may cause the mapping relationship to shift over time, during implementation, the system monotonic clock time and the corresponding video frame timestamp are continuously sampled at preset intervals during the recording process to form a set of time pair samples. Based on these samples, the initial mapping relationship is linearly fitted and corrected to compensate for fixed offsets and slow drifts, so that the error of the touch event mapping to the video frame is kept within a controllable range. The above methods can achieve precise alignment between operations and screen without relying on application internal logs or event tracking, and can also improve stability in multi-duration recording and multi-device environments.

[0024] Furthermore, the submission time of a user operation is determined as follows: When the touch event type is a click operation, the touch timestamp corresponding to the touch release event is determined as the submission time; When the touch event type is a swipe or pull-to-refresh operation, the touch timestamp corresponding to the touch lift event is determined as the submission time, provided that the touch movement trajectory meets the displacement threshold. When the touch event type is a long press operation, if the duration of the touch press reaches a preset duration threshold, the touch timestamp that reaches the duration threshold is determined as the submission time.

[0025] In this embodiment, to ensure that the start time of the operation is consistent with the actual business trigger time, and to avoid mistaking the moment of pressing, the sliding process, or an invalid touch as a valid start point, differentiated rules for determining the submission time are adopted for different touch event types: For click operations, the click semantics are usually triggered by the touch lift event and the business request is submitted. Therefore, the touch timestamp corresponding to the touch lift event is determined as the submission time, so that the calculation start point is consistent with the actual submission when the user completes a click. For swipe or pull-to-refresh operations, there may be cases of accidental touch with only slight displacement or scrolling failure during the swipe process. To improve effectiveness, the operation is only considered successful when the touch movement trajectory meets the displacement threshold. The touch timestamp corresponding to the touch release event is determined as the submission time, so that the starting point falls on the position where the user releases the swipe and the list refresh or page scrolling inertia is actually triggered. For long press operations, the long press semantics are usually established and the corresponding function is triggered when the press duration reaches a preset duration threshold. Therefore, under the condition that the touch press duration reaches the duration threshold, the touch timestamp that reaches the duration threshold is determined as the submission time, so that the starting point can reflect the long press trigger point rather than the release point. Through the above classification rules, this solution can uniformly obtain the submission time with clear business semantics under different interaction methods, and then map it to the operation start time point through time alignment, providing a stable and consistent starting point benchmark for subsequent causal region construction and feedback end determination.

[0026] Furthermore, the inter-frame variation of the blocks is calculated as follows: The continuous frames of the screen-recorded video are converted to grayscale, and each frame is divided into image blocks of a preset size; The pixel grayscale difference between corresponding blocks in two adjacent frames is accumulated and used as the inter-frame change of the corresponding block.

[0027] In this embodiment, in order to convert the interface changes in the screen recording video into quantifiable and comparable time-series signals for subsequent causal correlation screening and convergence determination, the continuous frames of the screen recording video are first grayscale processed to reduce color differences, compress noise dimensions and improve computational efficiency. Each frame is then divided into multiple image blocks of a preset size, so that changes in the global screen are decomposed into changes in local areas. This makes it easier to locate the response area triggered by user operation and also suppresses the influence of irrelevant local changes on the overall judgment. Based on this, the pixel grayscale difference is calculated for the blocks corresponding to the positions in two adjacent frames and accumulated to obtain the inter-frame change of the block at that time step. This change can characterize the intensity of the brightness change of the block between adjacent frames, and thus reflect visual changes such as the appearance, disappearance, movement or content filling of interface elements. By using the above-mentioned block-based and differential accumulation processing methods, stable change metrics can be obtained without relying on control trees, logs, or semantic recognition.

[0028] Furthermore, the causal correlation block set is obtained by filtering according to positive lag correlation, specifically as follows: The inter-frame changes of each block are formed into a change sequence over time. The Pearson correlation coefficient between the change sequence and the change sequence of the seed block is calculated within a preset lag range. The maximum correlation value within the lag range is taken as the correlation of the block. When the maximum correlation value meets a preset threshold and the corresponding lag is greater than zero, the block is determined as a causal related block.

[0029] In this embodiment, in order to automatically locate the response area that is causally related to the current user operation from the screen recording, and to avoid including background animations, status bar changes or other area disturbances that are unrelated to the operation in the calculation range, the inter-frame change of each block is constructed as a change sequence along the time dimension, and the seed block change sequence corresponding to the touch coordinate is used as a reference signal. Then, within a preset lag range, the Pearson correlation coefficient of any block change sequence relative to the seed block change sequence is calculated. By comparing the correlation coefficients under different lags and taking the maximum correlation value as the correlation of the block, the delay characteristics of the interface response relative to the user touch can be captured. Furthermore, the maximum correlation value is required to meet a preset threshold and the corresponding lag is greater than zero, in order to constrain the change of the block to occur after the touch rather than before, thereby ensuring in principle that the selected blocks are more in line with the temporal causal relationship of "triggered by operation and then generating response"; The segments that meet the above conditions are identified as causal segments and form a set of causal segments. This set can cover areas such as transition animations after a button is pressed, the appearance of loading controls, and changes in the main content of the page.

[0030] Furthermore, the causal correlation block set is iteratively expanded, specifically as follows: The mean of the inter-frame change sequence of each block in the causal correlation block set is used as the reference change sequence, wherein the candidate blocks are other blocks besides the causal correlation block set. The causal correlation block set is updated based on the positive hysteresis correlation between the reference change sequence and the inter-frame change sequence of the candidate blocks, until the causal correlation block set no longer adds blocks in at least one consecutive iteration.

[0031] In this embodiment, in order to enable the response area to gradually expand from local changes near the touch to the main page change area triggered by the current operation, thereby adapting to various feedback forms such as page jump, list filling, and pop-up appearance, after obtaining the initial causal related block set, the inter-frame change sequence of each block in the set is first aggregated, and the average value is taken to form a reference change sequence. The reference change sequence is used to characterize the overall temporal change pattern of the currently identified response area. Then, other blocks besides the causal correlation block set are used as candidate blocks. The positive hysteresis correlation between the inter-frame change sequence of the candidate blocks and the reference change sequence is calculated. Candidate blocks that meet the preset judgment conditions are incorporated into the causal correlation block set to expand the response region. By continuously incorporating newly included blocks into the update of the reference change sequence, the expansion process can gradually absorb regions with consistent temporal change characteristics with the identified response regions, thereby forming a causal region covering the key interface changes caused by this operation. When the set of causal related blocks no longer adds blocks in at least one consecutive iteration, it indicates that the response blocks that can be included under the current correlation criterion have converged. The iterative expansion stops and the final causal region is output, thereby obtaining a stable response range related to user operation without relying on control semantics and business rules.

[0032] Furthermore, the periodic change elimination is performed on the causal region, specifically as follows: The inter-frame change sequence of each block within the causal region is periodically determined. The periodic determination includes calculating the peak value at non-zero lag using autocorrelation or obtaining the main frequency peak value using frequency domain analysis. When the peak value is greater than a preset period threshold and the duration reaches a preset duration, the corresponding block is determined as a periodically changing block, and the periodically changing block is removed from the causal region to obtain the effective region.

[0033] In this embodiment, to avoid the delay or misjudgment caused by areas with continuous periodic movement, such as carousel ads, animated images, and partial video previews, a periodic change elimination process is further performed after the causal region is formed: Periodicity discrimination is performed on the inter-frame change sequence of each block within the causal region. The periodicity discrimination uses autocorrelation to calculate the peak value of the change sequence at non-zero lag to determine whether there is a stable and repeating time structure. Alternatively, frequency domain analysis can be used to obtain the main frequency peak value to determine whether there is a significant periodic principal component. When the above peak value is greater than the preset periodic threshold and the duration of the periodic feature reaches the preset duration, it is considered that the change corresponding to the block belongs to a long-term periodic change rather than a loading process of a single operation. Therefore, the block is identified as a periodic change block and removed from the causal region to obtain the effective region for completing the judgment. By eliminating the continuously changing blocks, the long-term disturbance of the activity intensity sequence by periodic animations can be significantly reduced without changing the main response range of the causal region, making it easier for subsequent convergence determination to converge to the time when the user can perceive the completion of the business feedback.

[0034] Furthermore, the sequence of activity intensity over time is the cumulative value of the inter-frame change of each block within the effective region at the same time, and the convergence condition includes the activity intensity being less than an adaptive threshold within a preset duration window and the absolute value of the rate of change of the activity intensity being less than a preset rate of change threshold, wherein the adaptive threshold is a linear combination of the median of the activity intensity within the preset duration window and a dispersion measure, and the dispersion measure includes the median of absolute deviation.

[0035] In this embodiment, in order to characterize the overall strength of changes in the effective area over time with a unified quantitative index and determine whether the business feedback has entered the completion state, the inter-frame changes of each block within the effective area at the same time are accumulated to obtain the sequence of activity intensity over time. The greater the activity intensity, the more the interface is still in the loading, filling or transitional change stage, and the smaller the activity intensity, the more the interface change tends to stop. Based on this, convergence conditions are set to determine the feedback end time point, which requires that the activity intensity is less than the adaptive threshold within the preset duration window and the absolute value of the rate of change of the activity intensity is less than the preset rate of change threshold. The rate of change of the activity intensity is used to characterize the trend of the activity intensity and avoid misjudging short-term low-change segments as completed states. The adaptive threshold is generated by a linear combination of the median of the activity intensity within a preset continuous window and the dispersion measure, which includes the median absolute deviation. This allows the threshold to be automatically adjusted according to factors such as different device resolutions, encoding noise, and interface complexity. In principle, this robustly resists occasional spikes and noise fluctuations, ensuring that the convergence determination can be triggered in a timely manner when the change stops significantly, and is not easily triggered by local jitter or short-term fluctuations.

[0036] Furthermore, within the effective area, the mean optical flow modulus of adjacent frames or the directional consistency of the inter-frame pixel difference is calculated. When the mean optical flow modulus is less than a preset motion threshold, the motion amount is determined to meet the suppression condition; or when the directional consistency is less than a preset consistency threshold, the motion amount is determined to meet the suppression condition.

[0037] In this embodiment, to avoid prematurely determining the feedback end time point when the activity intensity has decreased but residual motion such as slow scrolling, gradual transition, or local displacement still exists on the interface, a motion suppression condition is introduced as an auxiliary gating within the effective area: On the one hand, optical flow is calculated for adjacent frames within the effective area and the average value of the optical flow modulus is taken as the motion index to characterize the average displacement intensity of the area between adjacent frames. When the average value of the optical flow modulus is less than the preset motion threshold, it indicates that the overall displacement within the effective area tends to be stationary, and the motion is determined to meet the suppression condition. On the other hand, a directional consistency index is constructed based on the inter-frame pixel difference to characterize whether the change presents obvious uniform direction motion characteristics. When the directional consistency is less than the preset consistency threshold, it indicates that there is no significant same-direction motion in the effective area, and it can also be determined that the motion amount meets the suppression condition. By combining the motion suppression condition with the activity intensity convergence condition, it is possible to effectively suppress false triggers in situations such as page inertial scrolling, slow animation endings, and partial sliding, ensuring that the feedback end time is closer to the stable completion state visible to the user, thereby improving the accuracy of UI operation feedback duration measurement.

[0038] This invention also provides a UI operation feedback duration calculation system based on screen recording analysis, characterized in that it includes: The screen recording and touch capture module is used to capture screen recording videos and touch event streams during user operation. The touch event stream includes touch event type, touch coordinates, and touch timestamp. The time alignment module is used to establish the alignment relationship between the touch timestamp and the screen recording video timeline; The operation start determination module is used to determine the submission time of a user operation based on the touch event stream, and to map the submission time to the operation start time point through the alignment relationship; The block change calculation module is used to divide the continuous frames of the screen recording video into blocks and calculate the inter-frame change after the operation start time point. The block corresponding to the touch coordinate is used as the seed block. The causal related block set is obtained by filtering the positive hysteresis correlation between the inter-frame change of each block and the inter-frame change of the seed block. The causal related block set is iteratively expanded to obtain the causal region corresponding to the current user operation. The causal region construction module is used to perform periodic change elimination on the causal region to obtain a valid region for completing the judgment. The effective region generation module is used to obtain the sequence of activity intensity over time based on the inter-frame change of the blocks within the effective region. When the activity intensity and its rate of change meet the convergence condition within a preset duration window, and the causal region no longer expands and the motion within the effective region meets the suppression condition, the feedback end time point is determined. The feedback end confirmation and duration calculation module is used to calculate the UI operation feedback duration based on the feedback end time and the operation start time.

[0039] Working Principle: This invention uses the period from "user's first operation submission" to "user-visible business feedback completion" as the measurement boundary. It provides visible evidence through screen recording video and objective temporal data of operation events through touch event streams, thus achieving end-to-end experience duration measurement without relying on application-internal tracking points or control trees. Screen recording video reflects page transitions, the appearance and disappearance of loading controls, and the gradual filling of content. Touch event streams record touch event types, touch coordinates, and touch timestamps, providing anchor points for starting point location and causal region construction. These two types of data serve distinct functions: the former ensures "what is measured is what the user sees," while the latter ensures "the starting point is the user's actual submitted operation."

[0040] Since touch event timestamps and video frame timestamps typically originate from different timing sources, direct use can lead to offsets and drifts, causing the operation start point to map to an incorrect frame. This invention records the system monotonic clock time and the timestamp of the first video frame at the start of screen recording, establishing an initial time mapping relationship so that any touch timestamp can be converted into a moment on the video timeline. To offset potential encoding buffering, thread scheduling, and clock drift during recording, this invention samples the system monotonic clock time and the corresponding video frame timestamp at preset intervals, using a linear fitting method to correct the initial mapping, ensuring the alignment remains stable throughout the recording cycle. This guarantees that touch events can be accurately located within a range of adjacent frames.

[0041] This invention uses the submission time of a user operation as the source of the operation start time point, avoiding mistaking the moment of pressing, the swiping process, or invalid touch as the starting point. Click operations typically trigger click semantics at the moment of release, therefore the release event timestamp is taken as the submission time; swipe or pull-to-refresh operations require a displacement threshold to be valid, and once valid, the release time is taken as the submission time; long-press operations trigger semantics when the press duration reaches a threshold, therefore the moment the threshold is reached is taken as the submission time. Through time alignment, the submission time is mapped to the operation start time point S1 on the video timeline, ensuring that S1 has a consistent, verifiable, and cross-application-usable definition.

[0042] Screen-recorded videos are continuous image sequences. Directly comparing the entire screen is easily affected by interference from irrelevant areas and is not conducive to locating the response area. This invention converts the continuous frames after step S1 to grayscale and divides them into blocks of a preset size. It calculates and accumulates the pixel grayscale differences between corresponding blocks in adjacent frames to obtain the inter-frame change. This change can intuitively reflect the intensity of change in the local area over time. For example, button press ripples, page transitions, skeleton screen appearance, new list rows, and image filling will all cause an increase in the block change.

[0043] Simply measuring the changes in individual blocks is insufficient to distinguish between "changes triggered by the operation" and "changes in the background itself." This invention uses the block corresponding to the touch coordinates as the seed block, and generates a sequence of changes in each block over time. Within a preset lag range, the Pearson correlation coefficient between each block sequence and the seed sequence is calculated, and the maximum correlation value within the lag range is taken as the correlation of that block. If the maximum correlation value reaches a threshold and the corresponding lag is greater than zero, it indicates that the change in that block lags behind the change in the touch seed in time, conforming to the causal direction of "operation first, response later," and is thus identified as a causally correlated block. The positive lag constraint effectively eliminates prior changes such as periodic animations and status bar jumps that exist before screen recording begins, and also reduces the impact of random noise unrelated to touch on the results.

[0044] Actual business feedback often starts near the touch point and then expands to a larger area; for example, clicking a list item switches the entire page, and clicking the search button gradually fills the results list area. If only seed-related blocks are relied upon, the response area may be too small. This invention iteratively expands the already screened set of causal related blocks, taking the mean of the change sequences of each block within the set as the reference change sequence, and using other blocks outside the set as candidate blocks. The positive hysteresis correlation between candidate blocks and the reference sequence is calculated, and blocks that meet the criteria are included in the set, and the reference sequence is updated accordingly. This process is equivalent to expanding the "response pattern" from a point-like seed to a surface region, gradually absorbing blocks with similar temporal change patterns to the identified response region, and finally converging to obtain the causal region corresponding to this operation. When no new blocks are added for at least one consecutive iteration, it indicates that the response region that can be included under the current correlation criterion has stabilized, and expansion stops.

[0045] Within the causal region, there may still be long-term periodic changes such as carousel ads, animated cards, and partial video previews. Even if these areas are on the same screen as page transitions, they will still generate changes over a long period, making it difficult to determine the "stable completion state." This invention periodically identifies the change sequence of each block within the causal region. It uses the peak value of autocorrelation at non-zero lag or frequency domain analysis to obtain the dominant frequency peak. When the peak value exceeds the periodic threshold and the duration reaches a preset duration, the corresponding block is identified as a periodically changing block and removed, resulting in the valid region. This mechanism is equivalent to excluding "always-moving" regions from the completion determination, allowing subsequent convergence determination to focus on the loading and filling process triggered by a single operation, avoiding the end time being delayed by periodic animations.

[0046] Within the effective region, the inter-frame changes of each block at the same time are summed to obtain a sequence of activity intensity over time. Higher activity intensity indicates more drastic interface changes, typically corresponding to loading, transition, or filling in progress. Lower activity intensity indicates a more stable interface. This invention uses "activity intensity less than an adaptive threshold and the absolute value of the activity intensity change rate less than a change rate threshold" as the convergence condition. The adaptive threshold is generated by a linear combination of the median of activity intensity within a preset duration window and a dispersion measure, including the median absolute deviation. This robust statistical method can adapt to differences in device resolution, encoding noise, interface complexity, and network conditions, reducing spike interference and making convergence determination more stable and less prone to false triggering.

[0047] Relying solely on motion intensity might lead to misjudgments of completion during slow scrolling, the end of a gradual animation, or slight displacement. This invention introduces motion suppression conditions as a gating mechanism within the effective area. It calculates the average optical flow modulus of adjacent frames; when this value is below a motion threshold, it indicates that the displacement has reached a standstill. Alternatively, it calculates the directional consistency of pixel differences between frames; when this value is below a consistency threshold, it indicates that there is no significant unidirectional motion. Combining motion suppression with convergence conditions effectively avoids premature termination due to inertial scrolling or slow transitions, ensuring that the feedback end time S2 is closer to the stable completion state visible to the user.

[0048] Taking "in-app search" as an example, after a user enters keywords and clicks the search button, the moment the touch is released is taken as the submission moment and mapped to t1. After t1, the page transitions, the skeleton screen appears, the results list is filled line by line, and images are loaded with a delay. The block change will significantly increase in the button area and the results list area. Through positive lag correlation filtering and iterative expansion, the causal region will expand from the vicinity of the button to the main body of the list. If there are carousel ads on the page, their block change is periodic and is excluded after periodic elimination. As the text and images are loaded, the activity intensity decreases and converges within the continuous window, and the motion index also decreases. The system determines t2 based on this. The final duration is t2 minus t1, achieving stable measurement for complex asynchronous loading scenarios. Taking "pull-to-refresh" as an example, the moment the user lifts the touch button after the displacement meets the threshold is taken as t1. During the refresh process, the changes in the list content are concentrated in the list area. A focused effective area is constructed through the causal region. When the refresh ends and the list stabilizes and the motion decreases, t2 is determined, thus obtaining the refresh feedback duration.

[0049] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0050] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for calculating UI operation feedback duration based on screen recording analysis, characterized in that, include: Step S1: Collect screen recording video and touch event stream of the user's operation process. The touch event stream includes touch event type, touch coordinates and touch timestamp. Step S2: Establish the alignment relationship between the touch timestamp and the screen recording video timeline; Step S3: Determine the submission time of a user operation based on the touch event stream, and map the submission time to the operation start time point through the alignment relationship; Step S4: After the start time of the operation, the continuous frames of the screen recording video are divided into blocks and the inter-frame change of the blocks is calculated. The block corresponding to the touch coordinate is used as the seed block. The causal related block set is obtained by filtering according to the positive hysteresis correlation between the inter-frame change of each block and the inter-frame change of the seed block. The causal related block set is iteratively expanded to obtain the causal region corresponding to the current user operation. Step S5: Perform periodic change elimination on the causal region to obtain the effective region for completing the determination; Step S6: Based on the inter-frame change of the blocks within the effective region, obtain the sequence of activity intensity over time. When the activity intensity and its rate of change meet the convergence condition within a preset duration window, and the causal region no longer expands and the motion within the effective region meets the suppression condition, determine the feedback end time point. Step S7: Calculate the UI operation feedback duration based on the feedback end time and the operation start time.

2. The UI operation feedback duration calculation method based on screen recording analysis according to claim 1, characterized in that, Step S2 specifically involves: At the start of screen recording, the system monotonic clock time and the timestamp of the first video frame are recorded. An initial time mapping relationship is established based on the system monotonic clock time and the timestamp of the first video frame. During the recording process, the system monotonic clock time and the corresponding video frame timestamp are obtained at preset intervals. The initial time mapping relationship is linearly fitted and corrected to compensate for time offset and time drift.

3. The UI operation feedback duration calculation method based on screen recording analysis according to claim 1, characterized in that, Determine the submission time of a user operation, specifically as follows: When the touch event type is a click operation, the touch timestamp corresponding to the touch release event is determined as the submission time; When the touch event type is a swipe or pull-to-refresh operation, the touch timestamp corresponding to the touch lift event is determined as the submission time, provided that the touch movement trajectory meets the displacement threshold. When the touch event type is a long press operation, if the duration of the touch press reaches a preset duration threshold, the touch timestamp that reaches the duration threshold is determined as the submission time.

4. The method for calculating UI operation feedback duration based on screen recording analysis according to claim 1, characterized in that, The calculation of inter-frame changes in blocks is as follows: The continuous frames of the screen-recorded video are converted to grayscale, and each frame is divided into image blocks of a preset size; The pixel grayscale difference between corresponding blocks in two adjacent frames is accumulated and used as the inter-frame change of the corresponding block.

5. The method for calculating UI operation feedback duration based on screen recording analysis according to claim 1, characterized in that, The causal correlation block set was obtained by filtering based on positive lag correlation, specifically as follows: The inter-frame changes of each block are formed into a change sequence over time. The Pearson correlation coefficient between the change sequence and the change sequence of the seed block is calculated within a preset lag range. The maximum correlation value within the lag range is taken as the correlation of the block. When the maximum correlation value meets a preset threshold and the corresponding lag is greater than zero, the block is determined as a causal related block.

6. The method for calculating UI operation feedback duration based on screen recording analysis according to claim 1, characterized in that, The causal correlation block set is iteratively expanded as follows: The mean of the inter-frame change sequence of each block in the causal correlation block set is used as the reference change sequence, wherein the candidate blocks are other blocks besides the causal correlation block set. The causal correlation block set is updated based on the positive hysteresis correlation between the reference change sequence and the inter-frame change sequence of the candidate blocks, until the causal correlation block set no longer adds blocks in at least one consecutive iteration.

7. The method for calculating UI operation feedback duration based on screen recording analysis according to claim 1, characterized in that, The periodic change removal is performed on the causal region as follows: The inter-frame change sequence of each block within the causal region is periodically determined. The periodic determination includes calculating the peak value at non-zero lag using autocorrelation or obtaining the main frequency peak value using frequency domain analysis. When the peak value is greater than a preset period threshold and the duration reaches a preset duration, the corresponding block is determined as a periodically changing block, and the periodically changing block is removed from the causal region to obtain the effective region.

8. The method for calculating UI operation feedback duration based on screen recording analysis according to claim 1, characterized in that, The sequence of activity intensity over time is the cumulative value of the inter-frame change of each block within the effective region at the same time. The convergence condition includes the activity intensity being less than an adaptive threshold within a preset duration window and the absolute value of the rate of change of the activity intensity being less than a preset rate of change threshold. The adaptive threshold is a linear combination of the median of the activity intensity within the preset duration window and a dispersion measure, and the dispersion measure includes the median absolute deviation.

9. The method for calculating UI operation feedback duration based on screen recording analysis according to claim 1, characterized in that, Within the effective area, the mean optical flow modulus of adjacent frames or the directional consistency of the inter-frame pixel difference is calculated. When the mean optical flow modulus is less than a preset motion threshold, the motion amount is determined to meet the suppression condition. Or when the directional consistency is less than a preset consistency threshold, the motion amount is determined to meet the suppression condition.

10. A UI operation feedback duration measurement system based on screen recording analysis, characterized in that, include: The screen recording and touch capture module is used to capture screen recording videos and touch event streams during user operation. The touch event stream includes touch event type, touch coordinates, and touch timestamp. The time alignment module is used to establish the alignment relationship between the touch timestamp and the screen recording video timeline; The operation start determination module is used to determine the submission time of a user operation based on the touch event stream, and to map the submission time to the operation start time point through the alignment relationship; The block change calculation module is used to divide the continuous frames of the screen recording video into blocks and calculate the inter-frame change after the operation start time point. The block corresponding to the touch coordinate is used as the seed block. The causal related block set is obtained by filtering the positive hysteresis correlation between the inter-frame change of each block and the inter-frame change of the seed block. The causal related block set is iteratively expanded to obtain the causal region corresponding to the current user operation. The causal region construction module is used to perform periodic change elimination on the causal region to obtain a valid region for completing the judgment. The effective region generation module is used to obtain the sequence of activity intensity over time based on the inter-frame change of the blocks within the effective region. When the activity intensity and its rate of change meet the convergence condition within a preset duration window, and the causal region no longer expands and the motion within the effective region meets the suppression condition, the feedback end time point is determined. The feedback end confirmation and duration calculation module is used to calculate the UI operation feedback duration based on the feedback end time and the operation start time.

Images