Application demonstration video understanding method based on computer vision division and multi-modal reasoning

Abstract

The invention provides an application demonstration video understanding method based on computer vision division and multi-modal reasoning. The method is used for automatically understanding an application operation process in a video shot by a camera. The method comprises the following steps: inputting a demonstration video containing screen contents and hand actions, firstly extracting a screen region through a text detection and segmentation model, and obtaining a screen image sequence; detecting interface change based on structural similarity, constructing a candidate action section, and obtaining an interactive effective section through jitter filtering and dynamic component filtering; on this basis, a plurality of key interface frames are extracted to construct a multi-frame image, and the multi-mode large model is input for comprehensive reasoning of interface difference and control semantics, so that corresponding operation actions are judged. Accuracy and stability of action recognition in the application demonstration video can be improved, manual analysis cost is reduced, and video interaction analysis efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the fields of video understanding, computer vision, and automatic human-computer interaction parsing technology, specifically relating to a method for understanding application demonstration videos based on computer vision segmentation and multimodal large model discrimination. This method is applicable to scenarios such as automatic parsing, automated testing, and video interaction understanding of mobile application demonstration videos. Background Technology

[0002] Application demo videos, as a direct record of mobile application operation, have become an important medium for showcasing functional modules, demonstrating interactive flows, and recording usage. Real-device demo videos are typically captured by a camera and include screen content, user hand gestures, and the surrounding environment. Automated recognition of actions within these videos can provide crucial data for tasks such as generating automated test scripts, learning GUI agents, and analyzing user interface interactions.

[0003] Existing research utilizes techniques such as pixel difference analysis, silhouette detection, and UI structure analysis to identify user interactions in screen-recorded videos. However, in application demonstration videos captured by a camera, traditional visual methods are prone to misjudging the timing and type of actions due to interference factors such as hand occlusion, changes in ambient lighting, lens shake, screen reflection, and compression noise. On the other hand, Multimodal Large Language Models (MLLMs) excel in scene understanding and semantic reasoning, but using complete videos as input is subject to interference from redundant information. They are also insensitive to slight interface changes and struggle to accurately identify the specific timing of operations.

[0004] Therefore, existing technologies are still insufficient to achieve a video understanding method that combines robust motion localization and accurate motion discrimination under real shooting conditions. Summary of the Invention

[0005] The purpose of this invention is to address the problem that application demonstration videos captured by cameras in the prior art are easily affected by factors such as hand obstruction, changes in ambient lighting, and camera shake, resulting in unstable interface change detection and insufficient accuracy in judging operation actions. This invention proposes an application demonstration video understanding method based on computer vision segmentation and multimodal reasoning to achieve reliable recognition and accurate positioning of user operation actions.

[0006] To achieve the above objectives, this invention provides an application demonstration video understanding method. The method processes the input application demonstration video frame by frame. First, it extracts the purest possible screen content by extracting screen regions. Then, it determines candidate intervals corresponding to interface changes by using keyframe segmentation based on structural similarity. Subsequently, it uses a keyframe filtering mechanism to remove invalid intervals caused by jitter and dynamic components. Based on this, it constructs and splices together an action frame group containing interface states at multiple times. Finally, it inputs this action frame group into a multimodal inference model to complete the operation action discrimination, thereby obtaining a set of operation events containing operation type and action occurrence frame number.

[0007] Specifically, the technical solution of the present invention includes:

[0008] First, the application demonstration video captured by the camera is preprocessed to reduce interference introduced by factors such as background environment, lighting changes, and hand obstruction, and to ensure that subsequent analysis steps can be carried out on a clear and stable screen area.

[0009] Step S1 involves screen region extraction. By performing text detection on each frame of the image, the approximate area of ​​the screen is inferred based on the distribution of text blocks. Since video frames captured by the camera often contain device borders, hands, desktop elements, or other background elements, direct interface analysis can lead to external interference affecting subsequent processing algorithms. Therefore, after inferring the screen region, this step further uses a segmentation model within that region to generate a screen mask. The original image is then cropped using this mask to obtain an image sequence containing only the screen content. This method ensures that subsequent structural similarity analysis and keyframe extraction focus on the screen content itself, thereby improving the accuracy and robustness of the overall processing.

[0010] Furthermore, in step S1, text detection preferably employs an optical character recognition method. The screen boundary is inferred from the position, number, and distribution area of ​​the identified text boxes. Then, several reference points are selected within this boundary region and input into the generative segmentation model, from which a more stable screen mask is chosen. This method can maintain reliable extraction of the screen region even when encountering hand occlusion, uneven lighting, or complex backgrounds, improving the robustness of screen localization and providing stable input for subsequent keyframe segmentation.

[0011] Step S2 involves keyframe segmentation based on structural similarity. This step converts the screen image sequence into a luminance channel and calculates the structural similarity index of any two adjacent frames, generating a similarity curve that changes over time. Structural similarity reflects the differences between two frames in terms of brightness, contrast, and structure. Therefore, when the interface content changes due to user actions such as jumping, switching, or scrolling, the structural differences between adjacent frames increase significantly. Positions where the similarity abruptly decreases from high to low and then stabilizes are identified as the boundaries of intervals where the interface has changed, thus obtaining candidate frame intervals where significant changes have occurred.

[0012] Furthermore, in step S2, the structural similarity preferably adopts a comprehensive index based on brightness distribution, contrast, and structural information, and is calculated only for the area covered by the screen mask. In camera shooting scenarios, non-screen factors such as hands, reflections, tabletop or background movement may cause pixel changes. If the structural similarity is calculated directly across the entire image area, misjudgments are likely to occur. Therefore, calculating similarity only for the screen mask area can significantly reduce interference and ensure more accurate change detection.

[0013] Step S3 is the keyframe filtering mechanism. This step further filters the candidate intervals obtained in step S2 to eliminate changes caused by non-interactive factors. First, the structural similarity of the first and last frames of the candidate interval is judged. When the similarity is higher than a preset threshold, it indicates that the overall change amplitude of the interval is small, which is very likely caused by slight shooting shake or lens micro-movement, and the interval should be eliminated. Subsequently, several frames are sampled from the remaining candidate intervals and sent to the image multimodal model. The model determines whether the interval is caused by internal interface animation or automatic refresh based on interface semantics and other information. If it is determined to be non-user interaction, the interval is further eliminated. After the above filtering, the effective change intervals corresponding to the actual user operation can be obtained.

[0014] Furthermore, in step S3, the multimodal model determines whether an interval is triggered by user interaction by comparing the semantic differences between sampled frames, changes in control states, and the logic before and after the interface structure. When the interface only displays a looping animation or video playback without explicit changes in control states, interface transitions, or semantic changes related to interaction, the model classifies this change as a dynamic component change and removes it to further improve the reliability of subsequent operation judgments.

[0015] Step S4 involves keyframe construction and stitching. Within each valid interval, keyframes representing different states of the interface before and after a change are selected, including stable frames before the change, frames at the start of the change, intermediate frames during the change, and stable frames after the change. These frames at multiple key moments are stitched together in chronological order into a multi-frame structure image, facilitating contextual reasoning in the multimodal model. This step integrates interface changes that were originally scattered across the time dimension into a single visual structure, providing complete and contrasting contextual input for subsequent reasoning steps.

[0016] Furthermore, in step S4, the selection of keyframes can be determined based on the interval length using a fixed frame interval method. This ensures the integrity of the input information while controlling computational overhead, enabling the model to maintain performance while achieving high operating efficiency.

[0017] Step S5 involves operation action discrimination based on multimodal reasoning. This step inputs the stitched keyframe image group into an image multimodal large language model. Pre-designed prompts guide the model to focus on differences in interface structure, semantic changes in controls, and interface logic between keyframes. By analyzing interface changes and combining interface layout, control meanings, and the magnitude of visual changes, the model performs semantic reasoning on the interface changes corresponding to each valid interval, outputting the operation action type corresponding to that change, such as a click, swipe, or other common interaction forms, and further generating the final operation event recognition result.

[0018] Furthermore, in step S5, the prompt words include constraints on the comparison of the interface before and after, the semantic focus of the controls, and the rationality of the operation results. This guides the multimodal model to perform implicit chain reasoning in the order of "analyzing changes - comparing semantics - inferring actions", which improves the stability of action category discrimination, avoids misclassification caused by misreading local details, and thus enhances the overall recognition effect.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] First, by combining screen region extraction with keyframe segmentation based on structural similarity, this invention can accurately capture the interface change range in complex shooting environments, significantly reducing the impact of hand occlusion, ambient light changes, and shooting shake on interface change detection.

[0021] Second, the present invention eliminates non-interactive changes caused by jitter and dynamic components at the candidate interval level through a keyframe filtering mechanism, reducing invalid intervals from entering the subsequent reasoning process, thereby improving the overall reliability of recognition.

[0022] Third, this invention utilizes the keyframe group construction and splicing method to uniformly splice multiple states of the interface before, during and after the change into a multi-frame structure image, and uses a multimodal large language model combined with prompts to perform semantic reasoning, enabling the model to comprehensively consider spatial layout and temporal sequence, thereby more accurately distinguishing different operation actions such as clicking and swiping.

[0023] Fourth, the method of the present invention does not require parameter-level training of the multimodal model. It can achieve high-quality operation recognition of camera-captured application demonstration videos by relying solely on prompt design and input organization. It has the advantages of low implementation cost, wide applicability, and easy deployment in actual systems. Attached Figure Description

[0024] Figure 1 This is a general flowchart of an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the screen area extraction process of the present invention;

[0026] Figure 3 This is a schematic diagram of keyframe segmentation based on structural similarity according to the present invention;

[0027] Figure 4 This is a schematic diagram of the keyframe filtering mechanism of the present invention;

[0028] Figure 5 This is a schematic diagram of the keyframe group construction and splicing structure of the present invention;

[0029] Figure 6 This is a flowchart illustrating the operation action discrimination based on multimodal reasoning in this invention. Detailed Implementation

[0030] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0031] like Figure 1 As shown, this invention integrates computer vision segmentation and multimodal reasoning in its application demonstration video understanding method to process video sequences captured by a camera step by step, thereby achieving automatic recognition of user actions. The method sequentially includes steps such as screen region extraction, keyframe segmentation, keyframe filtering, keyframe group construction and stitching, and action discrimination based on multimodal reasoning. The steps are clearly connected, forming a stable video interaction understanding process that can operate in real-world shooting environments.

[0032] Step 1: Screen area extraction. For example... Figure 2As shown, this step extracts the screen region frame by frame from the input video to ensure that subsequent change detection and semantic reasoning are based on the screen content. Specifically, optical character recognition is first performed on each frame of the image. The maximum boundary of the text blocks is delineated based on the distribution position of the text blocks in the video frame to determine the approximate rectangular area where the screen is located. Further, candidate points are selected within this area based on the distance between the text blocks and the center point of the area. Then, a generative segmentation model is used for segmentation. The mask with the largest number of text blocks and the largest area in the result is selected as the screen region mask. Finally, the original image is cropped using the mask to form an image sequence composed of screen content. This step can effectively remove interference factors such as hands, background, and reflections, providing clear screen visual data for subsequent steps.

[0033] Step 2: Keyframe segmentation based on structural similarity. For example... Figure 3 As shown, this step locates the start and end intervals of interface changes based on the structural similarity changes between adjacent frames in the screen image sequence. Specifically, it includes: converting screen frames to the YUV color space and selecting the Y channel as the luminance feature; then calculating the structural similarity (SSIM) of adjacent frames frame by frame and forming a similarity curve; when the similarity decreases significantly from high to low, it is determined that the interface state has begun to change, and the starting frame of the change is determined by forward scanning; then, the ending frame of the interface change is determined by reverse scanning. This generates candidate intervals where the interface is in a continuously changing state. This method can accurately capture interface change points caused by user operations.

[0034] Step 3: Keyframe filtering mechanism. For example... Figure 4 As shown, this step filters the candidate intervals obtained in step 2 to eliminate changes caused by non-user interaction. The filtering mechanism includes two types:

[0035] (1) Jitter filtering. By comparing the structural similarity of the first and last frames of the candidate interval, when the value is higher than the preset threshold, it is determined that the change is caused by slight jitter and is not regarded as a real interface change;

[0036] (2) Dynamic component filtering. Further, in order to eliminate changes caused by internal interface animations or automatic refresh mechanisms, this step samples several keyframes of the candidate interval and inputs them into the image multimodal model. The model analyzes the semantic differences of the interface to determine whether the change originates from user interaction. If the model infers that the change reflects automatic interface updates or dynamic element animations, the corresponding interval is eliminated.

[0037] After the above filtering process, an effective range of variation that is highly relevant to the user's actual operation and has low noise can be obtained.

[0038] Step 4: Keyframe group construction and stitching. For example... Figure 5As shown, this step constructs a keyframe group for subsequent inference based on the effective interval obtained in step 3. Specifically, this includes extracting stable frames before the interface change begins, the change's initiation frame, any intermediate frames during the change, and stable frames after the change ends. These frames are then stitched together in chronological order into a multi-frame structure image, with the corresponding original frame number labeled at the corner of each frame, ensuring the stitched structure contains both spatial relationships and temporal information. This keyframe group serves as the basic input for subsequent multimodal model action type recognition.

[0039] Step 5: Action discrimination based on multimodal reasoning.

[0040] like Figure 6 As shown, this step inputs the keyframe group constructed in step 4 into the multimodal large language model of the image, and guides the model to focus on the differences in interface structure, semantic changes of controls, and layout logic between spliced ​​frames through prompt words.

[0041] The prompt template consists of three parts: image description, guiding prompt, and output format constraints.

[0042] (1) Image description used to indicate that the input image is a sequential screenshot of the application's operational behavior, including:

[0043] - The portion containing the stable frame represents the interface state before the user operation.

[0044] - The start frame represents the instantaneous state when the user completes the operation but the interface has not yet started to change;

[0045] - The intermediate frames and end frames during the change process represent the state of the interface after the user operation and the subsequent changes.

[0046] (2) Guidance prompts are used to guide the model to analyze the page content and its changes in the multiple frames of images under the constraint of only one user operation, in order to determine the type and parameters of the operation performed by the user. This part includes:

[0047] - When determining the type of operation, it is necessary to consider the changes in the interface before and after the operation, and to distinguish between page switching animations caused by clicking and overall displacement of interface content caused by swiping.

[0048] - When analyzing operation actions, the semantics of the operable elements on the interface and the semantics of the new interface content that appears after the operation are combined to make the inferred hand actions and interface results consistent with the actual performance in regular applications.

[0049] - When the action is inferred to be a click, the specific control that was clicked is determined based on the user's finger position and the surrounding controls. The consistency between the action type, control semantics and interface changes is verified. If the control is an icon, its appearance style also needs to be described.

[0050] - When the operation is inferred to be a swipe, the rationality of the swipe assumption is verified based on the changes in the interface content and the results, and the swipe direction is determined to be one of up, down, left or right;

[0051] (3) Output format constraints are used to constrain the model to return the operation recognition results in the form of a standard JSON array. The array elements include:

[0052] - A field used to identify the sample number of the current operation;

[0053] - A field used to indicate the type of operation; values ​​must include at least click and swipe.

[0054] - Fields used to characterize specific operational goals or operational parameters;

[0055] - A field used to provide a judgment basis explanation of no more than twenty characters.

[0056] Finally, the operation action category will be extracted from the model output and the frame number of the action will be matched with it to achieve automatic recognition of user operations in the application demonstration video.

[0057] In summary, by executing the above steps sequentially, the present invention can accurately detect interface changes and determine operation actions in complex environments such as changes in lighting, hand obstruction, and camera shake, thereby achieving a stable understanding of application demonstration videos.

[0058] To fully demonstrate the effectiveness of the application demonstration video understanding method that integrates computer vision segmentation and multimodal reasoning proposed in this invention, the performance of the method on actual video data was tested and evaluated in this embodiment. The following indicators were used to measure the performance of the method:

[0059] (1) Operation start frame detection accuracy, used to measure the positioning accuracy of the method of the present invention at the moment when the user operation occurs;

[0060] (2) Operation action category recognition accuracy is used to measure the correctness of the judgment of action types such as click and swipe, provided that the initial frame detection is correct.

[0061] In specific experiments, we selected demonstration videos captured by cameras from various common mobile applications as test data. The videos covered different interface forms, lighting conditions, and operation processes with hand occlusion. We compared and evaluated the interface change detection method based solely on traditional visual difference, the end-to-end multimodal inference method, and the method of this invention. Experimental results show that, under the same dataset and evaluation criteria, the method of this invention outperforms the baseline method using only visual difference in both the accuracy and false negative rate of operation start frame detection, and also outperforms the end-to-end multimodal inference method in terms of action category recognition accuracy. Furthermore, it maintains stable performance even in complex scenarios such as hand occlusion, background changes, and slight shaking, thus verifying the applicability and robustness of the method of this invention.

[0062] The above description represents the preferred embodiments of the present invention. It should be noted that, for those skilled in the art, various improvements and modifications can be made to the above embodiments without departing from the principles and essence of the present invention, and all such improvements and modifications should be considered to fall within the protection scope of the present invention.

Claims

1. A computer vision-based application demo video understanding method based on partitioning and multi-modal reasoning, characterized in that, The method comprises the following steps: S1 screen region extraction; first, text detection is used to identify the text region in the video frame, then the screen boundary is preliminarily determined according to the text distribution, and finally a screen region mask is generated by using a segmentation model and the original video frame is cropped to obtain a screen image sequence; S2 key frame division based on structural similarity; the screen image sequence is converted into a luminance channel, the structural similarity of adjacent frames is calculated, and a similarity curve is formed, and the interface change interval is determined according to the similarity mutation; S3 key frame filtering; the candidate interval is filtered by jitter filtering and dynamic component filtering, and the effective change interval triggered by user operation is retained; S4 key frame group construction and splicing; the key frames representing different states of the interface are extracted from the effective change interval and spliced into multiple images in chronological order; S5 operation action discrimination based on multi-modal reasoning; the multi-modal model is inputted with the multiple images, the prompt words are constructed, the operation action is discriminated according to the interface structure difference and the control semantics, and finally the final recognition result is obtained in combination with the corresponding starting frame number.

2. The method of claim 1, wherein, Step S1 comprises: (1) The text region in the video frame is detected by using an optical character recognition method, the spatial position information of all the detected text blocks is counted, the extreme coordinates in the horizontal direction and the vertical direction are obtained, the outer boundary box is constructed according to the extreme coordinates, and the candidate boundary range of the screen in the video frame is determined; (2) In the screen candidate boundary range, reference points are selected from near to far according to the distance relationship between the text block vertex and the boundary range center point, the reference points are inputted into a target recognition model, a plurality of candidate screen image regions are generated, and the region with the largest area and more than n text blocks is selected as the final screen region. (3) The final screen region is used to crop the original video frame sequence to obtain a frame-level image sequence containing only screen content, which contains only screen interface content and can effectively reduce the influence of factors such as hand occlusion, uneven illumination and complex background.

3. The method of claim 1, wherein, Step S2 comprises: converting the screen image sequence obtained in step S1 into a luminance channel, calculating the structural similarity of each frame and the next frame to obtain the similarity value of each adjacent two frames, constructing a similarity curve, when the similarity curve appears mutation, that is, the falling amplitude exceeds the preset judgment threshold, and after the similarity rises, multiple frames remain relatively stable, the falling starting point of this mutation is taken as the change starting frame, and the stable point of the rising is taken as the change ending frame, so as to determine the video frame sequence candidate interval caused by the interface change triggered by the action.

4. The method of claim 1, wherein, Step S3 comprises two parts of screening on the video frame sequence candidate interval obtained in step S2: (1) Sudden jitter: compare the structural similarity of the first and last frames of the candidate interval, when the structural similarity of the first and last frames of the candidate interval is higher than the preset threshold, it is determined that the change is caused by slight camera jitter, and the interval is removed; (2) Non-interactive animation: input a plurality of screen frames sampled in the interval into a multi-modal model, according to the prompt words, let the model judge whether the change of the interval is caused by the interface internal animation or automatic refreshing mechanism according to the interface semantics and layout change, if it is determined that it is not user interaction, the interval is removed. After the above two screenings, the effective interface change interval is obtained.

5. The method of claim 1, wherein, Step S4 includes: sequentially selecting the stable frame before the change, the change starting frame, the middle frame in the change process and the change ending frame from the effective interface change interval obtained in S3, and splicing them into multiple frames of images in time sequence to express the order of interface change, thereby providing complete context information for subsequent model reasoning.

6. The method of claim 1, wherein, Step S5 includes: inputting the multiple frames of images generated in step S4 into an image multi-modal large language model, and inputting a text prompt at the same time. The text prompt is a preset prompt word template, which consists of three parts: image description, guide prompt and output format constraint. (1) The image description is used to indicate that the input image is a sequential screenshot of application operation behavior, which is used to explain: - The stable frame represents the interface state before user operation; - The starting frame represents the instantaneous state when the user completes the operation and the interface has not yet started to change; - The middle frame in the change process and the change ending frame represent the state after the interface starts to change after the user operation; (2) The guide prompt is used to guide the model to analyze the page content and its changes before and after in the multiple frames of images under the constraint that there is only one user operation, to judge the operation type and parameters of the user, which at least includes: - When judging the operation type, the interface changes before and after the operation need to be considered to distinguish between page switching animation caused by clicking and interface content displacement caused by sliding; - When analyzing the operation action, the semantics of interface operable elements and the semantics of new interface content after the operation are combined to make the inferred hand action consistent with the actual performance in the conventional application program; - When inferring as a click operation, based on the user's finger position and the surrounding controls, the specific control that is clicked is determined, and the consistency of action type, control semantics and interface change is verified. If the control is an icon, its appearance style also needs to be described; - When inferring as a sliding operation, the rationality of the sliding hypothesis is verified based on the change form and result of the interface content, and the sliding direction is determined to be one of up, down, left or right; (3) The output format constraint is used to constrain the model to return the operation recognition result in the form of a standard JSON array, which includes: - A field for identifying the current operation sample number; - A field for representing the operation type, which takes at least click and sliding; - A field for representing the specific operation target or operation parameter; - A field for giving a twenty-word or less explanation of the basis for the judgment. Finally, the model makes a judgment on the operation action type of the interface change interval according to the prompt word, and then corresponds the action type with the change starting frame of the interval to form the final understanding result [change starting frame number: action type].

Images