Black box application state advancement and timing scenario capture methods, devices, and products

Abstract

The application discloses a black box application state advancing and timing scene capturing method, equipment and product, and belongs to the technical field of computer automation. The method comprises the following steps: determining a rendering area of an application and establishing a space reference; determining an interactive opportunity by analyzing visual changes of a picture sequence; generating physical input actions irrelevant to internal logic of the application; executing the actions and collecting feedback pictures; visually comparing the feedback pictures with historical states to generate or reject new observable state nodes; triggering a backup exploration strategy to generate supplementary actions when exploration stagnates; and finally outputting a timing scene evidence chain containing a state screenshot sequence and a trigger action record based on the formed state nodes. The application does not depend on application source code or interfaces, can effectively advance the state of a black box application through a visual feedback closed loop, capture credible scene evidence, and improves the robustness and escape ability of automatic processing of unknown applications.

Description

Technical Field

[0001] This application relates to the field of computer automation, and in particular to a method, device and product for black-box application state advancement and timing scene capture. Background Technology

[0002] In numerous scenarios such as software automated testing, content analysis, multimodal data annotation, content security auditing, and application retrieval database construction, systems need to capture interface screenshots that demonstrate the core functions or gameplay of a target application. However, if only the application's splash screen, loading page, or main menu is captured, subsequent analysis systems will find it difficult to determine the application's interaction methods, functional structure, and core value.

[0003] Existing web automation solutions, such as Selenium or Puppeteer, typically rely on the Document Object Model (DOM) to locate UI elements and execute interactions by reading button text, control IDs, or CSS selectors. However, many modern applications, especially lightweight interactive applications, interactive ads, and embedded applications, use HTML5's Canvas, WebGL, or WebGPU technologies for rendering. In these applications, the buttons, menus, and interactive objects seen by the user are merely collections of pixels within the rendering area; their internal structure and semantic information are invisible to external automation tools, rendering traditional DOM-based solutions completely ineffective. Furthermore, for remote streaming applications or applications running in simulators, the automation system can only access the final video stream, similarly facing the "black box" dilemma.

[0004] To address these black-box scenarios, the industry has proposed several automated testing solutions based on computer vision (CV). These solutions typically attempt to understand the semantics of interface elements through image recognition models (e.g., recognizing a "start" button) or utilize reinforcement learning to train agents to complete specific tasks. However, these solutions have inherent limitations. First, they often require a large number of samples for model training or predefined, explicit test cases and success criteria, which is costly and poorly adaptable for scenarios requiring batch processing of a large number of unknown applications. Second, these solutions aim to simulate human users or complete specific test tasks; their internal states are usually represented as complex scene vectors or model parameters, rather than a clear chain of evidence that can be directly verified externally.

[0005] Therefore, existing technologies generally suffer from the following shortcomings: Under black-box conditions where the application's internal state cannot be read, engine interfaces cannot be called, or control semantics cannot be relied upon, it is difficult to accurately determine whether the application has transitioned from a non-interactive phase such as loading or advertising to a stable interactive phase; it is difficult to generate a set of cross-application, engine-independent physical input actions to effectively explore the application state; after executing actions, it is difficult to determine whether the screen change forms a new meaningful state or is merely an insignificant animation or repeated frames, resulting in the storage of a large number of redundant or invalid screenshots; when encountering weak responsiveness situations such as menu blocking, authorization masking, apparent freezing, or loading lag, the system is prone to prematurely determining failure and interrupting the exploration; ultimately, it is also difficult to output the entire state progression process as a structured piece of evidence containing triggering actions and time sequences, which can be verified manually or by machines. Summary of the Invention

[0006] The purpose of this application is to provide a method, device, and product for black-box application state advancement and temporal scene capture, which aims to reliably advance the application state and capture a verifiable temporal scene evidence chain through an external visual feedback loop without relying on the application's internal state or semantic understanding.

[0007] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for black-box application state progression and timing scene capture, including: Determine the rendering area that carries the visual content of the target black-box application, and establish the spatial reference of the rendering area; The interactive timing of the target black-box application is determined based on the changes in the rendering area, and the rendering area corresponding to the interactive timing is added to the set of historical valid state nodes as the initial observable state node. A queue of candidate physical actions is generated based on the spatial reference and decoupled from the engine of the target black-box application. When the interactive moment is reached, at least some of the actions in the candidate physical action queue are executed sequentially within the rendering area, and the image of the rendering area after each action is executed is captured as the post-action feedback image. The visual content difference of the post-action feedback screen is compared with the screens of one or more historical valid state nodes in the historical valid state node set. When a post-action feedback screen does not have a preset similarity relationship with the screens of the compared historical valid state nodes, the post-action feedback screen is added as a new observable state node to the historical valid state node set. If a new observable state node cannot be generated by executing the actions in the candidate physical action queue within the preset exploration period, an alternative exploration strategy is triggered to generate a supplementary physical input action, and the supplementary physical input action is added to the candidate physical action queue for subsequent execution. Once the execution of the action meets the stopping condition, a structured temporal scene evidence chain is organized and output based on all observable state nodes and their associated actions in the set of historical valid state nodes.

[0008] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the black-box application state advancement and timing scene capture method described above.

[0009] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the black-box application state advancement and timing scene capture method described above.

[0010] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the black-box application state advancement and timing scene capture method described above.

[0011] According to the specific embodiments provided in this application, the following technical effects are disclosed: 1. Reduced dependence on application source code, engine interfaces, and control semantics. This invention achieves closed-loop control entirely based on externally observable visual information and physical input, without intruding into the application's internals. Therefore, it is widely applicable to various black-box applications employing technologies such as Canvas, WebGL, and remote streaming, solving the applicability problems of traditional automation solutions.

[0012] 2. Improved the success rate and efficiency of entering core functional states in unknown black-box applications. By dynamically determining interactive opportunities, generating standardized physical action queues, and judging new states based on visual differences, this invention constructs a universal state exploration framework, avoiding the huge cost of writing customized scripts for each application and realizing automated batch processing of heterogeneous applications.

[0013] 3. Enhanced ability to escape from weak response or stuck scenarios. This invention determines whether exploration has stalled by monitoring the generation of new state nodes, and triggers alternative exploration strategies, such as grid-based clicking, when stalled, thereby effectively dealing with difficult scenarios such as false readiness, menu blocking, and authorization masking, and significantly improving the robustness of state advancement.

[0014] 4. Improved evidentiary quality and usability of the output results. This invention outputs not scattered screenshots, but a structured, time-series chain of scene evidence, including a sequence of state screenshots, the precise actions triggering each state, and a grid-based evidence diagram. This form of output clearly records the evolution path of the states, providing high-quality, reliable input data for subsequent manual review, multimodal model analysis, anomaly localization, or database retrieval. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a black-box application state progression and timing scene capture method according to an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of a temporal scene evidence chain in one embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] In current fields such as automated testing and content analysis, handling black-box applications presents significant challenges. These applications, especially those heavily reliant on canvas, 3D graphics viewports, or remote streaming technologies, have inaccessible internal state variables, uncallable engine interfaces, and their interface elements (such as buttons and menus) do not exist as standard page controls. This renders traditional automation methods based on control semantics or internal APIs ineffective. Existing technologies often struggle to effectively determine the application's loading status, advance its logical flow, and capture scenarios that demonstrate core functionality without relying on source code or engines.

[0021] This invention proposes an innovative solution: constructing a closed-loop control system based on pure visual feedback. This system does not rely on any prior knowledge of the application's internal workings, but rather intelligently explores and advances the application's state through a continuous "action-feedback-comparison-decision" cycle. Its core lies in executing standardized physical input actions and analyzing the visual changes in the screen after the actions to determine whether a new, valuable state has been generated. Based on this, it generates evidence containing complete context that can be subsequently verified manually or by machine.

[0022] The black-box application state advancement and timing scene capture method in this application embodiment is executed by a computer device, which may be a local computer, server, cloud automation node, test terminal, mobile device simulator, remote desktop control node, or other computer device with screen capture and input simulation capabilities.

[0023] In a basic embodiment, the black-box application state advancement and timing scene capture method provided by the present invention specifically includes: Step S110: Determine the rendering area that carries the visual content of the target black-box application, and establish the spatial reference of the rendering area.

[0024] Step S120: Determine the interactive timing of the target black-box application based on the changes in the rendering area, and add the rendering area image corresponding to the interactive timing as an initial observable state node to the set of historical valid state nodes.

[0025] Step S130: Generate a queue of candidate physical actions based on the spatial reference and decoupled from the engine of the target black-box application.

[0026] Step S140: When the interaction is active, at least some of the actions in the candidate physical action queue are executed sequentially within the rendering area, and the image of the rendering area after each action is executed is captured as the post-action feedback image.

[0027] Step S150: Compare the visual content differences between the post-action feedback screen and the screens of one or more historical valid state nodes in the historical valid state node set. When a post-action feedback screen does not have a preset similarity relationship with the screens of the compared historical valid state nodes, the post-action feedback screen is added as a new observable state node to the historical valid state node set.

[0028] Step S160: When a new observable state node cannot be generated by executing actions in the candidate physical action queue within the preset exploration period, a backup exploration strategy is triggered to generate supplementary physical input actions, and the supplementary physical input actions are added to the candidate physical action queue for subsequent execution.

[0029] Step S170: After the execution of the action meets the stopping condition, based on all observable state nodes in the historical valid state node set and their associated actions, organize and output a structured temporal scene evidence chain.

[0030] The aforementioned technical solution, through closed-loop control and evidence generation mechanisms, delivers significant technical benefits. It greatly reduces the automation system's dependence on application source code, engine interfaces, and page control semantics, making it widely applicable to various black-box environments. Simultaneously, through intelligent state determination and weak-response escape strategies, it effectively improves the success rate of exploring and entering core functional states in unknown applications, ultimately generating high-value evidence with rich context and quality far exceeding that of a single screenshot.

[0031] The above technical solution is explained in detail below: The method first executes step S110 to determine the rendering area carrying the visual content of the target black-box application and establish a spatial reference for unifying input coordinates and the visual content acquisition range. Specifically, the computer device obtains the application's entry point (such as a URL or executable file path) and launches the application in its runtime environment (such as a browser or virtual machine). Subsequently, the actual rendering area carrying the visual content is located, such as the `` element in an HTML page. <canvas>`Element` <iframe>A video stream window for a container or remote desktop. Once located, the computer device records its screen position, width, and height to form a spatial reference. This design ensures that subsequent image capture and input simulation operations are based on the same coordinate system, thus achieving precise correspondence between input and output and avoiding interaction failures caused by coordinate mismatch.

[0032] Next, the method executes step S120, which analyzes the visual changes in the frame sequence of the rendered area to determine the interactive timing of the target black-box application. The rendered area is continuously captured at a preset frequency (e.g., 10 times per second). By comparing the visual differences between consecutive frames, the computer device can determine whether the application is currently in a loading animation, video advertisement, static menu, or other state. Thus, the computer device can dynamically determine when the application transitions from the loading or transition phase to a stable interactive state, avoiding premature interaction or unnecessary delays caused by fixed waiting times, significantly improving the efficiency and robustness of the automated process.

[0033] After determining the interactive timing, the process proceeds to step S130, where the computer device generates a series of standardized physical input actions based on a spatial reference. These actions are independent of the application's internal logic and the semantics of the interface controls (decoupled from the engine of the target black-box application) to form a candidate physical action queue. These actions are not targeted at specific buttons or controls, but are general physical interactions based on spatial distribution, such as clicking the center point of the rendering area, clicking the bottom central area, or swiping from left to right. This design, which is decoupled from the application logic, allows this method to conduct preliminary exploration of any black-box application without prior knowledge of the application's interface layout, and has extremely high universality.

[0034] Subsequently, in step S140, the computer device retrieves an action from the candidate physical action queue, executes the action within the rendering area, and captures the current rendering area image after the action is executed as the post-action feedback image. For example, after executing a click action, the computer device will wait for a short redraw time to ensure that the application has enough time to respond to the input and update the image before taking a screenshot. This step constitutes a key link in the "action-feedback" loop, providing the necessary visual input for subsequent state determination.

[0035] Step S150 is the core decision-making step of the present invention. The computer device compares the visual content differences between the action feedback screen collected in step S140 and the screens of one or more historical valid state nodes in the historical valid state node set.The computer device performs a new state condition judgment. If the post-action feedback screen is sufficiently "new" compared with all historical states (that is, does not form a preset similarity relationship), it is determined to be a new observable state, and the operation of generating a new node is performed, and the information such as the post-action feedback screen and its triggering action is taken as a new observable state node and added to the historical valid state node set. Conversely, if the post-action feedback screen is highly similar to a certain historical state (for example, the screen does not change after clicking a non-interactive area), it is determined to be an invalid or duplicate state, and the operation of refusing to link in is performed (that is, it is not added to the historical valid state node set). Through this mechanism, the computer device can effectively filter out meaningless animation frames or duplicate screens caused by invalid operations, and ensure that the evidence chain only contains key nodes that truly represent state transitions.

[0036] When the computer device encounters a difficulty during the exploration process, step S160 is triggered. When a new observable state node fails to be generated by executing the actions in the candidate physical action queue within a preset exploration period (for example, the screen does not change after consecutive multiple operations), a weak response typing judgment is performed, and a backup exploration strategy is triggered to generate supplementary physical input actions (appending supplementary exploration actions). These supplementary actions are added to the candidate physical action queue for subsequent execution. This mechanism enables the computer device to not give up easily when facing "weak response" scenarios such as menu stuck, authorization mask, and novice guide blockage, but actively try more diverse exploration behaviors, thereby significantly improving the ability to get out of trouble and continue to advance the state.

[0037] Finally, after the stop condition is met (such as reaching a preset number of nodes or exploration duration), the process proceeds to step S170. The computer device organizes and outputs a structured time-series scenario evidence chain based on all observable state nodes in the historical valid state node set and their associated actions. With reference to Figure 2, exemplarily, the evidence chain clearly shows a complete path starting from the initial state node N0, reaching state node N1 by executing action A1, then reaching state node N2 by action A2, and so on. The output content includes a time-sorted state screenshot sequence (composed of post-action feedback screens), action records triggering each state, and the like. This context-rich evidence chain provides valuable information far exceeding a single screenshot for subsequent manual review or multimodal model understanding.

[0038] In a specific implementation, the backup exploration strategy includes: dividing the rendering area into grids to obtain a number of grids, and generating a series of supplementary physical input actions that sequentially click on the center point of each grid.

[0039] For example, the rendering area is divided into a 5x5 grid, and 25 actions are generated to click the center of each grid in sequence. This "carpet-bombing" exploration strategy can trigger interactive elements in special positions that are not covered by standard actions with a high probability, effectively dealing with menus or buttons with unknown layouts, thereby enhancing the ability to get out of trouble.

[0040] Furthermore, the preset exploration period can be determined by counting the actions that fail to generate new nodes consecutively. When the count reaches a preset number (e.g., 10 times), a backup exploration strategy is triggered.

[0041] In one specific embodiment, the interactive timing of the target black box application is determined based on the changes in the rendering area. Specifically, this includes: continuously acquiring the rendering area; calculating the image difference between adjacent images; when the image difference is continuously lower than a preset stable threshold, determining that the target black box application has entered a stable interactive state, and the interactive timing is when the target black box application is in a stable interactive state.

[0042] For example, a computer device can scale the screen to a smaller size (e.g., 128x128 pixels) and convert it to grayscale, then calculate the average absolute pixel difference (MAD) between two adjacent frames. When the MAD value is below a stable threshold (e.g., 5.0) for N consecutive frames (e.g., N=5), and the current screen is not pure black or pure white, the application screen can be considered stable and has entered a stable interactive state, which is the time for interaction. This data-driven judgment method is more accurate and reliable than fixed delay.

[0043] In one specific embodiment, visual content difference comparison is achieved by calculating at least one of image hash distance, structural similarity SSIM, mean absolute pixel difference MAD, or perceptual hash; the similarity relationship is determined by comparing the result of visual content difference comparison with a threshold dynamically adjusted according to the exploration phase.

[0044] For example, in the early stages of exploration, the user may be in the application's start menu interface, which often has background animations or breathing light effects. At this time, the threshold for perceptual hash distance can be set to a relatively high 10 to tolerate these meaningless visual changes. However, when entering the core gameplay interface, in order to capture subtle but crucial state changes such as character movement and item usage, the threshold can be dynamically lowered to 3. This dynamic threshold strategy makes state determination more intelligent and accurate, effectively improving the quality of evidence.

[0045] In one specific embodiment, the generation of a candidate physical action queue based on a spatial reference and decoupled from the engine of the target black-box application specifically includes: generating a click action or a swipe action based on a proportional coordinate relative to the spatial reference; and converting the proportional coordinate into an absolute coordinate within the rendering area according to the position and size of the spatial reference for the execution of the click action or swipe action.

[0046] For example, to generate an action that clicks the lower right corner of the rendering area, its proportional coordinates can be defined as (0.8, 0.8). If the upper left corner screen coordinates of the spatial reference are (100, 200) and the size is 800x600 pixels, then the final absolute click coordinates are (100 + 800 0.8, 200 + 600 0.8) = (740, 680). By using proportional coordinates, the generated action sequence is decoupled from the specific resolution and window position, resulting in better portability.

[0047] In one specific embodiment, the temporal scene evidence chain includes: a grid evidence diagram formed by stitching together screenshots (feedback screens after actions) of multiple observable state nodes in the set of historical valid state nodes in chronological order.

[0048] For example, the five captured key state screenshots (such as the loading page, start page, level selection page, gameplay page, and score page) are stitched together into a 2x3 grid image. This grid-based evidence image can compactly display the entire state progression within a single image, greatly facilitating manual review and rapid understanding.

[0049] The method of this invention is particularly applicable to certain types of black-box applications, namely applications running in a browser environment and rendered using HTML5 Canvas or WebGL, or cloud applications that present images via remote streaming protocols. For these applications, their interface elements are drawn entirely at the pixel level and cannot be interacted with using conventional web automation tools. The pure visual closed-loop feedback mechanism of this invention can precisely solve this pain point, providing an effective technical approach for its automated state exploration and scene capture.

[0050] For example, the target black box application is a black box game.

[0051] The following example illustrates the above-mentioned method for advancing the state of a black box application and capturing a time sequence scene, taking a black box game as the target black box application.

[0052] In an exemplary embodiment, the black-box application state advancement and timing scene capture method provided by the present invention specifically includes: S1: Establish a black-box game runtime environment, locate the game rendering area, and establish a spatial reference.

[0053] The computer device obtains the target game's local directory, online page address, remote streaming entry point, or emulator running entry point, and opens the corresponding running page or running sandbox of the target game. During the running process, it does not read the game's internal state, does not call the game engine interface, and does not rely on the page control tree or DOM control semantics to determine the real button.

[0054] The computer device locates the rendering area that carries the game screen. This rendering area can be a canvas element, a 3D graphics rendering viewport, an embedded page container, an encapsulated runtime viewport, a remote streaming screen area, a mobile device emulator screen, or a virtual machine window. For the canvas area, the canvas drawing width and height can be read, and the browser viewport size can be adjusted accordingly; for the embedded container, the container bounding box can be obtained; for the remote screen or emulator screen, its screen area boundary can be determined.

[0055] This step outputs the spatial reference of the rendering area, including its position, width, height, scaling ratio, or viewport size. This spatial reference is used in subsequent steps for both image capture and cropping and physical input motion coordinate conversion, thereby avoiding inconsistencies between the screenshot area and the input area.

[0056] S2: Determine the timing of loading, transition, or interaction based on continuous screen difference.

[0057] The computer device continuously captures images of the game rendering area according to a preset cycle, and converts adjacent images into grayscale thumbnails or other image representations suitable for difference calculation. The computer device calculates the image differences between adjacent images.In one implementation, the average absolute pixel difference is used. : ; in, Indicates the current frame In coordinates grayscale value at that location Indicates the previous frame In coordinates grayscale value at that location and These represent the width and height of the thumbnail, respectively.

[0058] When the image difference is greater than or equal to the motion threshold and the minimum observation time is met, the computer device determines that the game screen has completed loading, advertising, or initialization. When the image difference is continuously lower than the stability threshold and the current screen is not blank or black, the computer device determines that the target game is in a stable and interactive state. When the above conditions are not met even after the maximum waiting time is exceeded, a conservative exploration process can be entered to avoid infinite waiting.

[0059] This step outputs the interactive timing determination result, the starting screen and differential statistics information, which are used for subsequent generation of action queues and initialization of state nodes.

[0060] S3: Generate a candidate physical action queue based on spatial reference and decoupled from the game engine.

[0061] The computer device generates a candidate physical action queue based on the spatial reference of the rendering area obtained in step S1. The candidate actions may include keyboard key presses, mouse clicks, mouse drags, touch clicks, touch slides, gamepad buttons, joystick direction inputs, or scroll wheel inputs.

[0062] For click-type actions, multiple proportional coordinates relative to the rendering area can be preset or dynamically generated, such as the center of the screen, the lower middle part, the bottom confirmation area, the left and right navigation areas, the level selection area, or a possible closing mask area. Proportional coordinates ( This can be converted to absolute screen coordinates. ): ; ; in, and The coordinates of the top-left corner of the bounding box of the rendering area. and This defines the width and height of the rendering area.

[0063] This step outputs a candidate physical action queue. This action queue is not a recorded user script, nor is it an operation sequence based on a certain in-game object ID, but rather an engine-independent input set generated based on the rendering region spatial reference.

[0064] S4: Execute candidate physical actions and collect game feedback screens after the actions.

[0065] The computer device executes one or a group of actions from the candidate physical action queue in the game rendering area. After the action is executed, the computer device waits for a preset redraw time or determines the end of the wait based on the changes in the screen, and then captures the screen of the game rendering area again to obtain the feedback screen after the action.

[0066] The output of this step is the post-action feedback screen and the action information that triggered the feedback screen, including the action type, proportional coordinates or button presses, action rounds, execution time, and waiting time. This output is not directly saved as the final screenshot, but is used for the status node judgment in step S5.

[0067] S5: Compare the post-action feedback screen with the historical valid state nodes, and generate or reject new observable state nodes.

[0068] The computer device compares the post-action feedback screen with the images corresponding to the saved historical valid state nodes. To reduce computation and preserve state loopback, it can prioritize comparison with the K most recent valid state nodes.

[0069] When the post-action feedback screen meets the new state determination conditions relative to all historical valid state nodes, the computer device generates a new observable state node for the feedback screen and records the triggering action, time, round, differential result, node number, and necessary anomaly markers. When the post-action feedback screen does not meet the new state conditions, the computer device refuses to add it to the state node chain and accumulates the weak response count or updates the current screen state.

[0070] In one embodiment, the new state determination condition includes a dynamic deduplication threshold. A higher threshold is used in the early stages of exploration to filter out background flickering, breathing animations, or low-value particle effects on the start page; a lower threshold is used in subsequent exploration stages to retain fine-grained state changes such as character movement, elimination actions, object changes, or perspective changes in the gameplay screen.

[0071] This step outputs the updated set of observable state nodes, node entry results, weak response counts, and action-state correspondences. This step is the key step that distinguishes this application from "directly taking a screenshot and saving it after the action" and "game state vector parsing". Only the feedback screen after the action that meets the new state determination conditions will be saved as a screenshot.

[0072] S6: When no new observable state nodes are formed consecutively, additional exploration actions are added and the closed-loop process continues.

[0073] When at least one of the following conditions is met, the computer device determines that the target game is in a weak response or stuck state: no new observable state nodes are formed after multiple rounds of physical input actions are executed; the number of saved state nodes is less than a preset number; the current screen remains blank, black screen, loading, menu, level selection, authorization mask, novice guide blocking or emulator stagnation state for a long time; the image difference between the current screen and the historical valid state nodes is continuously lower than a preset threshold.

[0074] The computer device generates supplementary exploration actions based on the weak response type.Additional exploration actions may include: generating click actions with gridded proportional coordinates within the rendering area; generating click actions covering the center area of ​​the screen, the bottom confirmation area, the left and right navigation areas, the close mask area, and the high-probability area of ​​the skip button; generating back actions, menu close actions, and drag actions; generating action templates based on successful actions from similar historical games; or generating actions based on candidate interaction areas identified by the visual model.

[0075] Preferably, the computer device adds supplementary exploration actions to the end of the existing physical action queue, rather than directly replacing the existing action queue. This method allows the original keyboard, center click, bottom confirmation area click, or drag actions to continue triggering key entry points after the main menu actually appears in a slow-loading game. At the same time, the supplementary actions can expand the exploration coverage in menus, level selection, masking, or fake ready scenes.

[0076] After the supplementary action is executed, the process returns to S4 and S5 to continue forming a closed loop of action execution, feedback acquisition, status node judgment and weak response processing.

[0077] S7: Output a temporal scene evidence chain with action association based on the formed observable state nodes.

[0078] When the number of observable state nodes, exploration rounds, execution duration, or output conditions meet the requirements, the computer device outputs a temporal scene evidence chain based on the formed state nodes. The evidence chain may include a sequence of valid state screenshots (feedback screens after actions based on new state judgment conditions) saved in the collection order, compatible screenshots of the start page, compatible screenshots of the gameplay page, a grid evidence diagram formed by stitching together multiple state screenshots in chronological order, a log of the correspondence between actions and state nodes, state paths, game identifiers, screenshot times, differential results, grid diagram paths, anomaly markers, and other metadata.

[0079] The evidence chain is used to express the process of a black-box game progressing from loading, menu, level selection or masking state to gameplay state. The output is not a single screenshot, nor a regular test report, but a set of action-state-time evidence that can be used for manual review, multimodal understanding, automated test review, anomaly sample analysis or database retrieval.

[0080] As can be seen from the above description, the black-box application state advancement and timing scene capture method provided by the present invention has the following technical effects: 1. Reduced dependence on application source code, engine interfaces, and control semantics. This invention achieves closed-loop control entirely based on externally observable visual information and physical input, without intruding into the application's internals. Therefore, it is widely applicable to various black-box applications employing technologies such as Canvas, WebGL, and remote streaming, solving the applicability problems of traditional automation solutions.

[0081] 2. Improved the success rate and efficiency of entering the core functional state in unknown black-box applications. By dynamically judging the interactive timing, generating a standardized physical action queue, and judging new states based on visual differences, this invention constructs a universal state exploration framework, avoiding the huge cost of writing customized scripts for each application, and realizing automated batch processing of heterogeneous applications.

[0082] 3. Enhanced ability to escape from weak response or stuck scenarios. The present invention determines whether the exploration has stalled by monitoring the generation of new state nodes, and triggers alternative exploration strategies, such as grid-based clicking, when stalled, thereby effectively dealing with difficult scenarios such as false readiness, menu blocking, and authorization masking, and significantly improving the robustness of state advancement.

[0083] 4. Improved the evidentiary quality and usability of the output results. The present invention outputs not scattered screenshots, but a structured temporal scene evidence chain, which includes a sequence of state screenshots, precise actions triggering each state, and a grid-based evidence diagram. This form of output clearly records the evolution path of the states, providing high-quality and reliable input data for subsequent manual review, multimodal model analysis, anomaly localization, or database retrieval.

[0084] In an exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0085] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0086] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0087] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0089] The databases involved in the various embodiments provided in this application may include at least one of relational databases and non-relational databases. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0090] The technical features of the above embodiments can be combined in any way. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] Specific examples have been used in this document to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. In summary, the content of this specification should not be construed as a limitation of this application.< / iframe> < / canvas>

Claims

1. A method for black-box application state progression and temporal scene capture, characterized in that, include: Determine the rendering area that carries the visual content of the target black-box application, and establish the spatial reference of the rendering area; The interactive timing of the target black-box application is determined based on the changes in the rendering area, and the rendering area corresponding to the interactive timing is added to the set of historical valid state nodes as the initial observable state node. A queue of candidate physical actions is generated based on the spatial reference and decoupled from the engine of the target black-box application. When the interactive moment is reached, at least some of the actions in the candidate physical action queue are executed sequentially within the rendering area, and the image of the rendering area after each action is executed is captured as the post-action feedback image. The visual content difference of the post-action feedback screen is compared with the screens of one or more historical valid state nodes in the historical valid state node set. When a post-action feedback screen does not have a preset similarity relationship with the screens of the compared historical valid state nodes, the post-action feedback screen is added as a new observable state node to the historical valid state node set. If a new observable state node cannot be generated by executing the actions in the candidate physical action queue within the preset exploration period, an alternative exploration strategy is triggered to generate a supplementary physical input action, and the supplementary physical input action is added to the candidate physical action queue for subsequent execution. Once the execution of the action meets the stopping condition, a structured temporal scene evidence chain is organized and output based on all observable state nodes and their associated actions in the set of historical valid state nodes.

2. The black-box application state progression and timing scene capture method according to claim 1, characterized in that, The alternative exploration strategies include: The rendering area is divided into several grids, and a series of supplementary physical input actions are generated by clicking the center point of each grid in sequence.

3. The black-box application state progression and timing scene capture method according to claim 1, characterized in that, Determining the interactive timing of the target black-box application based on changes in the rendered area specifically includes: Continuously capture images of the rendered area; Calculate the image differences between adjacent frames; When the image difference remains below a preset stable threshold, the target black-box application is determined to have entered a stable and interactive state. The interactive moment is when the target black-box application is in a stable and interactive state.

4. The black-box application state progression and timing scene capture method according to claim 1, characterized in that, The visual content difference comparison is achieved by calculating at least one of image hash distance, structural similarity (SSIM), mean absolute pixel difference (MAD), or perceptual hash. The similarity relationship is determined by comparing the results of the visual content difference comparison with a threshold that is dynamically adjusted according to the exploration phase.

5. The method for black-box application state progression and timing scene capture according to claim 1, characterized in that, Based on the spatial reference, a candidate physics action queue is generated, decoupled from the engine of the target black-box application, specifically including: Generate click or swipe actions based on scaled coordinates relative to the spatial reference; The proportional coordinates are converted into absolute coordinates within the rendering area based on the position and size of the spatial reference, so as to enable the execution of the click or swipe action.

6. The black-box application state progression and timing scene capture method according to claim 1, characterized in that, The temporal scene evidence chain includes: a grid evidence diagram formed by stitching together screenshots of multiple observable state nodes from the set of historical valid state nodes in chronological order.

7. The method for black-box application state progression and timing scene capture according to claim 1, characterized in that, The target black-box application is an application running in a browser environment that uses HTML5 Canvas or WebGL for rendering, or a cloud application that presents images through a remote streaming protocol.

8. The black-box application state progression and timing scene capture method according to any one of claims 1-7, characterized in that, The target black-box application is a black-box game.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the black-box application state advancement and timing scene capture method according to any one of claims 1-8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the black-box application state advancement and timing scene capture method as described in any one of claims 1-8.

Images