Software operation flow recording method, system, electronic device and medium

Abstract

This invention discloses a method, system, electronic device, and medium for recording software operation flows. The method relates to emerging software and new information technology services, and specifically includes: continuously acquiring operation behavior information generated during the operation of the target software; acquiring the interface elements corresponding to the operation behavior information, tracing the parent elements of the interface elements upwards to the top level, and recording the tracing path as a first operation path; converting the operation behavior information into semantic operation events according to preset rules, and determining a second operation path for each semantic operation event; extracting the common path among all second operation paths, and using the common path as the root node, constructing child-level nodes of the root node based on the remaining paths of each second operation path; and filling the semantic operation events into the corresponding child-level nodes to obtain the operation flow. This invention separates the interface context from the operation by extracting the common path, reducing the maintenance cost of recording the operation flow.

Description

Technical Field

[0001] This invention relates to the field of software testing, and in particular to a method, system, electronic device, and medium for recording software operation processes. Background Technology

[0002] In emerging software and new information technology services, with the continuous increase in the functionality and interaction complexity of software systems, recording and reproducing software operation processes has become a common requirement in software testing, defect reproduction, maintenance and troubleshooting, and user behavior analysis. Accurately recording user operations on the software interface helps testers quickly locate problems, developers efficiently reproduce defects, and maintenance personnel trace abnormal operation paths. Currently, common methods for recording software operation processes mainly include macro-based operation recording, script writing based on automated testing frameworks, and event tracking based on system logs. While these solutions can capture user operations to a certain extent, their recording results in each operation being inextricably linked to its corresponding interface context information in each operation record. When the interface context undergoes partial adjustments, the context information scattered across each operation record must be modified individually, and maintenance costs increase exponentially with the number of recorded operations. Summary of the Invention

[0003] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a method, system, electronic device, and medium for recording software operation processes, which can reduce the maintenance cost of software operation records.

[0004] A method for recording software operation flow according to a first aspect embodiment of the present invention includes: In response to the recording start signal, continuously acquire operational behavior information generated during the operation of the target software; Obtain the interface element corresponding to the operation behavior information, trace the parent element of the interface element upwards to the top level, and record the tracing path as the first operation path; In response to the recording end signal, the operation behavior information is converted into semantic operation events according to preset rules, and a second operation path is determined for each semantic operation event, wherein the second operation path is consistent with the first operation path of the operation behavior information corresponding to the semantic operation event; Extract the common path between all the second operation paths, and use the common path as the root node. Based on the remaining path of each second operation path, construct the sub-level nodes of the root node. The semantic operation events are filled into the corresponding sub-level nodes to obtain the operation flow.

[0005] The embodiments of the present invention have at least the following beneficial effects: The method provided by the present invention first records user operation behavior, obtains the interface element corresponding to the operation, and traces its parent element upwards to the top level to form a first operation path; then, the discrete operations are converted into semantic operation events according to rules, and a second operation path is determined for each event; then, the common part of all second operation paths is extracted as the root node, and the remaining paths are used to construct sub-level nodes; finally, the semantic operation events are filled into the corresponding child nodes, thereby generating a reproducible operation flow. The method provided by the present invention separates the interface context from the operation by extracting the common path. When the interface is partially adjusted, only the common path needs to be modified, avoiding the need to modify each operation record one by one, and significantly reducing the maintenance cost of operation flow records.

[0006] According to some embodiments of the present invention, the step of obtaining the interface element corresponding to the operation behavior information, tracing the parent element of the interface element upwards to the top level, and recording the tracing path as the first operation path includes: In response to the element determination signal, the region of interest is determined based on the size of the parent element corresponding to the interface element; BRISK feature points are extracted within the region of interest and a unique identifier is generated. The BRISK feature points are then associated with the unique identifier, wherein the BRISK feature points are used to replace the interface elements and the unique identifier is used to replace the first operation path.

[0007] According to some embodiments of the present invention, the step of converting the operation behavior information into semantic operation events according to preset rules and determining a second operation path for each semantic operation event includes: The operation behavior information is traversed in chronological order, wherein the event types of the operation behavior information include keyboard key actions, mouse wheel actions, and mouse button actions. When multiple consecutive keyboard key presses are encountered during the traversal, the keyboard key presses are merged to generate a string event. When multiple consecutive mouse wheel actions are encountered during the traversal, the mouse wheel actions are merged to generate a scroll event, wherein the scroll amount of the scroll event is the sum of the scroll amounts of the merged mouse wheel actions. When the mouse button behavior is encountered during the traversal, the mouse button behavior is classified as a click event or a drag event based on the movement distance between the press signal and the release signal in the mouse button behavior.

[0008] According to some embodiments of the present invention, the step of continuously acquiring operational behavior information generated during the operation of the target software further includes: When an abnormal interface appears during the operation of the target software, continuously acquire abnormal response information for the abnormal interface and the abnormal element corresponding to the abnormal interface, trace the parent element of the abnormal element upwards to the top level to obtain the first response path. The abnormal response information is converted into abnormal response events according to preset rules, and a second response path is determined for each abnormal response event; An abnormal response process is obtained based on the second response path and the abnormal response event, and the abnormal response process is associated with and stored with the operation process.

[0009] According to some embodiments of the present invention, after the step of obtaining the operation flow, the method further includes: The operation process is reproduced. When the semantic operation event fails to execute during the reproduction process, the current interface elements of the target software are obtained; If the interface element is the same as the abnormal element, then the abnormal response process is invoked based on the storage association between the operation process and the abnormal response process.

[0010] According to some embodiments of the present invention, after the step of obtaining the operation flow, the method further includes: The operation process is reproduced. If the target operation event fails to execute during the reproduction process, the target operation event is immediately re-executed. If the target operation event still fails to execute, the most recently successfully executed semantic operation event is obtained and re-executed, and then the target operation event is re-executed. If the target operation event still fails to execute, the process continues to backtrack to the earlier successfully executed semantic operation event. In each backtracking level, the event at that level and every semantic operation event between that level and the target operation event are re-executed in sequence, until the target operation event is successfully executed or the preset maximum backtracking level is reached.

[0011] According to some embodiments of the present invention, the step of obtaining the interface element corresponding to the operation behavior information further includes: When the interface element is acquired, it is highlighted on the operable interface of the target software to indicate that the interface element has been successfully recognized.

[0012] A software operation flow recording system according to a second aspect embodiment of the present invention includes: The process recording module is used to continuously acquire operational behavior information generated during the operation of the target software in response to the recording start signal; The process recording module is also used to obtain the interface element corresponding to the operation behavior information, trace the parent element of the interface element upwards to the top level, and record the tracing path as the first operation path. The process recording module is also used to respond to the recording end signal, convert the operation behavior information into semantic operation events according to preset rules, and determine a second operation path for each semantic operation event, wherein the second operation path is consistent with the first operation path of the operation behavior information corresponding to the semantic operation event; The process generation module extracts the common path between all the second operation paths, and uses the common path as the root node to construct the sub-level nodes of the root node according to the remaining path of each second operation path. The process generation module is also used to fill the semantic operation events into the corresponding sub-level nodes to obtain the operation process.

[0013] An electronic device according to a third aspect of the present invention includes a memory and a processor, the memory being used to store at least one program, and the processor being used to load the at least one program to execute the recording method of the software operation flow described in the above-described aspect embodiments.

[0014] A computer-readable storage medium according to a fourth aspect of the present invention includes a memory and a processor, the memory being used to store at least one program, and the processor being used to load the at least one program to perform the recording method of the software operation flow described in the above-described aspect of the embodiments.

[0015] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0016] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a software operation process recording method according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the relationship between semantic operation events and operation behavior information in an embodiment of the present invention; Figure 3 This is a structural diagram of an electronic device provided in another embodiment of the present invention. Detailed Implementation

[0017] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0018] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," "exceeding," etc. are understood to exclude the stated number, and "above," "below," "within," etc. are understood to include the stated number. If "first," "second," etc. are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of the indicated technical features.

[0019] The concepts involved in this invention are explained below: Interface elements: This is a collective term for all visible and operable components in a software user interface, including non-interactive components such as windows, panels, containers, static text, and dividers, as well as interactive controls such as buttons, input boxes, drop-down lists, and checkboxes. BRISK features: Binary Robust Invariant Scalable Keypoints, is a local image feature description algorithm in the field of computer vision. It detects interest points with significant gradient changes in an image and generates a binary descriptor for each interest point.

[0020] Please refer to Figures 1-2 , Figure 1 This is a flowchart illustrating a method for recording software operation flow according to an embodiment of the present invention. The method includes, but is not limited to, the following steps: Step S100: In response to the recording start signal, continuously acquire operation behavior information generated during the operation of the target software; It should be noted that the target software's operational behavior information includes keyboard events, mouse events, etc. In this embodiment of the invention, keyboard hook functions (keyboard.hook()) and mouse hook functions (mouse.hook()) can be registered to listen for keyboard events, utilizing the global hook mechanism provided by the operating system to capture user actions such as keyboard presses and releases, mouse movements, clicks, and scroll wheels in any application. The keyboard and mouse hooks can employ system-level hooks to ensure interception and recording are completed before input events are distributed to the target software. Furthermore, the recording start signal can be triggered by the user using a preset shortcut key; for example, pressing Ctrl+Alt+R triggers the start of recording, and pressing the same key combination or another preset key combination triggers the end of recording. The acquired operational behavior information is added to an event list (event_list) in real time. This event list serves as a cache container for operational behavior information during the recording phase, saving the original operational events in chronological order for subsequent structured processing of semantic operational events.

[0021] Step S200: Obtain the interface element corresponding to the operation behavior information, trace the parent element of the interface element upwards to the top level, and record the tracing path as the first operation path. It should be noted that macro recording and some automation tools use the screen pixel coordinates at the time of the operation as the basis for recording and playback. Once the window position of the target software shifts, the interface controls are rearranged due to version updates, or the display resolution of the operating environment changes, the original coordinates no longer point to the correct interface element, resulting in reproduction failure. Step S200 obtains the interface element corresponding to the operation behavior information and traces the parent element upwards along the interface element hierarchy to the top root node. The identification information of each level of interface element traversed during the tracing process is sequentially combined to form a first operation path based on the interface element hierarchy relationship. This operation path does not depend on the physical screen coordinates but is anchored to the logical position of the interface element in the interface hierarchy structure. This ensures that the operation positioning can still accurately match the target interface element after changes in screen layout and display parameters, significantly improving the stability and environmental adaptability of operation process reproduction. In some embodiments of the present invention, the UI automation interface is used to obtain the interface element object corresponding to the operation position coordinates in the operation behavior information. For example, by creating a desktop automation object (such as pywinauto.Desktop), and calling a coordinate positioning method (such as from_point()) with the screen coordinates of the operation position as input, the top-level UI element wrapper at those coordinates is obtained. This wrapper encapsulates the UI element's attribute information, including the element name, control type, bounding rectangle, and references to its parent element. Further, the process of tracing parent elements upwards to the top level is achieved by recursively calling the parent element retrieval function (such as top_level_parent()). Starting from the current UI element, its parent element references are accessed level by level, recording the identification information of each level of UI element sequentially, until the top-level container of the current window or the desktop root node is reached, ultimately forming a complete tracing path from the top level to the current UI element. Each node in this tracing path can record one or more attribute information such as the UI element's name, control type, and automation identifier.

[0022] Step S300: In response to the recording end signal, the operation behavior information is converted into semantic operation events according to preset rules, and a second operation path is determined for each semantic operation event, wherein the second operation path is consistent with the first operation path of the operation behavior information corresponding to the semantic operation event. It should be noted that existing operation recording schemes and the operation behavior information directly obtained from the target software in step S100 store and reproduce the underlying input events directly as recording units, resulting in a large number of recorded events, low semantic level, and inability to reflect the user's true operation intent. Step S300 transforms the operation behavior information obtained in step S100 through preset rules, such as merging consecutive keyboard key presses, merging mouse wheel scrolling, and distinguishing mouse clicks. Taking merging keyboard key presses as an example, if a series of consecutive operation behavior information is '1', '2', '3', that is, three consecutive keyboard key presses, in the existing technology these three events are independent behaviors. However, this embodiment transforms the above three behaviors into an independent operation through preset semantic rules, and combines the interface elements corresponding to the three behaviors to obtain a semantic operation event with semantic meaning, such as the event {password input -123}. Further, referring to... Figure 2 A second operation path corresponds to a semantic operation event, and the second operation path is essentially the same as the first operation path corresponding to the interface element. For example, if semantic operation event 2 is converted from a single operation behavior information 4, then the first operation path B corresponding to operation behavior information 4 is the second operation path B of semantic operation event 2; semantic operation event 1 is converted from multiple consecutive operation behavior information 1, 2, 3 (such as the {password input - 123} event in the above embodiment of the invention), then since the first operation path A corresponding to each keyboard key action is the same, the second operation path A of semantic operation event 2 is the first operation path A corresponding to the same keyboard key action of the interface element.

[0023] Step S400: Extract the common path between all second operation paths, and construct the child level nodes of the root node based on the common path as the root node and the remaining path of each second operation path. It should be noted that in existing technologies, each operation independently records its complete path from the top-level container to the target control. When a user performs multiple operations in the same dialog box or menu, the same parent path segment appears repeatedly in each operation record, causing the process data to swell and the interface context relationship between operations to be fragmented. The execution engine must parse the complete path of each operation one by one to infer whether they belong to the same parent container, making it impossible to directly utilize hierarchical relationships for efficient location and maintenance. Step S400 extracts the common path between the second operation paths of all semantic operation events, abstracts the interface context shared by each operation into a single root node, and constructs the path parts unique to each operation (the remaining paths excluding the common path) into child-level nodes belonging to this root node. This reorganizes the originally flat and redundant path set into a hierarchical structure containing common and differential levels, making the hierarchical relationship between operations clear in the structure.

[0024] Specifically, the process of extracting the common path is achieved through step-by-step comparison. The second operation path of the first semantic operation event is used as the initial common path, which is then divided into a list of nodes by a delimiter. The second operation path of each subsequent semantic operation event is traversed, similarly divided into a list of nodes, and comparisons are performed step-by-step starting from the first-level nodes. When two paths have the same node identifier (such as UI element name, control type, etc.) at the same level, that node is retained in the common path, and the comparison continues to the next level. When a mismatch occurs or one path reaches its end, the comparison terminates, and the currently accumulated sequence of matching nodes is the longest common path obtained after comparing all the second operation paths.

[0025] Specifically, the process of constructing sub-level nodes is based on the remaining paths of each second operation path. Using the common path as the root node, the prefix portion overlapping with the common path is removed from each second operation path to obtain the remaining path. This remaining path is then hierarchically divided into a sequence of nodes. Starting from the root node of the common path, each level of the node sequence is checked to see if it already exists in the currently constructed hierarchical structure. If not, a new sub-level node is created and attached to the parent node. If it already exists, the existing node is reused, and the next level node is processed. After traversal, the remaining hierarchical information of all second operation paths is organized into a tree-like hierarchical structure with the common path as the root node. In this structure, each leaf node or intermediate node can be associated with one or more semantic operation events, indicating that these operations occur on the interface element corresponding to that node.

[0026] Step S500: Fill the semantic operation events into the corresponding sub-level nodes to obtain the operation flow.

[0027] It should be noted that step S500 fills the semantic operation events into the corresponding sub-level nodes in the hierarchical description structure according to their generation order in the event list, so that the final generated operation flow data simultaneously possesses the temporal order of operations and the hierarchical relationship of interface elements. During reproduction, the engine can directly use this hierarchical structure to locate the context level by level without repeatedly parsing the complete paths of each operation. Furthermore, when the interface element corresponding to a certain level node changes, only the corresponding node needs to be adjusted to maintain the correct execution of subsequent operations. It is understandable that the filling process of the semantic operation events obtained from the event list strictly follows the generation order of the semantic operation events. Traversing the processed sequence of semantic operation events, for each semantic operation event, the innermost child node in the constructed hierarchical structure tree is matched according to its second operation path, and the semantic operation event is attached as a child item of that node. After filling, the leaf nodes and intermediate nodes of the hierarchical structure tree store their respective specific operations in chronological order, allowing multiple operations occurring on the same interface element to be executed sequentially according to their occurrence order.

[0028] Specifically, in one embodiment of the present invention, the constructed sub-level nodes correspond to nested context management code blocks in the specific implementation. For example, in an embodiment for generating an executable script, the common path generates an outermost context manager (such as the `with UIPath("main window")` code block), each unique remaining path node corresponds to an inner nested context manager, and the specific operation functions of semantic operation events (such as `click()`, `send_keys()`, etc.) are placed in the innermost corresponding code block, so that the final generated script presents an indentation hierarchy that corresponds one-to-one with the interface hierarchy in terms of structure, improving readability and maintainability. Specific embodiments are shown below: with UIPath("main window"): with UIPath("toolbar"): with UIPath("File menu"): Click ("New") Click "Save". with UIPath("Edit Menu"): Click ("Copy") with UIPath("menu bar"):# New branch click("help"); From the above nested structure, the following four complete hierarchical paths can be extracted: 1. Click "New" in the main window → Toolbar → File menu → New; 2. Click "Save" in the main window → toolbar → file menu → save; 3. Click "Copy" in the main window → toolbar → edit menu → copy; 4. Click the "Help" main window → menu bar → Help.

[0029] After semantic operation events are populated into child-level nodes, the recorded operation flow can be directly parsed and used by subsequent systems or modules by writing it to a Python script file. The specific process includes: creating a script text containing fixed import statements and template code; iterating through all processed event objects; calling the corresponding generation function (click, send_keys, menu_click, etc.) based on the event type; adding the generated code lines sequentially to the script text; and saving the file in UTF-8 encoding using the codecs.open() function. Alternatively, the operation flow can be persistently stored using formats or methods such as clipboard and JSON.

[0030] It should be noted that in steps S100 to S500, by extracting the common path between the operation paths associated with each semantic operation event as the root node, constructing the remaining parts of each path as sub-level nodes, and filling the semantic operation events into the corresponding sub-level nodes, the common part of the interface context and the difference part of each operation in the operation flow can be separated. Compared with the existing flat recording method that binds the interface context and each operation inseparably, and requires modifying each operation record one by one when the interface is partially adjusted, this invention centrally maintains the common part of the interface context in the common path and the upper-level node. When the interface is partially adjusted, only the corresponding level node needs to be modified, without modifying each operation record one by one, which significantly reduces the maintenance cost of the operation flow. At the same time, it makes the hierarchical relationship between operations explicit in the flow data, improving the readability and maintainability of the flow data.

[0031] Furthermore, step S200 in the above-described embodiments also includes, but is not limited to, the following steps: Step S210: In response to the element determination signal, determine the region of interest based on the size of the parent element corresponding to the interface element; Step S220: Extract BRISK feature points from the region of interest and generate a unique identifier. Associate the BRISK feature points with the unique identifier. The BRISK feature points are used to replace interface elements, and the unique identifier is used to replace the first operation path.

[0032] It should be noted that most existing software operation flow recording schemes rely on standard control interfaces provided by the operating system or UI framework. These schemes can only identify and locate conventional controls with independent handles and standard hierarchies, and cannot effectively identify non-standard interface elements such as custom-drawn interfaces, DirectUI, Qt canvas, and Canvas rendering. To address this issue, this embodiment of the invention also provides steps S210-S220, whereby, during or after the recording of the target software's operation flow, the system can be prompted to enter a "custom-drawn element" recognition mode via a combination key or any triggering method. In this mode, the system first checks what interface element corresponds to the current operation behavior information. Since the current interface element is not the correct element, the system also needs to find the parent element of the interface element and determine the region of interest (i.e., determine the area where the required custom-drawn control may exist) based on the size of the parent element. Further, step S220 extracts BRISK feature points within the determined region of interest. BRISK feature points are robust to changes in lighting, scaling, and rotation, and can stably characterize the visual features of the control area. After extraction, a globally unique identifier (such as a UUID) is generated. The coordinate data of the extracted BRISK feature points are associated with this unique identifier and persistently stored in the feature file. In this way, the current operation behavior information converts the corresponding interface into BRISK feature points, and its corresponding first operation path is isolated through the unique identifier. The second operation path determined by subsequent semantic operation events is also this unique identifier. During the reproduction phase, when the second operation path of a certain semantic operation event in the operation flow data is the unique identifier instead of the standard hierarchical path, it indicates that the operation corresponds to a non-standard control. The reproduction engine loads the BRISK feature points associated with the unique identifier in the feature file, re-extracts BRISK feature points within the same region of interest inside the parent element, and uses a feature matching algorithm (such as BFMatcher) to align and match the current feature points with the stored reference feature points. After sorting by matching confidence, the top few matching points are selected to calculate the minimum bounding rectangle. The coordinates of the center point of this minimum bounding rectangle are used as the actual position of the target control in the current interface, completing the precise positioning and operation execution of the self-drawn element.

[0033] Furthermore, step S300 in the above-described embodiments also includes, but is not limited to, the following steps: Step S310: Traverse the operation behavior information in chronological order. The event types of the operation behavior information include keyboard key actions, mouse wheel actions, and mouse button actions. Step S320: When multiple consecutive keyboard key presses are encountered during the traversal, the keyboard key presses are merged to generate a string event. Step S330: When multiple consecutive mouse wheel actions are encountered during the traversal, the mouse wheel actions are merged to generate a scroll event, wherein the scroll amount of the scroll event is the sum of the scroll amounts of the merged mouse wheel actions. It should be noted that steps S310 to S330 are embodiments of the semantic operation event generated based on preset rules in step S300. In one embodiment of the present invention, during the traversal, the press events of keys 'c', 'l', 'i', 'c', and 'k' are detected sequentially. If the time interval of each event meets the continuity condition, they are merged into a string event "click". After merging, the second operation path of the string event is consistent with the first operation path of each merged keyboard key behavior—these original key behaviors all act on the same interface element (such as the currently focused input box), and their first operation paths are the same. Therefore, the merged string event inherits the first operation path as its second operation path. In another embodiment of the present invention, when multiple mouse wheel behaviors that are consecutive in time are encountered during the traversal, the mouse wheel behaviors are merged to generate a scrolling event as a semantic operation event. In this invention, the scroll amount (delta value) of a scroll event is the sum of the scroll amounts of all merged mouse wheel behaviors. A scroll event can be obtained by deleting the individual mouse wheel behaviors in between, retaining only one mouse wheel behavior and its scroll amount as the scroll event. The second operation path of the scroll event is consistent with the first operation path of each mouse wheel behavior. In another embodiment of the invention, when a mouse button behavior is encountered during traversal, the mouse button behavior is classified into a click event or a drag event based on the movement distance between the press and release signals. When the mouse movement distance between the press and release signals exceeds a preset threshold, it is identified as a drag event, and the relative coordinates of the starting and ending points relative to the center of their respective controls are calculated. When the movement distance does not exceed the preset threshold, it is identified as a click event, and the relative coordinates of the click position relative to the center of the control are calculated. The second operation path of the click event or drag event generated after classification is consistent with the first operation path of the original mouse button behavior.

[0034] Furthermore, step S100 in the above-described embodiments also includes, but is not limited to, the following steps: Step S110: When an abnormal interface appears during the operation of the target software, continuously obtain the abnormal response information for the abnormal interface and the abnormal element corresponding to the abnormal interface, trace the parent element of the abnormal element upwards to the top level, and obtain the first response path. Step S120: Convert the abnormal response information into abnormal response events according to preset rules, and determine the second response path for each abnormal response event; Step S130: Obtain the abnormal response process based on the second response path and the abnormal response event, and store the abnormal response process in association with the operation process.

[0035] It should be noted that during the recording of the target software's operation process, abnormal situations may occur due to network fluctuations, program loading errors, etc. In such abnormal situations, the target software or its corresponding system will display interface elements such as error messages and pop-ups. During the reproduction process, if the reproduction script cannot handle unexpected situations such as abnormal pop-ups, error messages, or unexpected interface jumps that occur during the target software's operation, the execution process can only be interrupted. Therefore, steps S110 to S130 simultaneously start recording abnormal response information for abnormal interfaces that occur during the target software's operation during the recording phase, generating abnormal response process data with a structure consistent with the normal operation process, and storing it in association with the main operation process. This allows the corresponding response measures to be automatically invoked when the same abnormality is encountered during reproduction, realizing the autonomous recovery and continuous execution of the operation process under abnormal conditions. In some embodiments of the present invention, the abnormal interface may be a modal dialog box, error message window, permission request window, or other unexpected interface changes popped up by the target software. In steps S110-S130, the recording system continuously acquires the user's abnormal response information for the abnormal interface, and simultaneously acquires the interface elements operated by the user on the abnormal interface as abnormal elements, tracing their parent elements upwards to the top level to generate the first response path. This process is executed in parallel with the recording of normal operation behavior information, sharing the same set of acquisition and path tracing mechanisms. In step S120, the recorded abnormal response information is converted into abnormal response events according to the same preset rules as the normal operation behavior information, and a second response path is determined for each abnormal response event. This conversion process also includes semantic processing such as keyboard key merging, mouse wheel merging, and mouse key classification. In step S130, based on the second response paths of each abnormal response event, a common path is extracted, sub-level nodes are constructed and filled in sequence to generate structurally complete abnormal response process data. This abnormal response process data is stored in association with the main operation process data. The association method can be based on the interface element information of the abnormal interface to establish a mapping relationship, for example, using the window title or dialog box identifier of the abnormal interface as the index key, and storing the abnormal response process data in a subdirectory of the directory where the main operation process data is located (e.g., named error_process). During the reproduction phase, when a certain semantic operation event fails to be executed and the current interface is detected as an abnormal interface, the system queries the associated abnormal response process data based on the abnormal interface information, automatically loads and executes the response operation, and then returns to the main operation process to continue execution, thereby realizing the autonomous recovery of the operation process in abnormal situations.

[0036] In addition, after step S500 in the above-described embodiments, the following steps are included, but are not limited to: Step S510: Reproduce the operation process; Step S520: When the semantic operation event fails to execute during the reproduction process, obtain the current interface elements of the target software; Step S530: If the interface element is the same as the abnormal element, the abnormal response process is invoked based on the storage association between the operation process and the abnormal response process.

[0037] It should be noted that steps S510 to S530 address the problem that existing operation recording solutions lack automatic response capabilities when encountering abnormal interfaces during the reproduction phase, leading to process interruptions and requiring manual intervention to continue. Existing solutions only record the normal operation path during recording. During reproduction, if the target software displays unexpected error windows, permission prompts, or modal dialog boxes, the execution engine cannot handle the situation autonomously due to the lack of corresponding processing strategies, and can only pause execution and wait for manual judgment and intervention. Steps S510 to S530, by detecting the current interface state when the operation fails in real time during the reproduction phase and matching interface elements with pre-stored abnormal response processes, automatically invoke the associated response process when an abnormal interface is identified. This enables the operation process to autonomously recover and continue execution in abnormal situations, significantly improving the automation and robustness of operation process reproduction. Specifically, during the reproduction process in step S510, a log recording module can be used to track the execution status of each operation, including status information such as operation start, successful completion, timeout, or failure. The reasons for the failure of the semantic operation event in step S520 may include the target interface element not existing in the current interface, the interface element being obscured and unable to be operated, or the interface element's state not meeting the operation conditions. Step S530 compares the current interface element obtained in step S520 with the abnormal elements associated with each abnormal response process stored in step S130. If the current interface element is the same as an abnormal element—for example, the current window title is the same as the abnormal window title recorded when recording an abnormal response process—then it is determined that the current interface is indeed a recorded abnormal interface. At this time, the system automatically loads the abnormal response process data corresponding to the abnormal interface according to the storage association established in step S130, executes the abnormal response events therein according to the same reproduction mechanism, and after completion, returns to the main operation process to continue executing the previously failed semantic operation events.

[0038] In some embodiments of the present invention, the matching of interface elements can adopt a fuzzy matching strategy, which does not require complete consistency. Only the key identification information (such as window class name, part of the title text, etc.) need to be the same to trigger the call of the exception handling process, so as to deal with the situation that there may be slight differences in the abnormal interface in different versions.

[0039] In addition, after step S500 in the above-described embodiments, the following steps are included, but are not limited to: Step S540: Reproduce the operation process. If the target operation event fails to execute during the reproduction process, immediately re-execute the target operation event. Step S550: If the target operation event still fails to execute, obtain the most recently successfully executed semantic operation event and re-execute it, and then re-execute the target operation event. Step S560: If the target operation event still fails to execute, continue to backtrack to the semantic operation event that was executed earlier. In each backtracking level, re-execute the event at that level and each semantic operation event between that level and the target operation event, until the target operation event is executed successfully or the preset maximum backtracking level is reached.

[0040] It should be noted that steps S540 to S560 are used to address the problem that existing operation process reproduction schemes lack automatic fault tolerance and recovery capabilities when individual operations fail due to reasons such as instantaneous system performance fluctuations, interface response delays, or incomplete activation of previous operation states. This results in the need to interrupt the entire process and rely on manual intervention for re-execution. Step S540 is triggered when the target operation event fails. After receiving the operation failure signal, the reproduction engine does not immediately terminate the process but instead calls the execution function of the target operation event again with the same parameters for an immediate retry. This immediate retry can cover occasional failures caused by instantaneous system scheduling delays or lag in control state refresh. Step S550 is executed if the immediate retry still fails. The system retrieves the most recently successfully executed semantic operation event and its corresponding second operation path from the maintained successful operation history. Based on this second operation path, it relocates the corresponding interface element and re-executes the semantic operation event, and then executes the target operation event again. By re-executing the last successful operation, the interface context that may have been lost due to the instability of the preceding operation can be restored, recreating the conditions for the correct execution of the target operation. Step S560 is triggered when the backtracking retry still fails, and the system continues to backtrack to earlier successful execution events level by level. In each level of backtracking, the system obtains the semantic operation event of that level and its second operation path, relocates and executes the event, and then executes the subsequent semantic operation events in sequence until the target operation event is tried again. This level-by-level backtracking process continues until the target operation event is executed successfully or the backtracking level reaches the preset maximum backtracking level. If the maximum backtracking level is reached and the operation is still unsuccessful, the operation is determined to have failed, an error log is recorded, and the current screen image is captured as a basis for fault diagnosis. Through this backtracking recovery strategy that progresses from near to far and deepens level by level, the system can automatically overcome multiple levels of instantaneous execution obstacles without manual intervention during the reproduction process.

[0041] Furthermore, step S200 in the above-described embodiments also includes, but is not limited to, the following steps: Step S201: When the interface element is obtained, the interface element is highlighted on the operable interface of the target software to indicate that the interface element has been successfully identified.

[0042] It should be noted that step S201, by visually highlighting the interface element on the operable interface of the target software upon acquisition, provides testers with immediate feedback on successful recognition, enabling users to detect and correct recognition anomalies at any time during recording, thereby improving the efficiency and accuracy of recording operations. In one embodiment of the present invention, the above-mentioned highlighting is achieved by creating a semi-transparent overlay. After step S200 successfully acquires the interface element corresponding to the operation behavior information through the UI automation interface, the coordinates of the boundary rectangle of the interface element on the screen are obtained, and a highlighted border of the same size and position as the boundary rectangle is drawn on the overlay in a conspicuous color (such as green). The overlay is set with a certain transparency (such as 30% to 50%) to ensure that the border is clearly visible without obscuring the interface content of the target software and without affecting the user's normal operation.

[0043] Secondly, embodiments of the present invention provide a software operation flow recording system, the system specifically including: The process recording module is used to continuously acquire operational behavior information generated during the operation of the target software in response to the recording start signal; The process recording module is also used to obtain the interface elements corresponding to the operation behavior information, trace the parent elements of the interface elements upwards to the top level, and record the tracing path as the first operation path. The process recording module is also used to respond to the recording end signal, convert the operation behavior information into semantic operation events according to preset rules, and determine the second operation path of each semantic operation event, wherein the second operation path is consistent with the first operation path of the operation behavior information corresponding to the semantic operation event. The process generation module extracts the common path between all second operation paths, and uses the common path as the root node. Based on the remaining path of each second operation path, it constructs the child level nodes of the root node. The process generation module is also used to populate the semantic operation events into the corresponding sub-level nodes to obtain the operation process.

[0044] It should be noted that the software operation process recording system provided in this embodiment of the invention achieves complete recording and structured organization of the target software operation process through the collaborative work of the process recording module and the process generation module. These two modules correspond to the recording stage and the processing stage, respectively, organically integrating functions such as operation acquisition, path tracing, semantic conversion, common path extraction, hierarchy construction, and event population. The system adopts a modular design, with the process recording module and the process generation module decoupled through an event list and operation path data, allowing the two stages to be executed asynchronously, and ensuring that acquisition efficiency is not affected by processing complexity. The final generated operation process data separates the common parts of the interface context from the differences in each operation. When the interface is partially adjusted, only the corresponding hierarchical nodes need to be modified, reducing maintenance costs. At the same time, it preserves the temporal sequence of operations and the hierarchical relationship of the interface, providing a structured data foundation for subsequent process reproduction, analysis, and reuse.

[0045] like Figure 3 As shown, Figure 3 This is a structural diagram of an electronic device provided in one embodiment of the present invention. The present invention also provides an electronic device, comprising: The processor 801 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls the recording method for executing the software operation flow of the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0046] It should be noted that the electronic device used to record the software operation process in the embodiments of the present invention, as well as any other electronic device that can be used to execute the method, are not specifically limited in the present invention.

[0047] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0048] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0049] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for recording software operation flow, characterized in that, include: In response to the recording start signal, continuously acquire operational behavior information generated during the operation of the target software; Obtain the interface element corresponding to the operation behavior information, trace the parent element of the interface element upwards to the top level, and record the tracing path as the first operation path; In response to the recording end signal, the operation behavior information is converted into semantic operation events according to preset rules, and a second operation path is determined for each semantic operation event, wherein the second operation path is consistent with the first operation path of the operation behavior information corresponding to the semantic operation event; Extract the common path between all the second operation paths, and use the common path as the root node. Based on the remaining path of each second operation path, construct the sub-level nodes of the root node. The semantic operation events are filled into the corresponding sub-level nodes to obtain the operation flow.

2. The method for recording software operation flow according to claim 1, characterized in that, The step of obtaining the interface element corresponding to the operation behavior information, tracing the parent element of the interface element upwards to the top level, and recording the tracing path as the first operation path includes: In response to the element determination signal, the region of interest is determined based on the size of the parent element corresponding to the interface element; BRISK feature points are extracted from the region of interest and a unique identifier is generated. The BRISK feature points are associated with the unique identifier, wherein the BRISK feature points are used to replace the interface elements and the unique identifier is used to replace the first operation path.

3. The method for recording software operation flow according to claim 1, characterized in that, The step of converting the operation behavior information into semantic operation events according to preset rules, and determining the second operation path for each semantic operation event, includes: The operation behavior information is traversed in chronological order, wherein the event types of the operation behavior information include keyboard key actions, mouse wheel actions, and mouse button actions. When multiple consecutive keyboard key presses are encountered during the traversal, the keyboard key presses are merged to generate a string event. When multiple consecutive mouse wheel actions are encountered during the traversal, the mouse wheel actions are merged to generate a scroll event, wherein the scroll amount of the scroll event is the sum of the scroll amounts of the merged mouse wheel actions. When the mouse button behavior is encountered during the traversal, the mouse button behavior is classified as a click event or a drag event based on the movement distance between the pressed signal and the released signal in the mouse button behavior.

4. The method for recording software operation flow according to any one of claims 1 to 3, characterized in that, The step of continuously acquiring operational behavior information generated during the operation of the target software further includes: When an abnormal interface appears during the operation of the target software, continuously acquire abnormal response information for the abnormal interface and the abnormal element corresponding to the abnormal interface, trace the parent element of the abnormal element upwards to the top level to obtain the first response path. The abnormal response information is converted into abnormal response events according to preset rules, and a second response path is determined for each abnormal response event; An abnormal response process is obtained based on the second response path and the abnormal response event, and the abnormal response process is associated with and stored with the operation process.

5. The method for recording software operation flow according to claim 4, characterized in that, After obtaining the operation flow step, the method further includes: The operation process is reproduced. When the semantic operation event fails to execute during the reproduction process, the current interface elements of the target software are obtained; If the interface element is the same as the abnormal element, then the abnormal response process is invoked based on the storage association between the operation process and the abnormal response process.

6. The method for recording software operation flow according to any one of claims 1 to 3, characterized in that, After obtaining the operation flow step, the method further includes: The operation process is reproduced. If the target operation event fails to execute during the reproduction process, the target operation event is immediately re-executed. If the target operation event still fails to execute, the most recently successfully executed semantic operation event is obtained and re-executed, and then the target operation event is re-executed. If the target operation event still fails to execute, the process continues to backtrack to the earlier successfully executed semantic operation event. In each backtracking level, the event at that level and every semantic operation event between that level and the target operation event are re-executed in sequence, until the target operation event is successfully executed or the preset maximum backtracking level is reached.

7. The method for recording software operation flow according to any one of claims 1 to 3, characterized in that, The step of obtaining the interface element corresponding to the operation behavior information further includes: When the interface element is acquired, it is highlighted on the operable interface of the target software to indicate that the interface element has been successfully recognized.

8. A software operation process recording system, characterized in that, include: The process recording module is used to continuously acquire operational behavior information generated during the operation of the target software in response to the recording start signal; The process recording module is also used to obtain the interface element corresponding to the operation behavior information, trace the parent element of the interface element upwards to the top level, and record the tracing path as the first operation path. The process recording module is also used to respond to the recording end signal, convert the operation behavior information into semantic operation events according to preset rules, and determine a second operation path for each semantic operation event, wherein the second operation path is consistent with the first operation path of the operation behavior information corresponding to the semantic operation event; The process generation module extracts the common path between all the second operation paths, and uses the common path as the root node to construct the sub-level nodes of the root node according to the remaining path of each second operation path. The process generation module is also used to fill the semantic operation events into the corresponding sub-level nodes to obtain the operation process.

9. An electronic device, characterized in that, The method includes a memory and a processor, the memory being used to store at least one program, and the processor being used to load the at least one program to execute the recording method of the software operation flow according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the recording method of the software operation flow as described in any one of claims 1 to 7.

Images