Data processing method, system and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method, system and device, electronic equipment, a storage medium and a program product. In the scheme provided by the embodiment of the invention, on the basis of operation behavior data of a first user on a target application system and interface state change data of the target application system, which are acquired in parallel, interface frame interception trigger control on the target application system is realized, so that a multi-frame interface image of the target application system is obtained. And a more effective training resource is generated based on the multi-frame interface image obtained through frame interception, so that the training resource can be utilized to efficiently train the second user to execute the corresponding work task in the target application system.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data processing method, system, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In the process of promoting and applying various information systems, training relevant personnel to master the operation procedures of the target application system (such as a data annotation system) to complete specific tasks has become a key link in ensuring the efficient execution of tasks. Therefore, high-quality training resources, as a prerequisite and foundation for training, directly affect the effectiveness and efficiency of training. However, the current generation of training resources heavily relies on manual intervention. Specifically, it often involves manually capturing multiple interface images of the target application system during its operation, and then manually adding visual guidance information (such as annotation boxes, text descriptions, and arrow indicators) to each of these images. Furthermore, the quality of the generated training resources cannot adequately meet training needs.

[0003] Therefore, there is an urgent need to provide a technical solution that can automate the generation of training resources, and this technical solution needs to ensure that the generated training resources can meet the needs of high-quality training. Summary of the Invention

[0004] Several embodiments in this specification provide a data processing method, system, apparatus, electronic device, storage medium, and program product that enables the automatic generation of training resources. Among them, In a first embodiment, this specification provides a data processing method. The method includes: Collect interaction data between the first user and the target application system; wherein, the interaction data includes the first user's operation behavior data on the target application system and the interface state change data of the target application system; Based on the interaction data, the interface of the target application system is captured to obtain multiple frames of the interface image of the target application system. Based on the multi-frame interface images, training resources are generated; these training resources are used to train a second user to perform corresponding tasks in the target application system. In a second embodiment, this specification also provides a data processing system. The system includes: The client runs the target application system and is used to collect interaction data between the first user and the target application system, and transmit the interaction data to the server; the interaction data includes the first user's operation behavior data on the target application system and the interface state change data of the target application system. The server is used to control the client to capture frames of the target application system interface based on the interaction data, thereby obtaining multi-frame interface images of the target application system; and to generate training resources based on the multi-frame interface images; the training resources are used to train a second user to perform relevant work tasks in the target application system.

[0005] In a third embodiment, this specification provides a data processing device. The device includes: The data acquisition module is used to collect interaction data between the first user and the target application system; the interaction data includes the first user's operation behavior data on the target application system and the interface state change data of the target application system. The triggering module is used to trigger the capture of the interface frames of the target application system based on the interaction data, so as to obtain multi-frame interface images of the target application system. The generation module is used to generate training resources based on the multi-frame interface images; the training resources are used to train a second user to perform relevant work tasks in the target application system.

[0006] In a fourth embodiment, this specification provides an electronic device including a memory and a processor, wherein the memory stores executable program instructions, and when the processor executes the program instructions, it implements the method provided in the first embodiment.

[0007] In a fifth embodiment, this specification provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the method provided in the first embodiment.

[0008] In a sixth embodiment, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method provided in the first embodiment.

[0009] The solutions provided in the above embodiments of this specification are based on the parallel acquisition of first user operation behavior data and target application system interface state change data to achieve frame capture trigger control of the target application system interface, thereby obtaining multi-frame interface images of the target application system. This solution uses multimodal data (operation behavior data and interface state change data) to achieve frame capture control of the target application system interface, which can effectively improve the accuracy of frame capture triggering, avoid redundant frame capture, and utilize the subsequent generation of more effective training resources based on the multi-frame interface images obtained through frame capture. This training resource can then be used to efficiently train a second user to perform corresponding work tasks in the target application system. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely examples of the various embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. In the drawings: Figure 1 A schematic diagram of the technical architecture on which the method implementation in this specification is based, provided for exemplary embodiments of this specification; Figure 2 A schematic diagram illustrating the process of triggering interface frame capture control of a target application system, provided as an exemplary embodiment of this specification; Figure 3 A schematic diagram illustrating the process of generating training resources provided for exemplary embodiments of this specification; Figure 4 This is a schematic diagram of the structure of a data processing system provided for exemplary embodiments in this specification; Figure 5 A flowchart illustrating a data processing method provided as an exemplary embodiment of this specification; Figure 6 A schematic diagram of the structure of a data processing apparatus provided for exemplary embodiments in this specification; Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0011] In scenarios where training personnel to master the operation procedures of a specific application system to complete a particular task, the generation of training resources is a crucial prerequisite for effective training. For example, in training data labelers to master the operation procedures of a data labeling system to complete data labeling tasks, the generation of training resources currently relies heavily on manual intervention. Specifically, this often involves manually capturing frames during the operation of the data labeling system to obtain multiple interface images of the system in different operating states, which are then used to generate training resources. However, because data labeling systems generally employ dynamic loading and partial refresh interactive designs, their interfaces exhibit highly dynamic and transient characteristics, posing a significant challenge to accurately capturing the key interface states of the data labeling training system through manual frame capture. Specifically, during the operation of a data annotation system, key interface states, such as pop-up prompts, process step jumps, and changes in the enabled / disabled status of controls, are often transient, conditionally triggered, and visually abrupt. Furthermore, their short duration and uncertain timing make it difficult to reliably capture the interface images corresponding to core operation nodes in the complete data annotation operation chain at precise moments, especially when relying on manual observation and frame capture. This is because delays or omissions can easily lead to missing interface images corresponding to core operation nodes, causing gaps in the logical chain of the generated training resources. Moreover, in cross-platform scenarios, such as operating the data annotation system on different platforms like web and mobile for manual frame capture, differences in interface layout, element size, and rendering order within the same data annotation system, as well as the inconsistent order of asynchronously loaded content, further increase the cost of filtering, aligning, and reordering the interface images obtained through manual frame capture. This makes it even more likely that the generated training resources will have gaps in the logical chain and / or inconsistent visual information, severely impacting the reusability and teaching effectiveness of the training resources.

[0012] Furthermore, when generating training resources based on acquired interface images, the commonly used generation schemes can be mainly divided into three categories: static annotation schemes based on screen capture, intelligent prompting systems based on rule engines, and automated annotation schemes based on artificial intelligence (AI) element recognition. Although these schemes have achieved digital generation and presentation of training resources to a certain extent, they still have significant shortcomings in meeting the demand for high-quality training resource generation in modern dynamic interface environments. A detailed analysis follows: 1) The technical principle of the static annotation scheme based on screen capture is as follows: Manual annotation boxes, arrows, and text descriptions are added to the obtained interface image (for example, a highlighted rectangle is drawn around a certain interface element (i.e., an annotation box), supplemented with arrows and text descriptions to highlight the functional attributes and operation instructions of the control), ultimately integrated into a linearly playing graphic tutorial or video courseware. This training resource generation scheme suffers from high levels of manual intervention, lack of context awareness, and poor cross-platform compatibility. Specifically, the lack of context awareness includes the inability to recognize changes in interface state, such as redirecting to an error page after a form submission failure. This results in training resources only covering the ideal tutorial path and failing to handle abnormal scenarios. Poor cross-platform compatibility means that training resources generated on a computer (PC) are prone to errors in added content (such as annotation boxes and arrows) and scale distortion when displayed on a mobile device, affecting reusability.

[0013] 2) The technical principle of the rule-based intelligent prompt system is: by using a predefined operation path rule base (such as "click button A → enter field B → submit form C"), it matches the corresponding rules to the interface images and generates corresponding operation prompts or guidance overlays. This training resource generation solution suffers from problems such as high rule maintenance costs, poor dynamic adaptability, and a lack of real-time risk warning mechanisms. High rule maintenance costs, for example, require manual updates to the rule base when the interface layout or control IDs change. Poor dynamic usability means that it only supports preset linear processes and cannot handle conditional branch paths (such as deciding the next operation based on API return results). The lack of a real-time risk warning mechanism means that it cannot detect user actions that deviate from the standard process (such as skipping required fields or accidentally touching dangerous buttons), and lacks the ability to proactively correct errors or provide reminders.

[0014] 3) The technical principle of the automated annotation scheme based on artificial intelligence (AI) element recognition is to use computer vision models to detect interface elements in the interface image, identify interface elements such as buttons, input boxes, and drop-down menus, and generate basic operation annotations (such as clicking the "Submit" control). This training resource generation scheme suffers from problems such as missing semantic association, insufficient multimodal capabilities, and contextual disconnect. Missing semantic association means that it can only recognize the visual type of interface elements, but cannot understand their specific functional meaning (e.g., simply labeling the "Delete Account" button as a "red rectangle"), lacking contextual explanation. Insufficient multimodal fusion capabilities mean that it does not effectively combine OCR text recognition (optical character recognition), icon semantic analysis, and interface structure information (such as the interface's DOM tree (Document Object Model tree)), resulting in recognition accuracy limited to a single modality. Contextual disconnect means that the generated operation annotation information may be irrelevant to the current operation process state, failing to indicate prerequisites (e.g., "Authentication must be completed before execution") and / or subsequent effects (e.g., "This operation is irreversible").

[0015] To address the aforementioned issues encountered during the generation of training resources, the embodiments in this specification provide a solution. The basic idea of ​​this solution is as follows: First, dynamic frame capture is driven by multimodal perception to capture multiple interface images of the target application system. For example, during the operation of the data annotation system by the training designer, the system integrates the training designer's operational behavior data and the visual changes in the data annotation system's interface to determine whether the frame capture triggering conditions are met, and triggers frame capture when the conditions are met. Then, multimodal interface element recognition technology is used to identify interface elements in the interface images and add relevant visual guidance teaching information to the interface elements. For example, text extraction, icon matching, and semantic fuzzy matching can be combined to identify interface elements, and relevant annotation boxes, arrow guides, and text descriptions can be automatically added to the interface elements.

[0016] Multimodal perception refers to the precise identification and understanding of the correlation between user intent and system response through behavior-visual fusion analysis. This analysis combines collected user behavior data (such as click events) with visual change information of the target application system's interface (such as pixel differences and motion vectors) during a user's (e.g., a training designer's) operation of the target application system. Motion vectors, included in the visual change information, refer to the directional displacement characteristics exhibited by interface elements when their position or shape changes between consecutive frames. This displacement includes not only the magnitude but also the direction, describing the visual evolution of dynamic content. For example, when a drop-down menu expands, the menu items exhibit a downward motion vector; when a pop-up slides in from the bottom, it shows an upward motion trend, and so on.

[0017] Furthermore, multimodal interface element recognition, also known as multimodal UI element recognition, refers to the technical process of determining the functional attributes of interface elements by combining OCR text recognition, CV icon matching, and semantic fuzzy matching.

[0018] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0019] It should be noted that, for ease of description, the accompanying drawings only show the parts related to the relevant technical solutions. Unless otherwise specified, the embodiments and features described in this specification can be combined with each other. Furthermore, the terms "first," "second," and "third" used in the embodiments of this specification are for informational purposes only and do not constitute any limitation. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the stated elements is not excluded. Furthermore, in this specification, unless explicitly stated otherwise, "receiving and transmitting data" does not necessarily mean direct receiving and transmitting; it can be indirect receiving and transmitting. For example, when A receives data sent by B, it can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, when B sends data to A, it can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0020] Furthermore, it should be noted that specific terms are used in this specification to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples, without contradiction. Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible order of execution among many steps, and does not represent the only possible order. Therefore, when the claims involve method steps, adjustments to the order of such steps, or parallel execution between steps, are also within the scope of protection of the claims.

[0021] Furthermore, it should be noted that the user data obtained in this manual is authorized by the user and does not involve user privacy.

[0022] The technical solutions provided in the embodiments described below are based on Figure 1 The technical architecture implementation is shown. For example... Figure 1 As shown, the technical architecture mainly consists of two parts: a frame capture and combination layer 10 and a training resource generation layer 20. The frame capture and combination layer 10 is used to drive dynamic frame capture of the target application system through multimodal perception, capturing multiple frames of the target application system's interface images for subsequent training resource generation. Each frame of the multi-frame interface images records the interface running state of the target application system at a specific moment. The training resource generation layer 20 is used to perform interface element recognition and analysis on the multi-frame interface images to automatically generate training resources based on the interface element recognition and analysis results.

[0023] In this specification, the target application system refers to a software system whose operation requires training for relevant personnel. It has a graphical user interface and supports users in completing specific tasks through clicking, inputting, dragging, etc. For example, the target application system can be, but is not limited to, a data annotation system, a customer relationship management system (CRM), an office automation platform, and various application software.

[0024] To facilitate understanding, the above-mentioned frame combination layer 10 and training resource generation layer 20 will be described in detail below.

[0025] 1. Frame Combination Layer 10 See also Figure 1 As shown, the frame capture and combination layer 10 mainly includes three processing layers: a data acquisition layer 11, an intelligent decision-making layer 12, and an output optimization layer 13. The functions of these three processing layers are as follows: 1.1 Data Acquisition Layer 11 The data acquisition layer 11 includes a user behavior capture module 111 and an interface visual data acquisition module 112. The functions of the user behavior capture module 111 and the interface visual data acquisition module 112 are as follows: 1) User behavior capture module 111 is used to capture operation behavior data generated by the interaction between the user (such as a training designer, referred to as the first user in the following method embodiment) and the target application system interface during the operation of the target application system. The operation behavior data includes, but is not limited to, mouse input events, keyboard input events, and control focus switching events.

[0026] The aforementioned mouse input events refer to the low-level operation signals generated during user interaction with the target application system interface via a mouse device / touchpad, etc., including but not limited to: the interface coordinates of the mouse click (recording the screen coordinates of the mouse pointer at the time of the click), the click time, the click type (such as single click / double click / right click), the mouse movement trajectory, and the scrolling speed. Mouse scrolling speed refers to the distance or percentage of interface content displacement per unit time when the user scrolls the interface vertically or horizontally using the mouse wheel or touchpad, and can be used to reflect the browsing rhythm of user interface information. For example, the mouse trajectory can be processed by quadratic Bézier curve fitting to generate a smoothed and denoised mouse trajectory for subsequent spatiotemporal operation heatmap construction.

[0027] Specifically, the spatiotemporal operation heatmap is a three-dimensional data matrix (including the location coordinates of the operation (X coordinate, Y coordinate) and timestamp) used to quantify and record the distribution and duration of user operations in screen space. In this specification, the spatiotemporal operation heatmap is generated based on mouse input events (including interface coordinates of mouse clicks, mouse movement trajectories, and scrolling speeds) generated by the user during operation of the target application system, and can be used to quantify the focused interface areas (i.e., areas of active interactive operation) of user operations on the interface. That is, the spatiotemporal operation heatmap can not only show "which areas on the target application system's interface are frequently operated," but also "when the user is actively operating and when they are silent."

[0028] Furthermore, the aforementioned keyboard input events include keyboard key sequences (also known as halved key sequences) and content entered by the user on the target application system interface via the keyboard. The keyboard key sequence, also known as the keyboard input sequence, refers to an ordered set of key presses made by the user, arranged chronologically, recording the complete process of another user interacting with the target application system via the keyboard.

[0029] The aforementioned control focus switching event refers to the operation behavior in the target application system's interface where the user's input focus moves from one interface element (such as a text box, button, drop-down menu, etc.) to another interface element. For example, after a user clicks an input box, that input box gains focus, and the cursor blinks within the input box; furthermore, after pressing the Tab key, the focus jumps to the next interactive control (such as a button).

[0030] In this specification, the keyboard input events and control focus switching events mentioned above are high-order user behavior events captured. In practice, the capture of high-order user behavior events can be achieved through a system-level hook mechanism, and these events can be used to quantify the density of user operation intentions. Based on the density of user operation intentions, for example, the complexity and focus of user operations can be determined. The system-level hook mechanism is an event interception and detection technology implemented at the system level. Specifically, it involves inserting custom callback functions into the event propagation path through system-provided programming interfaces or kernel support to intercept or detect corresponding human-computer interaction events or system calls. In this specification, a system-level hook mechanism is used to capture high-order user behavior events such as keyboard input sequences and control focus switching, achieving non-intrusive, low-latency perception of user operation behavior.

[0031] 2) The interface visual data acquisition module 112 is used to capture the interface frame sequence of the target application system at a high frame rate, and can also monitor the rendering layer to collect interface state change data (also known as interface visual change data) of the target application system. The interface state change data may be triggered by user operations on the target application system, or it may be indirectly caused by the autonomous behavior of the target application system (such as timed refresh, background tasks, etc.).

[0032] The aforementioned "high frame rate capture" refers to capturing the interface frame sequence of the target application system during operation at a frequency no lower than a preset capture frequency (e.g., 60fps). During high frame rate capture, HDR (High Dynamic Range) content capture is supported. Supporting HDR content capture means being able to capture image or video content with high dynamic range characteristics. Specifically, this means capturing a wider range of brightness information, allowing for richer detail and color representation from the darkest shadows to the brightest highlights.

[0033] The aforementioned "rendering layer monitoring" captures Canvas / WebGL drawing commands through graphics processing unit (GPU) accelerated graphics programming interfaces (such as DirectX / OpenGL interfaces), thereby enabling the collection of interface state change data for the target application system. Capturing Canvas / WebGL drawing commands specifically refers to obtaining the drawing commands executed on the HTML5 `<canvas>` element. `<canvas>` is a bitmap canvas provided by HTML5. HTML5 is a hypertext markup language used to build and render web page interface content.

[0034] In this manual, the reason for using the aforementioned "rendering layer monitoring" to collect interface state change data of the target application system includes the following: Some interface content rendering updates are not achieved by modifying the interface structure (i.e., the DOM tree (Document Object Model Tree)). Therefore, DOM tree changes are not triggered, and cannot be captured by simply listening for DOM tree changes. This is why the "rendering monitoring" method provided in this manual is needed to capture these changes. For example, if some interface content uses non-DOM tree methods such as "Canvas / WebGL rendering" or "CSS animation" for rendering updates, the interface state changes resulting from such rendering updates cannot be captured by listening for DOM tree changes. Specifically, Canvas / WebGL rendering generates graphics entirely within the <canvas> canvas through corresponding drawing instructions; the DOM tree itself does not change. CSS animations are typically accelerated by the GPU, and the browser implements them by directly manipulating layers, also without triggering DOM tree changes.

[0035] Of course, in other instances, while capturing changes in the interface state of the target application system through the aforementioned "rendering layer monitoring" method, the interface visual data acquisition module 112 can also capture changes in the interface state of the target application system by listening to DOM tree changes. By adopting this dual-track capture method, comprehensive monitoring of interface rendering updates can be achieved, effectively capturing interface state change data more comprehensively and ensuring that key interface state changes can be captured completely. This avoids the limitations of capturing changes in interface data by simply listening to DOM tree changes.

[0036] By utilizing the user's operational behavior data on the target application system and the interface state change data generated by the target application system in response to the user's operation, which are collected through the data acquisition layer 11, intelligent control of the interface frame capture of the target application system can be realized in the subsequent process. This can effectively improve the accuracy of frame capture triggering and avoid redundant frame capture.

[0037] 1.2 Intelligent Decision-Making Layer 12 The intelligent decision-making layer 12 includes a behavior-visual fusion analysis module 121 and a dynamic frame capture triggering engine 122. The functions of the behavior-visual fusion analysis module 121 and the dynamic frame capture triggering engine 122 are as follows: 1) Behavior-Visual Fusion Analysis Module 121 The inputs to the behavior-visual fusion analysis module 121 include: user operation behavior data of the target application system collected by the data acquisition layer 11, and interface state change data of the target application system. Based on these inputs, the behavior-visual fusion analysis module 121 will trigger the execution of behavior-visual fusion analysis, realizing the spatiotemporal alignment of the distribution characteristics of user operation behavior in the target application system (such as operation heatmap) with the visual change area of ​​the target application system interface. By analyzing the overlap and temporal sequence relationship between the two in spatial location, it verifies whether the interface update is caused by effective user operation, thereby eliminating false triggers due to non-causal association (such as background automatic refresh, scheduled task pop-ups, etc.).

[0038] In specific implementation, the behavior-visual fusion analysis module 121 performs functions including spatiotemporal alignment and constructing a feature correlation matrix between behavioral features and visual features. a. Spatiotemporal alignment Spatiotemporal alignment, also known as cross-modal feature alignment, aims to eliminate the difference in acquisition latency between operation behavior data and interface state change data by using timestamp synchronization technology. It establishes a millisecond-level precision mapping relationship between operation events and interface responses, that is, to establish a millisecond-level precision mapping relationship between "cause" (user's operation events on the target application system) and "effect" (interface state changes of the target application system) to eliminate timing misalignment caused by asynchronous loading.

[0039] Based on this established mapping relationship between operation events and interface responses, it is possible to accurately distinguish between interface state updates triggered by valid operations and irrelevant interference, significantly reducing the proportion of subsequent low-redundancy interface frames and ensuring that the frame capture results closely match actual teaching needs. For example, when an interface state change is detected, based on the mapping relationship between operation time and the desired interface, it can be determined whether the interface state change was triggered by the user's current operation. If not, the interface state change can be determined to be an invalid change, and no frame capture is required.

[0040] This specification does not specify the technical means for achieving the above-mentioned "spatiotemporal alignment". For example, a pre-trained spatiotemporal alignment model can be used.

[0041] b. Construct a feature association matrix between behavioral features and visual features. Here, by analyzing the collected user operation behavior data of the target application system, multidimensional user behavior characteristics can be determined. These multidimensional behavior characteristics include, but are not limited to: click frequency (e.g., the number of clicks performed by the mouse per unit time), click density (e.g., the number of clicks per unit interface area, reflecting click concentration), scrolling acceleration (the rate of change of scrolling speed when the interface is scrolled horizontally or vertically using the mouse), movement speed (e.g., the average speed of the mouse pointer movement), and input latency (e.g., the time difference between user operation and the target application system's response). Furthermore, by analyzing the collected interface state change data of the target application system, multidimensional interface visual characteristics can be determined. These multidimensional interface visual characteristics include, but are not limited to: SSIM value (structural similarity index, used to measure the structural similarity between two interfaces), optical flow vector (the direction and speed of movement of each pixel or pixel region in a series of consecutive interface images), inter-frame difference pixel ratio (the proportion of pixels that change between adjacent frames out of the total number of pixels), and so on. Further, based on the multidimensional behavior characteristics and multidimensional interface visual characteristics determined above, a corresponding feature correlation matrix can be constructed. In this feature association matrix, the value at each position is the association weight, which is used to characterize the degree of association between the corresponding behavioral features and the interface visual features.

[0042] 2) Dynamic frame capture trigger engine 122 Here, the dynamic frame capture trigger engine 122 is an intelligent decision-making module based on a reinforcement learning model. It can analyze the operational context in real time and dynamically adjust the frame capture trigger conditions (including frame capture frequency, etc.) to achieve intelligent frame capture control. The operational context includes, but is not limited to, the frequency of user interaction with the target application system's interface, the complexity of the target application system's interface, and the current network status of the terminal device running the target application system.

[0043] In practical implementation, the dynamic frame capture triggering engine 122 can be, for example, an intelligent decision-making module constructed using a Deep Q-Network (DQN) model, used to dynamically decide whether to trigger frame capture. Specifically, the dynamic frame capture triggering engine 122 achieves intelligent frame capture control through the following mechanism: First, state space modeling. Here, "state space modeling" refers to quantifying the frame-triggered decision parameters into a representation that can be understood and processed by reinforcement learning models (such as DQN). In essence, the frame-triggered decision parameters refer to the sum of all features that influence the frame-triggered decision (i.e., determining whether the frame-triggered triggering condition is met). In this specification, the frame-triggered decision parameters used by the dynamic frame-triggered triggering engine 122 when determining whether to trigger a frame include, but are not limited to: the aforementioned mapping relationship between operation events and interface responses, the feature association matrix, the density of user operations, the complexity of interface elements, network status (such as current network transmission latency), and user historical behavior parameters (such as the user's historical false trigger rate).

[0044] When quantifying the complexity of interface elements, it can be done, but is not limited to, using at least one of the following metrics: interface element density, color contrast (such as the height difference between interface elements and their background, which affects the recognizability of interface elements), and the proportion of dynamic interface elements. The quantified results of interface complexity can be used to dynamically adjust the frame capture frequency (i.e., frame capture sensitivity). For example, when the complexity of interface elements is high, the frame capture frequency can be increased accordingly; when the complexity of interface elements is low, the frame capture frequency can be decreased accordingly.

[0045] Next, the action space is defined. Here, "action space definition" refers to matching the corresponding frame-capturing decision from the frame-capturing decision rule base based on the quantized frame-capturing decision criteria parameters to determine whether to trigger a frame capture and / or when to trigger it. For example, the frame-capturing decision rule base may include, but is not limited to, the following pre-set frame-capturing decisions: execute the frame-capturing operation immediately, delay the frame capture by a specified time (which can be used to capture instantaneous interface states), and ignore the current changes and do not trigger a frame capture (to avoid triggering invalid frame captures).

[0046] Finally, a reward mechanism is used to optimize the network parameters of the reinforcement learning model (such as DQN). Here, the teaching effectiveness of subsequently generated training resources is used as a feedback signal to continuously optimize the network parameters of the reinforcement learning model. The training resources are generated using multiple interface images of the target application system obtained by capturing frames. The effectiveness of the training resources includes, for example, whether the interface images contained in the training resources are marked as core nodes.

[0047] In addition to the aforementioned reinforcement learning approach, the dynamic frame capture triggering engine 122 can also employ other methods to achieve intelligent frame capture control, such as a multimodal conditional judgment decision system. Here, "multimodal conditional judgment system" refers to establishing hierarchical judgment logic to determine whether to trigger frame capture, thus ensuring that frame capture triggering is both accurate and robust.

[0048] The above-mentioned layered determination logic includes the following frame-triggered mechanisms: A. Main Trigger Mechanism In this main triggering mechanism, a frame capture triggering command is generated to achieve automatic frame capture when the following three conditions are met simultaneously: spatial clustering matching, significant interface change amplitude, and effective elimination of interference.

[0049] The aforementioned spatial focus matching refers to the spatial matching degree between the user's current focus interface area (which can be simply referred to as the focus area) and the visual change area of ​​the interface reaching a preset matching standard.

[0050] The user's current focus area can be determined based on the aforementioned spatiotemporal operation heatmap. For example, this focus area could be determined by the area of ​​the mouse click (centered on the click point, with a circular or matrix area extending outwards from this center point with a preset radius) or the area of ​​mouse pointer movement on the interface (the bounding box of the continuous movement trajectory of the mouse pointer over a period of time, such as recording all points the mouse pointer moves over a period and taking the smallest bounding rectangle of these points). Visually changing areas can be determined by detecting pixel differences or optical flow. Furthermore, the spatial matching degree between the user's current focus area and the visually changing areas can be calculated, but is not limited to, by calculating the overlap of regions. For example, if a user clicks a button and the button simultaneously changes color, the user's focus area and the visually changing areas can be considered to have a high degree of overlap, thus determining that the matching degree meets the preset matching standard.

[0051] By using the spatial focus matching described above, it can be determined that subsequent frame captures will capture the area that the user is truly interested in, thus avoiding triggering frame captures when changes occur in irrelevant areas.

[0052] The aforementioned significant interface change refers to an interface change exceeding a dynamically adjusted perception threshold. This interface change can be measured, but is not limited to, using methods such as: optical flow amplitude (the average or maximum length of the optical flow vector within the interface change area), pixel difference area (the number or percentage of pixels that have changed compared to the previous frame), and SSIM decrease value. Furthermore, the perception threshold is not a fixed value but can be dynamically determined based on the device performance of the terminal device running the target application system (such as GPU load, memory usage, and CPU usage), device network status, and the user's historical accidental touch rate. For example, if the device network latency is high and / or the device performance is low, the perception threshold can be increased.

[0053] By dynamically adjusting the perception threshold, it is possible to maintain good frame capture accuracy in different environmental scenarios.

[0054] Effective interference elimination refers to eliminating visual interference caused by rapid interface scrolling. For example, when the interface scrolling speed exceeds a preset scrolling threshold and the interface state changes into a continuous strip-like movement, it can be determined that the interface is in a rapid scrolling state, and frame capture is not triggered.

[0055] B. Forced Triggering Mechanism In this forced triggering mechanism, a bypass frame capture triggering channel independent of the main triggering mechanism is established for predefined key teaching nodes, such as the occurrence of predefined high-value events such as pop-up windows, interface jumps, button highlighting, and successful submission. When a predefined high-value event is detected, the frame capture command is directly generated and executed by bypassing the aforementioned trigger frame capture based on reinforcement learning or the main triggering mechanism, so as to capture the corresponding interface image of the target application system in a timely manner.

[0056] C. Suppression logic mechanism In this suppression logic mechanism, the frame capture strategy (such as frame capture frequency) is dynamically adjusted according to the user's operation activity. Specifically, it includes: suppression during high-frequency operation phases and enhancement during idle states.

[0057] The aforementioned high-frequency operation phase suppression refers to automatically relaxing the frame capture time interval limit (i.e., reducing the frame capture frequency) during periods of continuous high-frequency interaction between the user and the target application system interface (such as continuous form filling) to reduce repeated captures of similar interface states. Furthermore, the aforementioned idle state enhancement refers to enhancing detection sensitivity (i.e., increasing the frequency of detecting interface changes to improve the frame capture frequency) during periods of inactivity in the target application system (i.e., when the user has not actively interacted with the target application system for a period of time), ensuring that unexpected but important interface changes can be captured, such as pop-up notifications like "Away Status" or "Connection Interrupted."

[0058] 1.3 Output Optimization Layer 13 This output optimization layer 13 includes an interface image filtering module 131 and a normalized output module 132. The functions of the interface image filtering module 131 and the normalized output module 132 are as follows: 1) Interface Image Filtering Module 131 The interface image filtering module 131 is used to filter the multi-frame interface images obtained by capturing frames of the target application system to select interface images with high efficiency. The reason for filtering the multi-frame interface images obtained by capturing frames is that the previously obtained multi-frame interface images lack semantic understanding of the interface and lack judgment on the interface's educational value, which may result in the capture of too many redundant interface images. Therefore, the interface image filtering module 131 is needed to perform a secondary filtering of the initially captured multi-frame interface images.

[0059] In practice, the following filtering rules can be used, but are not limited to, to automatically identify and remove high-value, highly redundant frame interface images: a. Short animation filtering rules: For example, remove transition animation frames with a duration of less than 300ms (such as loading a rotating icon).

[0060] b. Blurred frame filtering rules: For example, remove motion-blurred frame interface images caused by rapid scrolling inertia. The motion-blurred frame interface images can be determined based on the sharpness detection algorithm of the Laplacian operator.

[0061] c. High repetition frame filtering rules: For example, interface images with a visual repetition rate of more than 90% can be removed. Among them, interface images with a high repetition rate can be compared using a perceptual hash algorithm.

[0062] d. User-focused region filtering rules: For example, removing interface images that do not contain user-focused regions. In practice, a pre-trained attention weight model can be used, but is not limited to, taking the captured interface image and the corresponding user action data as input. The internal attention mechanism automatically calculates the attention weight of each region in the interface image to predict whether the image contains a user-focused region. The attention weight model can, for example, be built based on a ResNet-50 + Transformer network. The attention weights can be calculated using the following formula: In the above formula, This represents the attention weight. This represents an activation function, such as the sigmoid function. FFN can represent a feedforward neural network, which is a labeled component in a Transformer network. Its input is the sum of two embedding vectors, where one embedding vector E... 视觉 It is determined based on the visual features of the interface image, another embedding vector E. 行为 It is determined based on the corresponding user operation data.

[0063] e. Instructional Value Filtering Rules: For example, a cognitive psychology model can be used to evaluate the instructional effectiveness of multiple interface images, and to remove interface images with low instructional value based on the evaluation results. The cognitive psychology model can be a Transformer architecture (a deep learning neural network architecture) pre-trained with eye-tracking data.

[0064] f. Key Operation Node Filtering Rules: For example, by analyzing whether the interface image is triggered by a user operation at a key node (i.e., whether the interface image corresponds to a key operation interface), interface images triggered by user operations at non-key nodes can be eliminated. Specifically, key nodes that must be operated during the application system process can be determined based on, but not limited to, user historical operation behavior data and / or task characteristics (such as task type, task objective, etc.). For example, in the "data annotation" task, "clicking submit for annotation results" is a key node, and / or high-frequency operations determined based on user historical operation behavior data can be considered key nodes.

[0065] 2) Standardized output module 132 The standardized output module 132 is used to process and output the filtered interface images according to a standardized output protocol. The standardized output protocol includes a processing pipeline that incorporates resolution adaptive cropping, non-uniform lighting compensation, and format optimization to ensure smooth processing across multiple scenes. Figure 1 To the point of being responsive.

[0066] For example, the standardized output module 132 can be used to crop the filtered interface image to retain the content that is the focus of the operation. For instance, the interface image can be cropped based on the attention weight of each region in the interface image to retain the core area. Furthermore, the standardized output module 132 can be used to optimize the format of the cropped interface image. Specifically, "format optimization" refers to adaptively selecting a better output format (such as PNG or WebP format) based on the visual content characteristics (such as visual complexity, number of colors, text density, etc.) of the acquired and processed interface image to achieve a balance between interface image clarity and storage cost.

[0067] Based on the above description of the frame combination layer 10, through Figure 2 This diagram illustrates the implementation process of capturing and outputting multiple frames of the target application system's interface images through the frame capture and combination layer 10.

[0068] II. Training Resource Generation Layer 20 The input to the training resource generation layer 20 is the multi-frame interface image of the target application system output by the aforementioned frame combination layer 10. Its function is to automatically add guiding teaching information to the multi-frame interface image through multimodal interface element recognition, and then adaptively generate training resources.

[0069] In practice, the training resource generation layer 20 includes three core modules: an intelligent recognition module for interface elements (UI elements) 21, a dynamic annotation generation module 22, and an adaptive training resource compilation module 23.

[0070] The functions of the three core modules included in the training resource generation layer 20 are described in detail below.

[0071] 2.1 Interface Element Intelligent Recognition Module 21 The interface element intelligent recognition module 21 is based on technologies such as OCR (Optical Character Recognition), CV icon matching, and semantic fuzzy matching to realize the functional recognition of interface elements in the interface image.

[0072] The aforementioned OCR-based interface element function recognition refers to using OCR (Optical Character Recognition) technology to detect and recognize text in interface images, extracting readable text (such as text labels on buttons, menu item names, etc.). This extracted readable text can then be used to automatically infer the function of the interface element containing that text. For example, if an interface element has the text "Submit," its function can be inferred to be submitting data. During the text detection and recognition process, adaptive thresholding techniques can be used to segment interface elements containing text before recognizing the text on those elements. This effectively avoids interference from complex backgrounds in the interface image and improves text recognition accuracy in low-contrast scenes.

[0073] The aforementioned interface element function recognition based on CV icon matching refers to: using computer vision (CV) technology, through algorithms such as scale-inconvenient feature transformation (SIFT), to identify interface elements (such as icons) in graphical or symbolic form that do not contain any text (e.g., icons). The icon ("magnifying glass") is used to perform feature matching (i.e., similarity analysis) with icons in a pre-built icon library to associate corresponding functional semantic tags with the interface element. For example, if the similarity between an interface element U1 presented graphically in the interface image and an icon in the icon library is greater than a preset similarity threshold (e.g., 85%), then the two are considered to be matched successfully, and the functional semantic label associated with the matched icon can be assigned to the interface element U1, that is, the interface element U1 is associated with the corresponding functional semantic tag. The functional semantic tag associated with the interface element can be, for example, "delete" or "save".

[0074] The aforementioned interface element function recognition based on text semantic similarity fuzzy matching refers to the following: For interface elements that cannot be accurately matched by precise text or icon matching, natural language processing techniques (such as the Word2Vec word vector model) are used to calculate the cosine similarity between the text label on the interface element (e.g., "upload") and words in the annotation template library (e.g., "import"). Based on the cosine similarity result, the corresponding words in the annotation template library are associated with the interface element. This allows for the subsequent addition of appropriate guiding teaching information to the interface element based on the word association results. For example, the text label on the interface element can be replaced, or corresponding functional explanations can be added to the interface element. For example, suppose the text label on an interface element U2 in the interface image is "Upload". After performing cosine similarity analysis with words in the annotation template library, it is determined that the similarity between the text label "Upload" and the word "Import" in the annotation template library is greater than a set similarity threshold (e.g., 0.7). Then, in the subsequent stage of adding guiding teaching information to the interface element, the text label "Upload" on the interface element U2 can be replaced with "Import". Alternatively, while adding an annotation box to the interface element U2, a corresponding functional explanation "indicating import function" can be added to the interface element U2 based on the word "Import". The implementation of adding an annotation box to the interface element U2 can be, for example, by drawing a rectangular bounding box around the interface element U2 while retaining the original text label "Upload". By adding an annotation box and functional explanation to the interface element U2 as described above, not only can the consistency of the original design of the interface element U2 be maintained, but additional information can also be provided to the user.

[0075] It should be noted that the annotation template library in this manual contains not only preset vocabulary but also graphics (such as arrows). These graphics can be used to add corresponding graphical annotations to interface elements for guiding instruction. The annotation template library employs a hierarchical design, allowing for the matching of different levels of annotation templates (such as graphical annotation templates like arrows) to interface elements based on their element type.

[0076] In this specification, the interface element intelligent recognition module 21 employs multimodal fusion technology (OCR, CV icon matching, and semantic fuzzy matching, etc.) to realize the functional semantic analysis of interface elements, which can effectively improve the accuracy of determining the function of interface elements and help improve the accuracy of adding subsequent guidance and teaching information.

[0077] It should be noted that after extracting the text content (i.e., readable text) from the interface elements, if the text content is found to contain professional field-specific terms (such as T+1 clearing and credit limit), the corresponding terminology explanation information can be generated by combining the industry knowledge base. This information can then be added to the professional field-specific terms contained in the text content of the interface elements during the subsequent stage of adding teaching information.

[0078] 2.2 Dynamic annotation generation module 22 This dynamic annotation generation module 22 is used to add corresponding guidance and teaching information to interface elements based on the functions of interface elements in the interface image identified by the aforementioned interface element intelligent recognition module 22, and / or the functional semantic tags associated with the interface elements, and / or the corresponding words in the annotation templates associated with the interface elements. This guidance and teaching information includes, but is not limited to, visual guidance elements (such as arrows, annotation boxes, etc.) added to the interface elements, as well as operation instruction text and / or function explanation text, and terminology explanation information added to the professional field-specific terms contained in the text content on the interface elements. Among the added guidance and teaching information, some can be interactive guidance elements (also known as interactive executable annotations, such as arrows, etc.), that is, they can not only be used for operation prompts, but also serve as clickable interactive entry points. When the user clicks on the interactive guidance element, it will trigger the automatic execution of the function operation corresponding to the interface element pointed to by the interactive guidance element, thus realizing an integrated teaching-operation experience of "what you see is what you control".

[0079] For example, a UI element (such as an "Upload" button) can be framed with a green pulse animation box, and a red arrow pointing to it can be added. Above the red arrow, the text "This is the import function; click the arrow to automatically perform the import operation" can be displayed. The green pulse animation box refers to a green rectangular bounding box drawn around a UI element in the UI image, with periodic scaling or brightness changes (simulating a "heartbeat" or "breathing") to achieve a combined purpose of visual focus and operation guidance.

[0080] Furthermore, the dynamic annotation generation module 22 introduces a context variable substitution mechanism. This mechanism allows for the generation of specific functional prompts for interface elements based on the service scenario of the current interface image. For example, assuming a file upload control is detected in the interface image, and the interface title "Import Customer Information" indicates a customer-related service scenario, a sample prompt for uploadable file types (e.g., "Upload Excel customer list") can be automatically generated for the "file upload control."

[0081] 2.3 Adaptive Training Resource Compilation Module 23 The adaptive training resource compilation module 23 is used to: dynamically assemble guided teaching information according to preset training user role tags (such as administrator, ordinary user) before generating training resources, so that the guided teaching information in the training resources presents guided teaching information of different granularities under different user ability modes and / or user role tags. It also performs multiple coordinate layouts of the training resources according to preset device types (such as PC, mobile) to achieve cross-platform adaptation of the training resources.

[0082] In practice, users can be categorized into two groups based on their user profile weights: beginners and experts. Correspondingly, the granularity of the guidance information in the training resources can include two presentation modes: beginner guidance mode and expert guidance mode. In beginner guidance mode, more detailed and complete guidance information can be displayed (e.g., complete operation steps and risk warnings). In expert guidance mode, simplified guidance information can be displayed (e.g., guidance information only for key operation nodes and quick operation entry prompts, omitting explanations of operation functions). For example, taking the file import function guidance scenario as an example, in beginner (ordinary user) guidance mode, the guidance content displayed in the training resources may include: Step 1, Click the [Import] button; Step 2, Select your Excel customer list (.xlsx format), warning: Do not import repeatedly, otherwise data duplication may occur; while in expert (administrator) guidance mode, the guidance content displayed in the training resources may include: Only a green pulse box is displayed next to the [Import] button, and a prompt "Import +" is displayed when hovering. "One-click execution".

[0083] The user profile weights mentioned above are calculated based on multi-dimensional features that characterize user proficiency (such as user role tags, historical operation behavior data, and environmental parameters such as device network status and device performance).

[0084] As can be seen, the training resources generated in this manual can have multiple teaching guidance modes (including the aforementioned novice guidance presentation mode and expert guidance presentation mode). When relevant personnel use the training resources in the future, the teaching guidance mode can be automatically switched according to the role of the relevant personnel, so as to achieve personalized training that is "personalized for each person".

[0085] Furthermore, to achieve cross-platform adaptation of training resources, the coordinate mapping relationship between multiple devices can be calculated based on the relative positions of interface elements in the interface image and the guiding teaching information added to the interface elements. For example, the coordinates (x, y) of interface elements on the PC can be converted into relative proportional coordinates (0.3x, 0.2y) on the mobile device, thereby enabling the training resources to be loaded and displayed using an adapted coordinate layout when running on different devices.

[0086] Furthermore, the adaptive training resource compilation module 23 can also be used to output training resources in multiple formats. For example, outputting in PDF manual format, video format, or interactive tutorial format. In the case of outputting in interactive tutorial format, it supports user interaction within the training resources, such as automatically triggering the corresponding operation function by clicking an arrow added to an interface element.

[0087] Based on the above description of training resource generation layer 20, through Figure 3 A schematic diagram of the implementation process of training resource generation through training resource generation layer 20 is shown.

[0088] The above text combines Figure 1 The technical architecture mentioned is based on a server-side and client-side implementation. See also... Figure 1 As shown, the data acquisition layer 11 in the technical architecture can be deployed on the client side, while the intelligent decision-making layer 12, output optimization layer 13, and training resource generation layer 20 can be deployed on the server side. The server side can be a server, server cluster, virtual server, or cloud, etc. The client side can be, but is not limited to, terminal devices such as smartphones, smart wearable devices, tablets, laptops, and desktop computers.

[0089] thus, Figure 4 A data processing system according to an embodiment of this specification is also shown, the system including a client 100 and a server 200. Wherein, Client 100 runs a target application system and is used to collect interaction data between the first user and the target application system, and transmit the interaction data to the server; the interaction data includes the first user's operation behavior data on the target application system and the interface state change data generated by the target application system in response to the user's operation. Server 200 is used to control the client to capture frames of the target application system based on the interaction data, thereby obtaining multi-frame interface images of the target application system; and to generate training resources based on the multi-frame interface images; the training resources are used to train a second user to perform corresponding work tasks in the target application system.

[0090] For a detailed description of the type of target application system described here, please refer to the relevant content described in other embodiments. Furthermore, the specific implementation of the functions of the server 200 and client 100 will be described in detail in the following method embodiments, and will not be repeated here.

[0091] The technical solutions provided in this specification will be described below by way of method embodiments.

[0092] Figure 5 A flowchart illustrating a data processing method according to an embodiment of this specification is shown, wherein the execution entity of the method is the aforementioned server 200. See also... Figure 5 As shown, the data processing method includes the following steps: 102. Collect interaction data between the first user and the target application system; wherein, the interaction data includes the first... User interaction data and interface state change data of the target application system; 104. Based on the interaction data, trigger the capture of the interface frames of the target application system to obtain multi-frame interface images of the target application system; 106. Based on the multi-frame interface images, generate training resources; the training resources are used to train the second user to perform corresponding work tasks in the target application system.

[0093] In this embodiment, the target application system may be, for example, a data annotation system, an office automation platform, or various application software.

[0094] For details on the specific implementation of collecting "interactive data" in section 102 above, please refer to the other embodiments mentioned above. Figure 1 The details related to "Data Acquisition Layer 11" will not be elaborated here.

[0095] Furthermore, in addition to controlling the capture of the target application system's interface frames based on interaction data, control can also be based on other data, such as the target application system's operating environment data and the first user's historical operation behavior data on the target application system (specifically, such as the first user's historical false trigger rate on the target application system). Based on this, the method provided in this embodiment may also include the following steps: S11. Obtain the current operating environment data of the target application system; wherein, the operating environment data includes, but is not limited to: device performance parameters and network status (such as network latency) of the terminal device running the target application system.

[0096] Furthermore, step 104 above, "based on the interaction data, triggering a frame capture of the target application system's interface," includes: 1042. Based on the interaction data and the runtime environment data, trigger a screenshot of the target application system's interface.

[0097] In practice, step 1042 here can be implemented using at least one of the following specific implementation schemes: In the first scheme, the specific implementation of step 1042 above includes: determining the frame capture decision basis parameters based on the interaction data and the operating environment data; inputting the frame capture decision basis parameters into the pre-trained frame capture decision model and outputting a frame capture control command; wherein, the frame capture control command is generated based on a frame capture decision matched from a preset frame capture decision rule library, and is used to indicate whether to trigger interface frame capture and the corresponding frame capture timing; according to the frame capture control command, triggering interface frame capture of the target application system.

[0098] For a detailed description of the implementation of the first approach, please refer to the other embodiments mentioned above. Figure 1 The description of the dynamic frame capture triggering engine 122 uses reinforcement learning to achieve intelligent frame capture control.

[0099] In the second scheme, the specific implementation of step 1042 above includes: based on the interaction data, determining the focus interface area currently operated by the first user in the target application system, the interface visual change area corresponding to the current interface state change of the target application system, and the interface visual change amplitude; if the overlap between the focus interface area and the interface visual change area is greater than or equal to the overlap threshold, and the interface visual change amplitude is greater than or equal to the change amplitude threshold, then triggering a frame capture of the current interface of the target application system; wherein, the change amplitude threshold is determined by adaptive adjustment based on at least one of the current operating environment data of the target application system and the current interface complexity of the target application system.

[0100] For a detailed description of the implementation of this second solution, please refer to the other embodiments mentioned above. Figure 1 The dynamic frame capture triggering engine 122 described uses a "main triggering mechanism" to implement frame capture control. Here, the "overlap threshold" and "change amplitude threshold" can be understood as the "preset matching standard" and "perception threshold" mentioned in the aforementioned "main triggering mechanism," respectively, only expressed differently in different implementation scenarios.

[0101] In the third approach, the specific implementation of step 1042 above includes: identifying the interface response event currently occurring in the target application system based on the interface state change data; if the interface response event belongs to a preset interface response event type, then triggering a frame capture of the currently displayed interface of the target application system.

[0102] Here, the preset interface response event types include, for example, predefined high-value events such as: pop-up appearance, interface jump, button highlighting, and successful submission.

[0103] For a detailed description of the implementation of this third approach, please refer to the other embodiments mentioned above. Figure 1 The description of the dynamic frame capture triggering engine 122 uses a "forced triggering mechanism" to implement frame capture control.

[0104] In the fourth scheme, the specific implementation of step 1042 above includes: determining the current activity level of the first user on the target application system based on the first user's operation behavior data; adjusting the frame capture frequency based on the operation activity level; and triggering the interface frame capture of the target application system according to the adjusted frame capture frequency.

[0105] For a detailed description of the implementation of this fourth approach, please refer to the other embodiments mentioned above. Figure 1 The description of the dynamic frame capture triggering engine 122 uses a "suppression logic mechanism" to implement frame capture control.

[0106] After acquiring multiple frames of the target application system's interface images through frame capture, these images can be filtered to remove redundant ones. The filtered images are then output in a suitable image format for use in subsequent training resource generation. For a detailed description of the filtering of the multiple frames and the output in a suitable image format, please refer to the aforementioned [link / reference]. Figure 1 The description relates to "Output Optimization Layer 13".

[0107] In the above 106, appropriate guiding teaching information can be added to the interface image based on the function of the interface elements in the interface image, thereby generating training resources.

[0108] Therefore, in one feasible solution, the above-mentioned step 106, "generating training resources based on the multi-frame interface images," may include the following steps: 1062. Perform functional semantic analysis on the interface elements in the multi-frame interface images to determine the functions of the interface elements; 1064. Based on the function of the interface elements, add appropriate guidance and teaching information to the multi-frame interface images; wherein the guidance and teaching information includes interactive guidance elements, and the interactive guidance elements can trigger the execution of the corresponding interface elements when operated. 1066. Based on the multi-frame interface images after adding the guiding teaching information, the training resources are generated.

[0109] In the above 1062, the functional recognition of interface elements can be achieved using multimodal recognition technologies (such as OCR (Optical Character Recognition), CV icon matching, and semantic fuzzy matching).

[0110] Specifically, assuming the target interface image is one of multiple interface images, step 1062 can be implemented in at least one of the following ways: Method 1: Extract the text content (readable text) from the interface elements in the target interface image; determine the function of the interface elements based on the text content; Method 2: Match the graphical interface elements in the target interface image with multiple icons contained in the template library; determine the function of the interface element based on the functional text description associated with the matched icon; Method 3: Match the text content (such as text labels used to characterize the function of the interface elements) on the target interface image with multiple functional terms contained in the template library; determine the function of the interface elements based on the matched functional terms.

[0111] Regarding the "functional analysis and determination of interface elements" in step 1062 above, please refer to the other embodiments mentioned above. Figure 1 The description relates to the "interface element intelligent recognition module 21". The "template library" mentioned here is the same as the "annotation template library" described earlier when introducing the "interface element intelligent recognition module 21".

[0112] In the aforementioned 1064, the added guided instruction information includes, but is not limited to: annotation boxes used to highlight interface elements, guide icons pointing to the interface elements, and operation instruction text and function explanation text associated with the interface elements. Among these, annotation boxes, guide icons, etc., can all be interactive guided elements.

[0113] For details on the specific implementation of adding appropriate guidance and teaching information to the interface image, please refer to the other embodiments mentioned above. Figure 1 The description relates to the content of "Dynamic Annotation Generation Module 22".

[0114] Furthermore, considering the possibility of omissions in frame capture, which could lead to missing intermediate interface states between multiple interface images, this embodiment will also perform intermediate interface state completion when intermediate interface states that should exist but were not captured are inferred from the multi-frame interface images. Therefore, the method provided in this embodiment may further include the following steps: S22. Perform interface state correlation analysis on the multi-frame interface images; S23. When it is determined that the interface state transition between two adjacent interface images is discontinuous, it is determined that there is a missing intermediate interface state between the two interface images. S24. Insert an adapted virtual interface image between the two frame interface images to complete the missing intermediate interface state.

[0115] As described above, the interface state of each frame can be determined by analyzing the image features of multiple interface images. For example, if the image features of interface image P1 are the display of "username" and "password" input boxes, it can be inferred that the interface state of interface image P1 is "login page"; if the image features of interface image P2 are the appearance of the text "Loading..." or a rotating icon, it can be inferred that the interface state of interface image P2 is "loading". Furthermore, multiple sets of adjacent two-frame interface images can be determined based on the timestamps carried by the multiple-frame interface images. When analysis based on the interface states corresponding to each of the multiple-frame interface images reveals that the interface state transition between two adjacent frames is discontinuous, it can be determined that there is a missing intermediate interface state between these two adjacent frames.

[0116] The determination of "the existence of a missing intermediate interface state" can be made, but is not limited to, when any of the following conditions are met: Visual jump: For example, the previous frame of the interface image does not have a "loading prompt", but the next frame of the interface image shows that it has been successful and the operation is "submitted".

[0117] The difference between operation and response time is too large: for example, after the user clicks, the delay in interface change is greater than the preset threshold (such as 800ms).

[0118] Abrupt changes to key elements: For example, the "Submit button" changes directly to "Success message" without any transition.

[0119] If it is determined that there is a missing intermediate interface state between two adjacent interface images, at least one adapted virtual interface image can be inserted between the two interface images, or an interface state transition prompt can be inserted to complete the missing intermediate interface state between the two interface images.

[0120] For example, consider the following scenario: A first user uploads a large file through a target application system. The target application system should display an upload progress bar. However, due to low frame capture frequency, only two interface images, "Start Upload" and "Upload Complete," are captured. By analyzing the interface state of these two images, it can be determined that there is a missing upload progress state between them. In this case, at least one virtual interface image can be generated to form a simulated upload animation. This simulated upload animation displays an upload progress bar progressing from 0% to 100% for 5 seconds, accompanied by text such as "Uploading...". The resulting simulated upload animation can be inserted between the "Start Upload" and "Upload Complete" interface images. This helps relevant personnel understand that uploading is a time-consuming operation that requires patience when training them using the generated training resources.

[0121] And, correspondingly, the aforementioned 1066 "generating the training resources based on the multi-frame interface images after adding guided teaching information" may include: 10662. The training resource is generated based on the multi-frame interface images after adding guiding teaching information and completing the intermediate interface states.

[0122] The generated training resources can take various forms, including but not limited to instructional videos, text and image documents, and simulation systems of the target application system. These resources can be used to train relevant personnel on the operational procedures for using the target application system to perform corresponding work tasks (such as data annotation tasks).

[0123] For specific implementation details of the steps described above in this embodiment, please refer to the relevant content in other embodiments, which will not be repeated here. Furthermore, the method provided in this embodiment may also include some steps disclosed in other embodiments, which can also be referred to the relevant content in other embodiments, and will not be repeated here.

[0124] The above text combined Figure 5 Specific embodiments of the embodiments described herein have been described. It should be noted that other embodiments are within the scope of the appended claims. Furthermore, in some cases, the actions or steps described in the specification may be performed in a different order than those shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0125] The apparatus embodiments corresponding to the method embodiments provided in this specification are described below.

[0126] Figure 6A schematic diagram of the structure of an apparatus provided in an exemplary embodiment of this specification is shown. For example... Figure 6 As shown, the device includes: a data acquisition module 62, a triggering module 64, and a generation module 66. Among them, The data acquisition module 62 is used to acquire interaction data between the first user and the target application system; wherein, the interaction data includes the first user's operation behavior data on the target application system and the interface state change data of the target application system; Trigger module 64 is used to trigger the capture of the interface frames of the target application system based on the interaction data, so as to obtain multi-frame interface images of the target application system. The generation module 66 is used to generate training resources based on the multi-frame interface images; the training resources are used to train a second user to perform relevant work tasks in the target application system.

[0127] Optionally, when generating training resources based on the multi-frame interface images, the generation module 66 is specifically used for: performing functional semantic analysis on the interface elements in the multi-frame interface images to determine the functions of the interface elements; adding appropriate guidance and teaching information to the multi-frame interface images according to the functions of the interface elements; wherein the guidance and teaching information includes interactive guidance elements, and the interactive guidance elements can trigger the execution of the corresponding interface elements when operated; and generating training resources based on the multi-frame interface images after adding the guidance and teaching information.

[0128] Optionally, the device further includes: an analysis module, a determination module, and an insertion module. The analysis module is used to perform interface state correlation analysis on the multi-frame interface images. The determination module is used to determine that there is a missing intermediate interface state between two adjacent interface images when the analysis shows that the interface state transition between two adjacent interface images is discontinuous. The insertion module is used to insert an adapted virtual interface image between the two interface images to complete the missing intermediate interface state.

[0129] Optionally, when generating training resources based on the multi-frame interface images after adding guiding teaching information, the above-mentioned generation module 66 is specifically used to generate the training resources based on the multi-frame interface images after adding guiding teaching information and completing the intermediate interface states.

[0130] Optionally, the target interface image is one of the multiple frame interface images. Furthermore, the generation module 66, when performing functional semantic analysis on the interface elements in the target interface image to determine the function of the interface elements, specifically performs at least one of the following: extracting text content from the interface elements in the target interface image; determining the function of the interface elements based on the text content; matching the graphically presented interface elements in the target interface image with multiple icons contained in a template library; determining the function of the interface elements based on the functional text descriptions associated with the matched icons; matching the text content on the interface elements in the target interface image with multiple functional terms contained in the template library; and determining the function of the interface elements based on the matched functional terms.

[0131] Optionally, the added guided instruction information includes at least one of the following: a highlighting box for emphasizing interface elements, a guide icon pointing to the interface element, and operation instruction text and function explanation text associated with the interface element. The interactive guided element includes the highlighting box and / or the guide icon.

[0132] Optionally, the device further includes: an acquisition module, configured to acquire the current operating environment data of the target application system; the operating environment data includes: device performance parameters and network status of the terminal device running the target application system. Furthermore, the triggering module 64, when used to trigger interface frame capture of the target application system based on the interaction data, is specifically configured to: trigger interface frame capture of the target application system based on the interaction data and the operating environment data.

[0133] Optionally, the triggering module 64, when triggering interface frame capture of the target application system based on the interaction data and the runtime environment data, is specifically configured to: determine frame capture decision basis parameters based on the interaction data and the runtime environment data; input the frame capture decision basis parameters into a pre-trained frame capture decision model, and output a frame capture control command; wherein the frame capture control command is generated based on a frame capture decision matched from a preset frame capture decision rule library, and is used to indicate whether to trigger interface frame capture and the corresponding frame capture timing; trigger interface frame capture of the target application system according to the frame capture control command; and / or determine, based on the interaction data, the focused interface area currently operated by the first user in the target application system, the interface visual change area corresponding to the current interface state change of the target application system, and the interface visual change magnitude; if the focused interface area and the interface... If the overlap of the visually changing area is greater than or equal to an overlap threshold, and the magnitude of the visual change in the interface is greater than or equal to a change magnitude threshold, then a frame capture of the current interface of the target application system is triggered. The change magnitude threshold is determined through adaptive adjustment based on at least one of the target application system's current operating environment data and the current interface complexity. And / or, based on the interface state change data, an interface response event currently occurring in the target application system is identified. If the interface response event belongs to a preset interface response event type, then a frame capture of the currently displayed interface of the target application system is triggered. And / or, based on the first user's operation behavior data, the first user's current operational activity level in the target application system is determined. Based on the operational activity level, the frame capture frequency is adjusted. The target application system's interface is then captured according to the adjusted frame capture frequency.

[0134] It should be noted that the above-mentioned devices can implement the technical solutions described in the corresponding method embodiments. The specific implementation principles of each module or unit can be found in the relevant content of the corresponding method embodiments, and will not be elaborated further here. Furthermore, for ease of description, the above devices are described by function as various modules or units. Of course, when implementing one or more of this specification, the functions of each module or unit can be implemented in one or more software and / or hardware, or a module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0135] Furthermore, embodiments of this specification also provide an electronic device. For example... Figure 7As shown, the electronic device 900 includes a memory 91 and a processor 92.

[0136] The aforementioned memory 91 can be implemented by at least one volatile or non-volatile storage device of any type, or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Furthermore, the memory, wholly or partially, can be integrated with the processor. The memory can contain both removable and non-removable components.

[0137] The processor 92 described above may include one or more general-purpose processors and / or special-purpose processors.

[0138] Furthermore, memory 91 may contain a non-transitory computer-readable medium storing executable program instructions 912 (e.g., compiled or uncompiled program logic and / or machine code). Processor 92 is capable of executing the program instructions 912 stored in memory to implement any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Additionally, execution of program instructions 912 by processor 92 may result in processor using corresponding data 911.

[0139] For example, the program instructions 912 described above may include an operating system 9122 (e.g., an operating system kernel, device drivers, and / or other modules) installed on the electronic device 900, and one or more application programs 9121 (e.g., a browser, social media application, or game application). Similarly, the data 911 described above may include operating system data 9112 and application data 9111. The operating system data 9112 is primarily accessible to the operating system 9122, while the application data 9111 is primarily accessible to one or more application programs 9121. The application data 9111 may reside in a file system visible or hidden from the user of the electronic device 900.

[0140] Application 9121 can communicate with operating system 9122 through one or more application programming interfaces (APIs). These APIs facilitate application 9121 in reading and / or writing application data, transmitting or receiving information via communication components, and receiving or displaying information on the user interface. In some terms, application 9121 may be simply referred to as "app". Furthermore, application 9121 can be downloaded to the electronic device through one or more online application stores or app markets. However, application 9121 can also be installed on electronic device 400 in other ways, such as through a web browser or a physical interface on electronic device 900 (e.g., a USB port). Furthermore, such as Figure 7 As shown, the electronic device also includes other components such as a communication component 93, a display 94, a power supply component 95, an audio component 96, and a user interface 97. Figure 7 The diagram only shows some components and does not mean that the electronic device 900 includes only these components. Figure 7 The components shown. Additionally... Figure 7 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the product form of the electronic device 900. The electronic device 900 in this embodiment can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device; it can also be a server-side device such as a conventional server, cloud server, or server array; or it can be an integrated device combining terminal and server-side devices. If the electronic device 900 in this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 7 The components within the dashed box; if the electronic device 900 in this embodiment is implemented as a server-side device such as a conventional server, cloud server, or server array, then it may not include... Figure 7 The component within the dashed box.

[0141] The aforementioned communication component 93 is configured to facilitate wired or wireless communication between the device housing the communication component and other devices. The device housing the communication component 93 can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component 93 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. Specifically, the communication component 93 includes a communication interface that enables the electronic device 900 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0142] The aforementioned display 94 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0143] The power supply component 95 provides power to various components of the device in which it resides. The power supply component 95 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.

[0144] The aforementioned audio component 96 can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0145] The user interface 97 described above includes receiving user input and providing output to the user. Therefore, the user interface 97 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. The user interface 97 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, the user interface 97 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, the electronic device 900 may support remote access from other devices via a communication interface or another physical interface (not shown). The user interface 97 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. The user interface 97 may also be configured as a display device for rendering or displaying text fragments.

[0146] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Furthermore, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform actions such as... Figure 5 The method described.

[0147] This specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements... Figure 5 The method described.

[0148] Those skilled in the art will recognize that the functions described in the various embodiments disclosed in this specification in one or more of the examples above can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0149] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above descriptions are merely specific implementations of the multiple embodiments disclosed in this specification and are not intended to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the multiple embodiments disclosed in this specification should be included within the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A data processing method, characterized in that, include: Collect interaction data between the first user and the target application system; wherein, the interaction data includes the first user's operation behavior data on the target application system and the interface state change data of the target application system; Based on the interaction data, the interface of the target application system is captured to obtain multiple frames of the interface image of the target application system. Based on the multi-frame interface images, training resources are generated; the training resources are used to train a second user to perform corresponding work tasks in the target application system.

2. The method according to claim 1, characterized in that, Based on the multi-frame interface images, training resources are generated, including: Perform functional semantic analysis on the interface elements in the multi-frame interface images to determine the functions of the interface elements; Based on the functions of the interface elements, appropriate guidance and teaching information is added to the multi-frame interface images; wherein, the guidance and teaching information includes interactive guidance elements, and when the interactive guidance elements are operated, they can trigger the execution of the functions of the corresponding interface elements. The training resources are generated based on the multi-frame interface images after adding guided teaching information.

3. The method according to claim 2, characterized in that, Also includes: Perform interface state correlation analysis on the multi-frame interface images; When it is determined that the interface state transition between two adjacent interface images is discontinuous, it is determined that there is a missing intermediate interface state between the two interface images. A suitable virtual interface image is inserted between the two interface images to fill in the missing intermediate interface state.

4. The method according to claim 3, characterized in that, Based on the multi-frame interface images after adding guided instruction information, the training resources are generated, including: The training resources are generated based on the multi-frame interface images after adding guiding teaching information and completing the intermediate interface states.

5. The method according to any one of claims 2 to 4, characterized in that, The target interface image is one of the multiple frame interface images; Furthermore, functional semantic analysis is performed on the interface elements in the target interface image to determine the functions of the interface elements, including at least one of the following: Extract the text content from the interface elements in the target interface image; determine the function of the interface elements based on the text content; The interface elements in the target interface image, which are presented graphically, are matched with multiple icons contained in the template library; the function of the interface element is determined based on the functional text description associated with the matched icon. The text content on the interface elements in the target interface image is matched with multiple functional terms contained in the template library; the function of the interface element is determined based on the matched functional terms.

6. The method according to any one of claims 2 to 4, characterized in that, The added instructional information includes at least one of the following: a highlighting box for highlighting interface elements, a guide icon pointing to the interface elements, and operation instructions and function explanations associated with the interface elements; The interactive guide element includes the label box and / or the guide icon.

7. The method according to any one of claims 1 to 4, characterized in that, Also includes: Obtain the current operating environment data of the target application system; The operating environment data includes: device performance parameters and network status of the terminal device running the target application system; And, based on the interaction data, triggering a frame capture of the target application system's interface, including: Based on the interaction data and the runtime environment data, a screenshot of the target application system's interface is triggered.

8. The method according to claim 7, characterized in that, Based on the interaction data and the runtime environment data, triggering the capture of interface frames of the target application system includes: Based on the interaction data and the operating environment data, frame capture decision parameters are determined; these parameters are input into a pre-trained frame capture decision model, and a frame capture control command is output; wherein, the frame capture control command is generated based on a frame capture decision matched from a preset frame capture decision rule base, and is used to indicate whether to trigger interface frame capture and the corresponding frame capture timing; according to the frame capture control command, interface frame capture of the target application system is triggered; and / or Based on the interaction data, the focus area of ​​the first user's current operation in the target application system, the visual change area corresponding to the change in the current interface state of the target application system, and the magnitude of the visual change are determined. If the overlap between the focus area and the visual change area is greater than or equal to an overlap threshold, and the magnitude of the visual change is greater than or equal to a magnitude threshold, then a frame capture of the current interface of the target application system is triggered. The magnitude threshold is determined through adaptive adjustment based on at least one of the following: the current operating environment data of the target application system, and the current interface complexity of the target application system. Based on the interface state change data, identify the interface response event currently occurring in the target application system; if the interface response event belongs to a preset interface response event type, trigger a frame capture of the currently displayed interface of the target application system; and / or Based on the first user's operation behavior data, determine the first user's current operation activity level on the target application system; based on the operation activity level, adjust the frame capture frequency; and trigger the capture of interface frames of the target application system according to the adjusted frame capture frequency.

9. A data processing system, characterized in that, include: The client runs the target application system and is used to collect interaction data between the first user and the target application system, and transmit the interaction data to the server; the interaction data includes the first user's operation behavior data on the target application system and the interface state change data of the target application system. The server is used to control the client to capture frames of the target application system based on the interaction data, thereby obtaining multi-frame interface images of the target application system. Training resources are generated based on the multi-frame interface images; The training resources are used to train a second user to perform relevant tasks in the target application system.

10. A data processing apparatus, characterized in that, include: The data acquisition module is used to collect interaction data between the first user and the target application system; the interaction data includes the first user's operation behavior data on the target application system and the interface state change data of the target application system. The triggering module is used to trigger the capture of the interface frames of the target application system based on the interaction data, so as to obtain multi-frame interface images of the target application system. The generation module is used to generate training resources based on the multi-frame interface images; The training resources are used to train a second user to perform relevant tasks in the target application system.

11. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores executable program instructions, and the processor executes the program instructions to implement the method of any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed in a computer, causes the computer to perform the method described in any one of claims 1 to 8.

13. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed by a processor, implement the method of any one of claims 1 to 8.

Images