A process automation implementation method and device and storage medium

Abstract

The application discloses a process automation implementation method and device, and a storage medium; the method comprises the following steps: determining a target operation command; querying a preset execution instruction set according to the target operation command, and determining target execution instruction information corresponding to the target operation command; the execution instruction set comprises at least one operation command and execution instruction information corresponding to each operation command in the at least one operation command; the execution instruction information is a knowledge graph constructed based on operation data; and operation is performed according to the target execution instruction information.

Description

Technical Field

[0001] This invention relates to the field of business support, and more particularly to a method, apparatus, and storage medium for implementing process automation. Background Technology

[0002] Robotic Process Automation (RPA) systems are applications that provide an alternative way to automate manually operated processes by mimicking the manual actions of end users on a computer. RPA systems typically consist of three parts: a designer, a control center, and an executor. The designer is responsible for designing the process, the control center distributes the process to the executors, and the executors are responsible for the final execution to complete the defined function. The goal of RPA software is to automate desktop-based business processes and workflows that conform to certain applicability criteria. These operations are generally highly repetitive, numerous, and can be defined by strict rules and results.

[0003] Related technologies provide an office automation device, such as Figure 1 As shown, the implementation method is as follows: Data acquisition module 1 collects data groups from the current office operation page, then numbers them sequentially and transmits them to data parsing module 2; data parsing module 2 parses each written data group to obtain the corresponding instruction information to be executed, then retrieves the corresponding instruction from the office system database 5, and executes the instruction using a macro program. In this process, the instruction information is preset, and the preset content varies for different office automation environments, which inevitably leads to a significant workload in implementing the solution. Furthermore, the closed-loop control method of acquisition-parsing-execution, which determines the instruction to be executed based on the state of the next stage, is opaque; that is, apart from the person who presets the instructions, others do not know how the process will be executed and cannot explicitly perceive the existence of the process. Summary of the Invention

[0004] In view of this, the main objective of the present invention is to provide a method, apparatus and storage medium for implementing process automation.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0006] This invention provides a method for implementing process automation, the method comprising:

[0007] Determine the target operation command;

[0008] The target operation command is queried according to a preset set of execution instructions to determine the target execution instruction information corresponding to the target operation command; the set of execution instructions includes at least one operation command and execution instruction information corresponding to each of the at least one operation command; the execution instruction information is a knowledge graph constructed based on operation data;

[0009] Perform the operation according to the target execution instruction information.

[0010] In the above scheme, the method further includes: constructing the execution instruction information; the construction of the execution instruction information includes:

[0011] Detection operation events;

[0012] Based on the detected operation events, generate operation data;

[0013] Based on the operation data, a set of triples is constructed; the set of triples includes at least one triple; each triple in the at least one triple represents any kind of information about the operation;

[0014] The set of triples is presented in the form of a knowledge graph.

[0015] In the above scheme, the operation events include: mouse events and keyboard events; the mouse events include: mouse swipe events and mouse click events;

[0016] The step of generating operation data based on the detected operation includes:

[0017] Corresponding to the operation event being a mouse swipe event, operation data for the mouse swipe event is obtained according to a preset first rule; the first rule is used to indicate the operation data to be obtained for the mouse swipe event.

[0018] When the operation event is a mouse click event, the desktop image at the time of the mouse click event is captured according to a preset second rule; the second rule is at least used to instruct the desktop image to be captured and saved for the mouse click event.

[0019] Corresponding to the operation event being a keyboard event, the operation data of the keyboard event is obtained according to a preset third rule; the third rule is used to indicate the operation data to be obtained for the keyboard event.

[0020] In the above scheme, the operation data includes: at least one operation event and a resource entity corresponding to each operation event in the at least one operation event;

[0021] The step of constructing a set of triples based on the operation data includes:

[0022] Identify the resource entity corresponding to each operation event in at least one operation event in the operation data;

[0023] Each resource entity is assigned a unique identifier; the identifier is used to at least characterize the order in which each resource entity was generated with at least one other resource entity.

[0024] Determine the topic, attributes, and attribute values ​​of the operations in each of the resource entities;

[0025] Construct at least one triplet based on the topic, attribute, and attribute value of the operation in the resource entity;

[0026] Construct the set of triples based on at least one constructed triple.

[0027] In the above scheme, presenting the set of triples in the form of a knowledge graph includes:

[0028] The set of triples is identified using a pre-defined knowledge graph model, and the knowledge graph corresponding to the set of triples is obtained.

[0029] The method in the above scheme further includes:

[0030] Receive editing instructions; the editing instructions are used to perform at least one of the following: trim resource entities in the knowledge graph, add resource entities to the knowledge graph, and modify resource entities in the knowledge graph;

[0031] Modify the knowledge graph according to the editing instructions.

[0032] The method in the above scheme further includes:

[0033] Obtain a preset graph recognition model; the graph recognition model is obtained by training a neural network based on a knowledge graph training set;

[0034] The constructed knowledge graph is identified using the graph recognition model, and the recognition results are obtained and presented.

[0035] The identification result includes at least one of the following:

[0036] Are there any errors or incorrect content?

[0037] Suggested additions: knowledge graph fragments;

[0038] Suggested modifications to the knowledge graph fragment and the methods for modification.

[0039] In the above scheme, the knowledge graph model includes at least one of the following preset resources: condition switching subset, numerical judgment subset, location anomaly handling subset, pop-up detection subset, and pop-up handling subset;

[0040] The method further includes at least one of the following:

[0041] The pop-up detection subset is invoked to detect whether a pop-up has appeared. When a pop-up is detected, the pop-up processing subset is invoked to close the pop-up.

[0042] When the target execution instruction information includes a preset first type of operation, the condition switching subset is called to determine whether the conditions for executing the preset first type of operation are met. If the conditions for executing the preset first type of operation are met, the target execution instruction information is executed.

[0043] When the target execution instruction information includes a preset second type of operation, the numerical judgment subset is called to determine whether the conditions for executing the preset second type of operation are met. If the conditions for executing the preset second type of operation are met, the target execution instruction information is executed.

[0044] During the execution of the target execution instruction information, it is detected whether there is an anomaly. If an anomaly is determined, the location anomaly handling subset is invoked to eliminate the anomaly.

[0045] In the above scheme, the step of performing the operation according to the target execution instruction information includes:

[0046] When it is determined that the target execution instruction information includes a mouse click event, the image corresponding to the mouse click event is obtained;

[0047] Capture desktop images;

[0048] The desktop image is identified using image recognition technology to determine the same application icon in the desktop image that corresponds to the mouse click event stored in the image.

[0049] When the mouse click event is executed, the mouse click operation is performed based on the determined application icon.

[0050] This invention provides a device for implementing process automation, comprising: a determining module, a first processing module, and a second processing module; wherein,

[0051] The determining module is used to determine the target operation command;

[0052] The first processing module is configured to query a preset set of execution instructions based on the target operation command to determine the target execution instruction information corresponding to the target operation command; the set of execution instructions includes at least one operation command and execution instruction information corresponding to each of the at least one operation command; the execution instruction information is a knowledge graph constructed based on operation data;

[0053] The second processing module is used to perform operations based on the target execution instruction information.

[0054] In the above scheme, the device further includes a preprocessing module for constructing the execution instruction information;

[0055] The preprocessing module is specifically used to detect operation events; generate operation data based on the detected operation events; construct a set of triples based on the operation data; the set of triples includes at least one triple; each triple in the at least one triple represents any kind of information about the operation;

[0056] The set of triples is presented in the form of a knowledge graph.

[0057] In the above scheme, the operation events include: mouse events and keyboard events; the mouse events include: mouse swipe events and mouse click events;

[0058] The preprocessing module is specifically used to obtain operation data for a mouse swipe event according to a preset first rule, corresponding to the operation event being a mouse swipe event; the first rule is used to indicate the operation data to be obtained for the mouse swipe event.

[0059] When the operation event is a mouse click event, the desktop image at the time of the mouse click event is captured according to a preset second rule; the second rule is at least used to instruct the desktop image to be captured and saved for the mouse click event.

[0060] Corresponding to the operation event being a keyboard event, the operation data of the keyboard event is obtained according to a preset third rule; the third rule is used to indicate the operation data to be obtained for the keyboard event.

[0061] In the above scheme, the operation data includes: at least one operation event and a resource entity corresponding to each operation event in the at least one operation event;

[0062] The preprocessing module is specifically used to identify the resource entity corresponding to each operation event in at least one operation event in the operation data;

[0063] Each resource entity is assigned a unique identifier; the identifier is used to at least characterize the order in which each resource entity was generated with at least one other resource entity.

[0064] Determine the topic, attributes, and attribute values ​​of the operations in each of the resource entities;

[0065] Construct at least one triplet based on the topic, attribute, and attribute value of the operation in the resource entity;

[0066] Construct the set of triples based on at least one constructed triple.

[0067] In the above scheme, the preprocessing module is specifically used to identify the set of triples using a preset knowledge graph model, and obtain the knowledge graph corresponding to the set of triples.

[0068] In the above scheme, the preprocessing module is further configured to receive editing instructions; the editing instructions are configured to perform at least one of the following: trimming resource entities in the knowledge graph, adding resource entities to the knowledge graph, and modifying resource entities in the knowledge graph;

[0069] Modify the knowledge graph according to the editing instructions.

[0070] In the above scheme, the preprocessing module is further used to obtain a preset graph recognition model; the graph recognition model is obtained by training a neural network based on a knowledge graph training set;

[0071] The constructed knowledge graph is identified using the graph recognition model, and the recognition results are obtained and presented.

[0072] The identification result includes at least one of the following:

[0073] Are there any errors or incorrect content?

[0074] Suggested additions: knowledge graph fragments;

[0075] Suggested modifications to the knowledge graph fragment and the methods for modification.

[0076] In the above scheme, the knowledge graph model includes at least one of the following preset resources: condition switching subset, numerical judgment subset, location anomaly handling subset, pop-up detection subset, and pop-up handling subset;

[0077] The second processing module is further configured to perform at least one of the following:

[0078] The pop-up detection subset is invoked to detect whether a pop-up has appeared. When a pop-up is detected, the pop-up processing subset is invoked to close the pop-up.

[0079] When the target execution instruction information includes a preset first type of operation, the condition switching subset is called to determine whether the conditions for executing the preset first type of operation are met. If the conditions for executing the preset first type of operation are met, the target execution instruction information is executed.

[0080] When the target execution instruction information includes a preset second type of operation, a numerical judgment subset is called to determine whether the conditions for executing the preset second type of operation are met. If the conditions for executing the preset second type of operation are met, the target execution instruction information is executed.

[0081] During the execution of the target instruction information, check for any abnormalities. If an abnormality is found, call the location exception handling subset to exclude the abnormality.

[0082] In the above scheme, the second processing module is used to obtain the image stored corresponding to the mouse click event when it is determined that the target execution instruction information includes a mouse click event;

[0083] Capture desktop images;

[0084] The desktop image is identified using image recognition technology to determine the same application icon in the desktop image that corresponds to the mouse click event stored in the image.

[0085] When the mouse click event is executed, the mouse click operation is performed based on the determined application icon.

[0086] This invention provides an apparatus for implementing process automation, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the methods described on the server side.

[0087] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described on the server side.

[0088] This invention provides a method, apparatus, and storage medium for implementing process automation. The method includes: determining a target operation command; querying a preset set of execution instructions based on the target operation command to determine target execution instruction information corresponding to the target operation command; the set of execution instructions includes at least one operation command and execution instruction information corresponding to each of the at least one operation command; the execution instruction information is a knowledge graph constructed based on operation data; and performing an operation based on the target execution instruction information. Thus, by utilizing knowledge graph technology, a visualized and editable knowledge graph (i.e., execution instruction information) is provided, improving process transparency and editability. Attached Figure Description

[0089] Figure 1 This is a schematic diagram of a process automation implementation device in related technologies;

[0090] Figure 2 A flowchart illustrating a method for implementing process automation provided in an embodiment of the present invention;

[0091] Figure 3 A schematic diagram of a knowledge graph fragment provided in an embodiment of the present invention;

[0092] Figure 4 A schematic diagram of a device for implementing process automation based on knowledge graph technology, provided in an embodiment of the present invention;

[0093] Figure 5 A schematic diagram illustrating a subset classification method provided in an embodiment of the present invention;

[0094] Figure 6 A flowchart illustrating a method for implementing process automation based on knowledge graph technology, provided in an embodiment of the present invention;

[0095] Figure 7 A schematic diagram of a process automation implementation device provided in an embodiment of the present invention;

[0096] Figure 8 This is a schematic diagram of another process automation implementation device provided in an embodiment of the present invention. Detailed Implementation

[0097] The present invention will be further described in detail below with reference to the embodiments.

[0098] Figure 2 A flowchart illustrating a method for implementing process automation provided in an embodiment of the present invention; as shown. Figure 2 As shown, the method can be applied to smart devices, such as servers and computers; the smart device has an application installed that creates and executes AutoHotkey; the method includes:

[0099] Step 201: Determine the target operation command;

[0100] Step 202: Query a preset set of execution instructions based on the target operation command to determine the target execution instruction information corresponding to the target operation command; the set of execution instructions includes at least one operation command and execution instruction information corresponding to each of the at least one operation command; the execution instruction information is a knowledge graph constructed based on operation data;

[0101] Step 203: Perform the operation according to the target execution instruction information.

[0102] In practical applications, users can pre-build execution instruction information through the human-computer interaction module of smart devices (including displays, mice, keyboards, etc.), and set operation commands to call this execution instruction information. Thus, during the application process, the operation of the corresponding execution instruction information can be triggered by the operation commands.

[0103] Based on this, in some embodiments, the method further includes: constructing the execution instruction information.

[0104] Constructing the execution instruction information includes:

[0105] Detection operation events;

[0106] Based on the detected operation events, generate operation data;

[0107] Based on the operation data, a set of triples is constructed; the set of triples includes at least one triple; each triple in the at least one triple represents any kind of information about the operation;

[0108] The set of triples is presented in the form of a knowledge graph. This knowledge graph is used to automate processes using AutoHotkey.

[0109] Specifically, the operation events include: mouse events and keyboard events; the mouse events include: mouse swipe events and mouse click events.

[0110] Accordingly, the operation data includes: mouse event data and keyboard event data; the mouse event data includes: mouse swipe event data, mouse click event data, and the image corresponding to the mouse click event (such as the image of the clicked application icon).

[0111] Here, you can use modules including but not limited to PyHook3 (a program that provides callbacks for global mouse and keyboard events in the Windows system) to detect system-wide mouse and keyboard events.

[0112] The step of generating operation data based on the detected operation includes:

[0113] Corresponding to the operation event being a mouse swipe event, operation data for the mouse swipe event is obtained according to a preset first rule; the first rule is used to indicate the operation data to be obtained for the mouse swipe event.

[0114] When the operation event is a mouse click event, the desktop image at the time of the mouse click event is captured according to a preset second rule; the second rule is at least used to instruct the desktop image to be captured and saved for the mouse click event.

[0115] Corresponding to the operation event being a keyboard event, the operation data of the keyboard event is obtained according to a preset third rule; the third rule is used to indicate the operation data to be obtained for the keyboard event.

[0116] Specifically, you can pre-set which operation data needs to be collected and saved, i.e., preset the first rule, the second rule, and the third rule.

[0117] For example, the first rule is used to indicate the operation data that needs to be obtained for a mouse swipe event. For example, to obtain the mouse's starting position (which can be identified by the mouse's coordinates on the screen) and ending position, the mouse's coordinates can be obtained, i.e., Position in Table 1 below.

[0118] The second rule is used at least to instruct the capture and saving of a desktop image in response to a mouse click event. For example, when the mouse is clicked, the screenshot function of Windows can be used to capture a screenshot of the screen and extract the image of the clicked icon, and save it in a specified location. When constructing a triple about the image, the captured image can be represented by a link or shortcut.

[0119] The third rule is used to indicate the operation data that needs to be obtained for keyboard events. For example, to obtain which keys on the keyboard were operated and the order in which the keys were operated, the ASCII values ​​in Table 1 can be obtained.

[0120] The basic data format for operational data can be shown in Table 1 below:

[0121]

[0122]

[0123] Table 1

[0124] In practical applications, considering that there may be unnecessary or invalid data in the operation data, such as meaningless mouse movements, a method is provided to pre-process the operation data in order to improve the efficiency of constructing triplet sets.

[0125] Based on this, in some embodiments, before constructing the triplet set according to the operational data, the method further includes:

[0126] Merge similar events, filter simple events, and categorize events to form subsets.

[0127] Among them, merging similar events includes: analyzing the detected operations according to the preset fourth rule, and when it is determined that there are continuous repeated operations, merging the repeated operations to obtain the first target operation;

[0128] Filtering simple events includes: analyzing the detected operations according to the preset fourth rule, filtering out invalid operations when invalid operations are determined to obtain the second target operation;

[0129] The classification of events forms a subset, including: analyzing the detected operations according to the preset fourth rule, and when it is determined that there are at least two operation categories, classifying the detected operations to obtain sub-operation data corresponding to each of the at least two operation categories.

[0130] The preset fourth rule describes the method for preliminary processing of operational data and can be preset and saved by developers. For example, to achieve a search, the following operations are required: copy search keywords from a document, move the mouse to the browser, click the browser icon to open the browser, click the search box (or access the search website), paste the search keywords, and click the search button. During this process, the mouse can move freely on the desktop to locate the browser. This movement may not be a straight line, but the most important starting and ending positions in locating the browser are crucial. Random movements in between can be considered meaningless and need to be filtered; this filtering process is for simple events. Similarly, clicking the browser icon to open the browser may involve clicking twice or three times, with the sole purpose of opening the browser. Therefore, this clicking action can also be combined to achieve the goal of opening the browser; this merging process is for similar events. Regarding the formation of subsets for categorized events, since the above operations involve two applications—document and browser—they can be categorized according to different applications. For example, operations in a document (such as copying search keywords) and operations in a browser (such as clicking the search box (or accessing a search website), pasting search keywords, clicking the search button, etc.) can be categorized into two subsets, thus obtaining sub-operation data corresponding to the two operation categories respectively. Of course, other methods can be used for categorization in practice; this is merely an example and not a limitation.

[0131] The preliminary processing can filter the operational data to obtain effective operational data, thereby improving the efficiency of subsequent operations.

[0132] In some embodiments, the operation data includes: at least one operation event and a resource entity corresponding to each operation event in the at least one operation event;

[0133] The step of constructing a set of triples based on the operation data includes:

[0134] Identify the resource entity corresponding to each operation event in at least one operation event in the operation data;

[0135] Each resource entity is assigned a unique identifier; the identifier is used to at least characterize the order in which each resource entity was generated with at least one other resource entity.

[0136] Determine the topic, attributes, and attribute values ​​of the operations in each of the resource entities;

[0137] Construct at least one triplet based on the topic, attribute, and attribute value of the operation in the resource entity;

[0138] Construct the set of triples based on at least one constructed triple.

[0139] Specifically, as in the search example above, a user's execution of a task may involve multiple operations, and these operations have a specific order. For instance, copying search keywords is necessary before pasting them. Therefore, it is necessary to determine the order of these operations. Here, a unique identifier is assigned to each resource entity, and the order of operations can be determined through this assigned identifier.

[0140] Each operation entity may include multiple operation information, each of which includes: topic, attribute, and attribute value. Taking the mouse events in Table 1 as an example, for a certain mouse event, it specifically includes: mouse swipe event and mouse click event; each mouse swipe event and mouse click event corresponds to a resource entity.

[0141] The resource entities for mouse hover events may include the following:

[0142] Operation Information 1: Subject (Event Name (message_name)), Attribute (is), Attribute Value (mousemove);

[0143] Operation information 2: Subject (window_name), Attribute (is), Attribute value (FolderView).

[0144] The resource entities for a mouse click event may include the following:

[0145] Operation information 1: Subject (image (target)), attribute (is), attribute value (image location indicator, such as test1\1545421218.png);

[0146] Operation Information 2: Subject (coordinates relative to the entire screen when the event occurs (position)), attribute (is), attribute value (specific coordinates, such as (1593,553)).

[0147] In some embodiments, presenting the set of triples in the form of a knowledge graph includes:

[0148] The set of triples is identified using a pre-defined knowledge graph model, and the knowledge graph corresponding to the set of triples is obtained.

[0149] Specifically, a knowledge graph is a series of various graphs that display the development process and structural relationships of knowledge. It uses visualization technology to describe knowledge resources and their carriers, and to mine, analyze, construct, draw, and display knowledge and the interrelationships between them. Here, a pre-defined knowledge graph model is used to convert a set of triples into a knowledge graph format, which can more intuitively present operational data to users for easier viewing.

[0150] In practical applications, considering the possibility of adjustments being made during the construction of knowledge graphs for process automation, methods for modifying knowledge graphs are provided.

[0151] Based on this, in some embodiments, after presenting the knowledge graph through a human-computer interaction interface, the method further includes:

[0152] Receive editing instructions; the editing instructions are used to perform at least one of the following: trim resource entities in the knowledge graph, add resource entities to the knowledge graph, and modify resource entities in the knowledge graph;

[0153] Modify the knowledge graph according to the editing instructions.

[0154] Modifying resource entities in a knowledge graph can include: modifying the specific content of a resource entity, modifying the link relationships between resource entities, etc.

[0155] Specifically, during the process of building a knowledge graph using the methods described above, users may consider a certain operation to be invalid or deletable, or they may think that a certain operation needs to be added to improve efficiency. Therefore, the knowledge graph needs to be modified.

[0156] like Figure 3 As shown, Figure 3 This is a schematic diagram of a knowledge graph fragment provided in an embodiment of the present invention. After obtaining the knowledge graph, it can be presented through a human-computer interaction interface. Modifications to the knowledge graph can be made, such as deleting the selected dotted line portion to achieve knowledge graph trimming; for example, a link relationship can be established between resource entity "17821623" and resource entity "22012881"; of course, other resource entities can also be added or the link relationships of other resource entities can be modified, etc.

[0157] In practical applications, to improve the efficiency of knowledge graph construction, a rule-based reasoning method for knowledge graphs is provided.

[0158] Based on this, in some embodiments, the method further includes:

[0159] Obtain a preset graph recognition model; the graph recognition model is obtained by training a neural network based on a knowledge graph training set;

[0160] The constructed knowledge graph is identified using the aforementioned graph recognition model to obtain recognition results; the recognition results include at least one of the following:

[0161] Are there any errors or incorrect content?

[0162] Suggested additions: knowledge graph fragments;

[0163] Suggested modifications to the knowledge graph fragment.

[0164] Here, the process automation implementation method in this embodiment of the invention can be combined with machine learning methods to perform rule reasoning and searching on the knowledge graph, thereby improving the ability to handle anomalies and helping users build the required knowledge graph.

[0165] Specifically, the method further includes: training a graph recognition model; specifically including:

[0166] Obtain at least one knowledge graph training sample and the label of each knowledge graph training sample in the at least one knowledge graph training sample, as a knowledge graph training set;

[0167] The neural network is trained using the knowledge graph training set and the neural network to obtain the trained neural network as the knowledge graph recognition model.

[0168] The label can be at least one of the following:

[0169] Does the corresponding knowledge graph training sample contain errors or erroneous content?

[0170] The recommended knowledge graph fragments to be added to the corresponding knowledge graph training samples;

[0171] The corresponding knowledge graph training samples suggest modifications to the knowledge graph fragments and the modification methods.

[0172] For example, when constructing a knowledge graph, a simple knowledge graph can be obtained based on operational data, but it may contain anomalies. For instance, in a search operation, the following steps are required: click the browser icon to open the browser, click the search box (or access the search website), enter search keywords, and click the search button. However, the actual knowledge graph constructed might be "click the browser icon to open the browser, click the search box (or access the search website), click the search button, and enter search keywords." In this process, the order of "click the search button" and "enter search keywords" is incorrect. A knowledge graph recognition model can identify this, suggesting that the corrected knowledge graph segment should be "click the search button, enter search keywords," and the modification method should be "enter search keywords, click the search button." Similarly, after "enter search keywords," the "click search button" step might not be executed, indicating an incomplete process, even though this step is necessary. The same knowledge graph recognition model can identify this, suggesting that the correct knowledge graph segment should be added, i.e., clicking the search button.

[0173] The above examples only illustrate that graph recognition models can be used to process knowledge graphs to achieve anomaly detection, graph inference, etc., and do not limit the methods for obtaining graph recognition models.

[0174] In practical applications, methods for handling interference are provided to improve the anti-interference capability of process automation.

[0175] Based on this, in some embodiments, the knowledge graph model includes, but is not limited to, the following preset resources: condition switching subset, numerical judgment subset, location anomaly handling subset, pop-up detection subset, pop-up handling subset, etc.

[0176] The method further includes at least one of the following:

[0177] The pop-up detection subset is invoked to detect whether a pop-up has appeared. When a pop-up is detected, the pop-up processing subset is invoked to close the pop-up.

[0178] When the target execution instruction information includes a preset first type of operation, the condition switching subset is called to determine whether the conditions for executing the preset first type of operation are met. If the conditions for executing the preset first type of operation are met, the target execution instruction information is executed.

[0179] When the target execution instruction information includes a preset second type of operation, a numerical judgment subset is called to determine whether the conditions for executing the preset second type of operation are met. If the conditions for executing the preset second type of operation are met, the target execution instruction information is executed.

[0180] During the execution of the target instruction information, check for any abnormalities. If an abnormality is found, call the location exception handling subset to exclude the abnormality.

[0181] The following examples illustrate the condition switching subset, numerical judgment subset, location anomaly handling subset, pop-up detection subset, and pop-up handling subset.

[0182] In one example, a graph fragment for pop-up detection (which includes a set of triples) obtained from the pop-up detection subset can be inserted into the target execution instruction information (i.e., the knowledge graph corresponding to the target operation command) to detect whether a pop-up has appeared. When a pop-up is determined to have appeared, a graph fragment for pop-up processing is obtained from the pop-up processing subset and inserted into the target execution instruction information (i.e., the knowledge graph corresponding to the target operation command) to process the pop-up, such as closing the pop-up or switching the pop-up.

[0183] In another example, when it is determined that the target execution instruction information includes a preset first type of operation, such as an operation requiring file upload, it is necessary to determine whether the file to be uploaded exists. Therefore, the graph segment used to determine the existence of the file in the condition switching subset can be called to determine whether the file exists (i.e., whether the conditions for executing the preset first type of operation are met). When it is determined that the conditions for executing the preset first type of operation are met, the target execution instruction information is executed.

[0184] In another example, when it is determined that the target execution instruction information includes a preset second type of operation, such as the target execution instruction information including the requirement to upload ten files, it is necessary to determine whether there are ten files. Therefore, a numerical judgment subset can be called to determine whether there are files and whether the number of files is ten (i.e., to determine whether the condition for executing the preset second type of operation is met). When it is determined that the condition for executing the preset second type of operation is met, the target execution instruction information is executed.

[0185] In another example, a map segment from the location anomaly handling subset can be inserted into the target execution instruction information, such as detecting whether a button was clicked incorrectly, in order to detect and eliminate anomalies that occur during the operation of the target execution instruction information.

[0186] The above examples are only for illustrating the provision of subsets such as fault detection and troubleshooting, and do not limit the type of subset. The aforementioned preset resources exist as subsets, which can be easily linked to key resources in the knowledge graph through specific attributes. When execution reaches a resource and encounters a specific exception, it will execute according to the specific attributes to resolve the exception, and finally return to the resource to continue execution. The knowledge graph can link subsets to resources all at once through knowledge reasoning. From a programming perspective, preset resources can be understood as a series of pre-written exception handling functions that can be triggered and executed under specific conditions, and are key to the system's robustness and resistance to interference.

[0187] In practical applications, considering that application icons may move, be enlarged, shrunk, or be partially obscured, an image processing method is provided to accurately locate the application.

[0188] Based on this, in some embodiments, the operation according to the target execution instruction information includes:

[0189] When it is determined that the target execution instruction information includes a mouse click event, the image corresponding to the mouse click event is obtained;

[0190] Capture desktop images;

[0191] The desktop image is identified using image recognition technology to determine the same application icon in the desktop image that corresponds to the mouse click event stored in the image.

[0192] When the mouse click event is executed, the mouse click operation is performed based on the determined application icon.

[0193] The method provided in this invention collects user operation behavior by acquiring global mouse and keyboard events. Utilizing knowledge graph technology, it pre-constructs execution instruction information based on the collected mouse and keyboard events, proposing a method for graph visualization, graph editing, and inference engine to allow users to edit processes, offering advantages such as process transparency and flexible logic implementation. Furthermore, by leveraging knowledge graph search and inference, it introduces preset resources and corresponding triggering methods, while also possessing certain exception handling capabilities, thereby improving system robustness.

[0194] Figure 4 A schematic diagram of the structure of a process automation system based on knowledge graph technology is provided as an application embodiment of the present invention; such as Figure 4 As shown, the system includes: a learning module, a knowledge graph module, an image processing module, and an execution module.

[0195] The learning module is used to learn user actions. Specifically, user actions can be operations performed by the user through the human-computer interaction module, and these actions can include the following events: mouse movement events, mouse click events, keyboard events, etc. The learning module records these events in the order they occur and performs preliminary processing on the operation data for each event to obtain operation data. This operation data reflects the complete process of the user performing a certain operation or completing a certain function. The basic data format of the operation data is shown in Table 1 above.

[0196] For example, to accomplish a task, a user's actions may involve mouse movement events, mouse click events, and keyboard events simultaneously. For instance, to search for certain content, the actions involved include: moving the mouse to the browser (i.e., mouse movement event), opening the browser (i.e., mouse click event), moving the mouse to the search box after the browser is open (i.e., mouse movement event), clicking the search box (i.e., mouse click event), and entering search keywords (i.e., keyboard event). Therefore, the learning module needs to record and learn all of the above actions, and the recording needs to be done in chronological order.

[0197] Specifically, the learning module learns keyboard and mouse events using methods including but not limited to the PyHook3 module (a program that provides callbacks for global mouse and keyboard events in the Windows system), which learns system-wide mouse and keyboard events.

[0198] The preliminary processing methods include, but are not limited to: merging similar events, filtering simple events, and classifying events to form subsets.

[0199] Among them, merging similar events includes: analyzing the detected operations according to the preset fourth rule, and when it is determined that there are continuous repeated operations, merging the repeated operations to obtain the first target operation;

[0200] Filtering simple events includes: analyzing the detected operations according to the preset fourth rule, filtering out invalid operations when invalid operations are determined to obtain the second target operation;

[0201] The classification of events forms a subset, including: analyzing the detected operations according to the preset fourth rule, and when it is determined that there are at least two operation categories, classifying the detected operations to obtain sub-operation data corresponding to each of the at least two operation categories.

[0202] The preset fourth rule is used to describe the method of preliminary processing of operational data, and can be preset and saved by the developers.

[0203] For example, such as Figure 5As shown, a subset is a collection of resources grouped together according to specific attributes, with the aim of facilitating user management. For example, subset A could be a collection of mouse movement events, and subset B could be a collection of mouse click events. Within each subset, the content within the left circle is the event name (message_name), such as "mouse move," which refers to mouse movement; the content within the right circle is the event identifier or marker, such as "22012881," which can be used to record the sequence of events. As shown in the figure, the numbers can be automatically generated by the learning module. The generated numbers can be unique, consecutive, or non-consecutive, but regardless of whether the numbers are consecutive, there must be a sequential order between them; that is, the numbers must reflect the sequence of events.

[0204] When learning mouse events, the learning module is also used to call the image processing module to capture the image at the mouse click location for mouse click events (i.e., L2 in the figure). The captured image is saved to the system disk, and the image path is passed to the knowledge graph module (i.e., L1 and L3 in the figure) along with the user operation data.

[0205] The image processing module is used to crop images using, but not limited to, OpenCV and machine learning methods to obtain accurate images of mouse click icons or buttons. Furthermore, the image processing module will be called again during subsequent executions to help the execution module locate the browser on the screen based on the first cropped image, thus achieving better system robustness. Using the above retrieval example, if the browser's position on the screen changes, the image processing module can be used to identify the browser's icon. During application execution, the icons of various applications on the screen are identified, and the browser's icon is determined, thereby achieving the browser's location on the screen.

[0206] The knowledge graph module is the most important module in the entire system. It displays user operation data and image paths in the graph as triples, and connects them according to the order of user operations to achieve data persistence. Knowledge graph management and persistence operations can be performed using, but are not limited to, RDF operation libraries (rdflib library).

[0207] Through the knowledge graph module, users can edit the graph visually, including but not limited to cropping, adding, and linking. For example... Figure 5 As shown, Figure 5 This is a schematic diagram of a knowledge graph fragment provided in an embodiment of the present invention; the knowledge graph module can be presented in a visual interface and provides modification functions; that is, users can also operate on the knowledge graph through the visual interface, such as selecting the dotted line part in the figure to delete it, so as to realize the trimming of the graph.

[0208] Users can also define reasoning rules to perform knowledge reasoning on the knowledge graph, thereby generating a richer graph. The knowledge graph module also includes many preset resources to help users achieve more flexible functions when editing the graph.

[0209] Specifically, the triple is represented in the form of "subject-attribute-attribute value", such as "I eat tomatoes" and "the building is 50 meters tall". Here, "I" and "the building" are subjects, "eat" and "tall" are considered attributes, and the following is the attribute value. The basic format for storing operational data and image paths in the knowledge graph module is as follows:

[0210]

[0211]

[0212] The above provides two resources from the knowledge graph module. These resources are based on... <rdf:description>A resource is enclosed in elements, and its identifier is the attribute value following `rdf:about`. A resource consists of multiple triples. In the above resources... <ns1:target> test1\1545421218.png< / ns1:target> The image path is recorded. In the above resource, "ns1:next rdf:resource="http: / / www.press.spirit / 48432298"" indicates the next resource connected to this resource. This directed element identifier reflects the order of the recorded operation data. The execution module will parse the resources one by one in this order to fully present the user operation.

[0213] The preset resources in the knowledge graph module include, but are not limited to, subsets of condition switching, numerical judgment, anomaly handling, pop-up detection, and pop-up processing. These preset resources exist as subsets and can be easily linked to key resources in the knowledge graph through specific attributes. When the execution module reaches a resource and encounters a specific anomaly, it will execute according to the specific attribute to resolve the anomaly and finally return to the resource to continue execution. The knowledge graph module can link subsets to resources all at once through knowledge reasoning.

[0214] From a programming perspective, pre-defined resources can be considered a series of pre-written exception handling functions that can be triggered and executed under specific conditions, and are key to the robustness of the system. For example, the "locate_exception" attribute is used to declare the location of a subset of exception handling functions linked to resources.

[0215]

[0216] The execution module is used to read the edited and optimized knowledge graph (i.e., L4 in the diagram) from the knowledge graph module, parse its resources, and execute them according to certain rules. During execution, if a mouse click event is encountered, a combination of coordinate and image positioning is used to achieve flexible positioning. That is, even when the user shrinks the operation window, changes the icon, or holds down the coordinates, the location can still be determined. The image positioning function is performed by the execution module calling the image processing module (L5).

[0217] The execution module uses, but is not limited to, PyAutoGUI (an automation tool that can automatically control mouse and keyboard operations using a program) to manipulate the system mouse and keyboard and restore mouse and keyboard actions.

[0218] Figure 6 A flowchart illustrating a method for implementing process automation based on knowledge graph technology, provided in an embodiment of the present invention; as shown. Figure 6 As shown, the method includes:

[0219] Step 601: The learning module learns the user's operational behavior;

[0220] Specifically, the learning module learns mouse and keyboard events based on the recorded system-wide mouse and keyboard events.

[0221] Step 602: The learning module checks whether it is a mouse click event. If it is determined to be a mouse click event, proceed to step 603; if it is determined not to be a mouse click event, proceed to step 604.

[0222] Step 603: When the learning module detects that the current event is a mouse click event, it calls the image processing module to capture and save the image.

[0223] Specifically, the image processing module captures the image when the mouse is clicked. For example, if the mouse clicks on an application, the image can show the mouse cursor at the application's icon.

[0224] Step 604: Send the operation data to the knowledge graph module. The knowledge graph module constructs RDF triples based on the operation data and presents the constructed triples through the knowledge graph.

[0225] Specifically, the operation data includes: mouse event data and keyboard event data; the mouse event data includes: mouse movement event data, mouse click event data, and the image corresponding to the mouse click event. For mouse movement events and keyboard events, the mouse movement event data and keyboard event data are sent to the knowledge graph module; for mouse click events, the mouse click event and the image captured by the image processing module corresponding to the event are sent to the knowledge graph module.

[0226] Step 605: The knowledge graph module provides a human-computer interaction interface to receive user editing operations on the knowledge graph;

[0227] Specifically, the editing operations include, but are not limited to, cropping, adding, and linking. Users can edit the knowledge graph through a human-computer interaction interface, thereby enabling visualization of operational behaviors and editing of the knowledge graph, improving the transparency and flexibility of automated processes.

[0228] Step 606: The knowledge graph module performs rule-based reasoning on the graph.

[0229] Here, the knowledge graph module can call a preset graph recognition model, use the graph recognition model to recognize the constructed knowledge graph, and obtain the recognition result; adjust the obtained knowledge graph according to the recognition result.

[0230] Regarding the graph recognition model, it has been... Figure 2 The method is explained in the diagram and will not be repeated here.

[0231] Step 607: The execution module determines the target operation command, reads the knowledge graph corresponding to the target operation command, and performs the restoration and execution based on the read knowledge graph.

[0232] It should be noted that during the execution process, if there is a mouse click operation, the image processing module can be invoked to locate elements based on the pre-captured image.

[0233] Figure 7 This is a schematic diagram of a process automation implementation device provided in an embodiment of the present invention; as shown below. Figure 7 As shown, the device includes: a determining module, a first processing module, and a second processing module; wherein,

[0234] The determining module is used to determine the target operation command;

[0235] The first processing module is configured to query a preset set of execution instructions based on the target operation command to determine the target execution instruction information corresponding to the target operation command; the set of execution instructions includes at least one operation command and execution instruction information corresponding to each of the at least one operation command; the execution instruction information is a knowledge graph constructed based on operation data;

[0236] The second processing module is used to perform operations based on the target execution instruction information.

[0237] In some embodiments, the apparatus further includes a preprocessing module for constructing the execution instruction information;

[0238] The preprocessing module is specifically used to detect operation events; generate operation data based on the detected operation events; construct a set of triples based on the operation data; the set of triples includes at least one triple; each triple in the at least one triple represents any kind of information about the operation;

[0239] The set of triples is presented in the form of a knowledge graph.

[0240] In some embodiments, the operation events include: mouse events and keyboard events; the mouse events include: mouse swipe events and mouse click events;

[0241] The preprocessing module is specifically used to obtain operation data for a mouse swipe event according to a preset first rule, corresponding to the operation event being a mouse swipe event; the first rule is used to indicate the operation data to be obtained for the mouse swipe event.

[0242] When the operation event is a mouse click event, the desktop image at the time of the mouse click event is captured according to a preset second rule; the second rule is at least used to instruct the desktop image to be captured and saved for the mouse click event.

[0243] Corresponding to the operation event being a keyboard event, the operation data of the keyboard event is obtained according to a preset third rule; the third rule is used to indicate the operation data to be obtained for the keyboard event.

[0244] In some embodiments, the operation data includes: at least one operation event and a resource entity corresponding to each operation event in the at least one operation event;

[0245] The preprocessing module is specifically used to identify the resource entity corresponding to each operation event in at least one operation event in the operation data;

[0246] Each resource entity is assigned a unique identifier; the identifier is used to at least characterize the order in which each resource entity was generated with at least one other resource entity.

[0247] Determine the topic, attributes, and attribute values ​​of the operations in each of the resource entities;

[0248] Construct at least one triplet based on the topic, attribute, and attribute value of the operation in the resource entity;

[0249] Construct the set of triples based on at least one constructed triple.

[0250] In some embodiments, the preprocessing module is specifically used to identify the set of triples using a preset knowledge graph model, and obtain the knowledge graph corresponding to the set of triples.

[0251] In some embodiments, the preprocessing module is further configured to receive editing instructions; the editing instructions are configured to perform at least one of the following: trimming resource entities in the knowledge graph, adding resource entities to the knowledge graph, and modifying resource entities in the knowledge graph;

[0252] Modify the knowledge graph according to the editing instructions.

[0253] In some embodiments, the preprocessing module is further configured to obtain a preset graph recognition model; the graph recognition model is obtained by training a neural network based on a knowledge graph training set;

[0254] The constructed knowledge graph is identified using the graph recognition model, and the recognition results are obtained and presented.

[0255] The identification result includes at least one of the following:

[0256] Are there any errors or incorrect content?

[0257] Suggested additions: knowledge graph fragments;

[0258] Suggested modifications to the knowledge graph fragment and the methods for modification.

[0259] In some embodiments, the knowledge graph model includes at least one of the following preset resources: a condition switching subset, a numerical judgment subset, a location anomaly handling subset, a pop-up detection subset, and a pop-up handling subset;

[0260] The second processing module is further configured to perform at least one of the following:

[0261] The pop-up detection subset is invoked to detect whether a pop-up has appeared. When a pop-up is detected, the pop-up processing subset is invoked to close the pop-up.

[0262] When the target execution instruction information includes a preset first type of operation, the condition switching subset is called to determine whether the conditions for executing the preset first type of operation are met. If the conditions for executing the preset first type of operation are met, the target execution instruction information is executed.

[0263] When the target execution instruction information includes a preset second type of operation, a numerical judgment subset is called to determine whether the conditions for executing the preset second type of operation are met. If the conditions for executing the preset second type of operation are met, the target execution instruction information is executed.

[0264] During the execution of the target instruction information, check for any abnormalities. If an abnormality is found, call the location exception handling subset to exclude the abnormality.

[0265] In some embodiments, the second processing module is configured to, when determining that the target execution instruction information includes a mouse click event, acquire the image stored corresponding to the mouse click event;

[0266] Capture desktop images;

[0267] The desktop image is identified using image recognition technology to determine the same application icon in the desktop image that corresponds to the mouse click event stored in the image.

[0268] When the mouse click event is executed, the mouse click operation is performed based on the determined application icon.

[0269] It should be noted that the process automation implementation apparatus provided in the above embodiments is only illustrated by the division of the above program modules when implementing the corresponding process automation method. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the smart device can be divided into different program modules to complete all or part of the processing described above. In addition, the apparatus and the corresponding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0270] Figure 8 This is a schematic diagram of another process automation implementation device provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the process automation implementation device 80 includes: a processor 801 and a memory 802 for storing computer programs that can run on the processor; when the processor 801 runs the computer program, it performs the following: determining a target operation command; querying a preset set of execution instructions according to the target operation command to determine the target execution instruction information corresponding to the target operation command; the set of execution instructions includes at least one operation command and execution instruction information corresponding to each of the at least one operation command; the execution instruction information is a knowledge graph constructed based on operation data; and performing an operation according to the target execution instruction information.

[0271] Specifically, the device for automating the process can also perform actions such as Figure 2 The method shown is the same as Figure 2 The implementation methods for process automation shown in the examples belong to the same concept, and the specific implementation process can be found in the method examples, which will not be repeated here.

[0272] In practical applications, the process automation implementation device 80 may further include at least one network interface 803. The various components in the process automation implementation device 80 are coupled together via a bus system 804. It is understood that the bus system 804 is used to implement communication between these components. In addition to a data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 All buses are labeled as bus system 804. The number of processors 801 can be at least one. Network interface 803 is used for wired or wireless communication between the process automation implementation device 80 and other devices.

[0273] The memory 802 in this embodiment of the invention is used to store various types of data to support the operation of the process automation implementation device 80.

[0274] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 801. Processor 801 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 801 or by instructions in software form. The processor 801 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 801 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 802. Processor 801 reads the information in memory 802 and combines it with its hardware to complete the steps of the aforementioned method.

[0275] In an exemplary embodiment, the process automation implementation apparatus 80 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0276] This invention also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it performs the following: determining a target operation command; querying a preset set of execution instructions based on the target operation command to determine target execution instruction information corresponding to the target operation command; the set of execution instructions includes at least one operation command and execution instruction information corresponding to each of the at least one operation command; the execution instruction information is a knowledge graph constructed based on operation data; and performing an operation based on the target execution instruction information. Specifically, the computer program can also perform the following: Figure 2 The method shown is the same as Figure 2 The implementation methods for process automation shown in the examples belong to the same concept, and the specific implementation process can be found in the method examples, which will not be repeated here.

[0277] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0278] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0279] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0280] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0281] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0282] It should be noted that "first," "second," etc., are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0283] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0284] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.< / rdf:description>

Claims

1. A method for implementing process automation, applied to a smart device, characterized in that, The method includes: Determine the target operation command; The target operation command is queried according to a preset set of execution instructions to determine the target execution instruction information corresponding to the target operation command; the set of execution instructions includes at least one operation command and execution instruction information corresponding to each of the at least one operation command; the execution instruction information is a knowledge graph constructed based on operation data; The operation is performed based on the target execution instruction information; the target execution instruction information is a knowledge graph corresponding to the target operation command. The method further includes: constructing the execution instruction information; the construction of the execution instruction information includes: Detection operation events; Based on the detected operation events, generate operation data; Based on the operation data, a set of triples is constructed; the set of triples includes at least one triple; each triple in the at least one triple represents any kind of information about the operation; The set of triples is presented in the form of a knowledge graph; The operation events include: mouse events and keyboard events; the mouse events include: mouse swipe events and mouse click events. The step of generating operation data based on the detected operation includes: Corresponding to the operation event being a mouse swipe event, operation data for the mouse swipe event is obtained according to a preset first rule; the first rule is used to indicate the operation data to be obtained for the mouse swipe event. When the operation event is a mouse click event, the desktop image at the time of the mouse click event is captured according to a preset second rule; the second rule is at least used to instruct the desktop image to be captured and saved for the mouse click event. Corresponding to the operation event being a keyboard event, the operation data of the keyboard event is obtained according to a preset third rule; the third rule is used to indicate the operation data to be obtained for the keyboard event. The operation data includes: at least one operation event and a resource entity corresponding to each operation event in the at least one operation event; The step of constructing a set of triples based on the operation data includes: Identify the resource entity corresponding to each operation event in at least one operation event in the operation data; Each resource entity is assigned a unique identifier; the identifier is used to at least characterize the order in which each resource entity was generated with at least one other resource entity. Determine the topic, attributes, and attribute values ​​of the operations in each of the resource entities; Construct at least one triplet based on the topic, attribute, and attribute value of the operation in the resource entity; Construct the set of triples based on at least one constructed triple.

2. The method according to claim 1, characterized in that, The presentation of the triple set in the form of a knowledge graph includes: The set of triples is identified using a pre-defined knowledge graph model, and the knowledge graph corresponding to the set of triples is obtained.

3. The method according to claim 1, characterized in that, The method further includes: Receive editing instructions; the editing instructions are used to perform at least one of the following: trim resource entities in the knowledge graph, add resource entities to the knowledge graph, and modify resource entities in the knowledge graph; Modify the knowledge graph according to the editing instructions.

4. The method according to claim 1, characterized in that, The method further includes: Obtain a preset graph recognition model; the graph recognition model is obtained by training a neural network based on a knowledge graph training set; The constructed knowledge graph is identified using the graph recognition model, and the recognition results are obtained and presented. The identification result includes at least one of the following: Are there any errors or incorrect content? Suggested additions: knowledge graph fragments; Suggested modifications to the knowledge graph fragment and the methods for modification.

5. The method according to claim 1, characterized in that, The knowledge graph model includes at least one of the following preset resources: condition switching subset, numerical judgment subset, anomaly handling subset, pop-up detection subset, and pop-up handling subset; The method further includes at least one of the following: The pop-up detection subset is invoked to detect whether a pop-up has appeared. When a pop-up is detected, the pop-up processing subset is invoked to close the pop-up. When the target execution instruction information includes a preset first type of operation, the condition switching subset is called to determine whether the conditions for executing the preset first type of operation are met. If the conditions for executing the preset first type of operation are met, the target execution instruction information is executed. When the target execution instruction information includes a preset second type of operation, the numerical judgment subset is called to determine whether the conditions for executing the preset second type of operation are met. If the conditions for executing the preset second type of operation are met, the target execution instruction information is executed. During the execution of the target execution instruction information, it is detected whether there is an anomaly. If an anomaly is determined, the location anomaly handling subset is invoked to eliminate the anomaly.

6. The method according to claim 1, characterized in that, The operation based on the target execution instruction information includes: When it is determined that the target execution instruction information includes a mouse click event, the image corresponding to the mouse click event is obtained; Capture desktop images; The desktop image is identified using image recognition technology, and the same application icon in the desktop image that corresponds to the mouse click event is determined. When the mouse click event is executed, a mouse click operation is performed based on the determined application icon.

7. A device for implementing process automation, characterized in that, include: The module is defined as a first processing module and a second processing module; among which, The determining module is used to determine the target operation command; The first processing module is configured to query a preset set of execution instructions based on the target operation command to determine the target execution instruction information corresponding to the target operation command; the set of execution instructions includes at least one operation command and execution instruction information corresponding to each of the at least one operation command; the execution instruction information is a knowledge graph constructed based on operation data; The second processing module is used to perform operations based on the target execution instruction information; the target execution instruction information is a knowledge graph corresponding to the target operation command; The device further includes a preprocessing module for constructing the execution instruction information; The preprocessing module is specifically used to detect operation events; generate operation data based on the detected operation events; construct a set of triples based on the operation data; the set of triples includes at least one triple; each triple in the at least one triple represents any kind of information about the operation; and present the set of triples in the form of a knowledge graph. The operation events include: mouse events and keyboard events; the mouse events include: mouse swipe events and mouse click events. The preprocessing module is specifically used to obtain operation data for a mouse swipe event according to a preset first rule, corresponding to the operation event being a mouse swipe event; the first rule is used to indicate the operation data to be obtained for the mouse swipe event. When the operation event is a mouse click event, the desktop image at the time of the mouse click event is captured according to a preset second rule; the second rule is at least used to instruct the desktop image to be captured and saved for the mouse click event. Corresponding to the operation event being a keyboard event, the operation data of the keyboard event is obtained according to a preset third rule; the third rule is used to indicate the operation data to be obtained for the keyboard event. The operation data includes: at least one operation event and a resource entity corresponding to each operation event in the at least one operation event; The preprocessing module is specifically used to identify the resource entity corresponding to each operation event in at least one operation event in the operation data; assign a unique identifier to each resource entity; the identifier is used to characterize the order in which each resource entity is generated with at least one other resource entity; determine the topic, attribute, and attribute value of the operation in each resource entity; construct at least one triplet based on the topic, attribute, and attribute value of the operation in the resource entity; and construct the triplet set based on the constructed at least one triplet.

8. An apparatus for implementing process automation, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images