Method, apparatus and computer program product for processing workflow charts

Abstract

An example embodiment according to the present disclosure provides a method, apparatus, and computer program product for processing a workflow chart. The method comprises the following steps: encoding structure information of a workflow chart comprising a plurality of nodes and a plurality of edges by using a graph neural network to obtain vector representation of the structure information; obtaining text description data of the workflow chart at the node; based on the obtained text description data and the obtained vector representation, training a language model to obtain a pre-trained language model; and performing fine tuning on the pre-trained language model through the training data of the specific task to obtain a fine-tuned language model. Through the method for processing the flow chart, the combination of the graph neural network and the language model not only can process a large number of complex workflow charts, but also can generate natural language output, so that effective and organized customer services can be provided by utilizing the trained and fine-tuned language model.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of graph processing, and more particularly, to methods, devices, and computer program products for processing workflow graphs. Background Art

[0002] During the online after-sales service stage, customers (e.g., laptop purchasers) will consult product suppliers (e.g., laptop after-sales service providers) online about solutions to their product failures. However, multiple causes may lead to the same product failure (e.g., the computer cannot be turned on or automatically shuts down, etc.). There are different solutions for each cause. For example, if the computer cannot be turned on, it may be due to improper maintenance in the past, power failure, memory failure, computer motherboard failure, CPU failure, graphics card failure, etc. There are different solutions for different causes, so a large number of workflow diagrams will be generated for after-sales service to guide customers to choose appropriate solutions.

[0003] The latest developments in text language processing and machine learning have produced powerful language models, such as text-based language models (e.g., GPT), which can generate questions similar to human language to guide customers to answer or select options. As mentioned above, there are a large number of workflow diagrams for the failure of electronic products such as computers. In traditional technology, the workflow diagram is encoded into a picture, which can show steps and conditions. The staff of the customer service provider can provide consultation and service to customers in accordance with the steps and conditions shown on the picture. If GPT can be introduced to ask questions to customers and guide customers to find solutions based on their answers, this can greatly improve the efficiency of customer service and reduce after-sales service costs. However, it is not feasible to directly input the diagram representation of the workflow diagram into GPT, because this requires encoding the diagram representation into a long text prompt that GPT can understand and input it into GPT, which will be a major challenge. Summary of the invention

[0004] According to an exemplary embodiment of the present disclosure, a method, device and computer program product for processing a workflow diagram are provided. In an embodiment of the present disclosure, in the case of a known graph representation of a workflow diagram, a graph neural network (GNN, such as GCN and GAT) is used to encode the structural information of the graph representation to generate a soft prompt, which contains a complete encoding of the graph information and can be understood by a basic text language model (e.g., GPT) and can be directly input into the GPT. A hard prompt is a text description of a specific step of the workflow diagram, which is a comprehensive and detailed description of the specific step of the workflow diagram. The method according to the present disclosure takes into account both soft prompts and hard prompts. By using the generated soft prompts and hard prompts as input, the GPT is pre-trained using a graph completion task, so that the GPT can understand the basic structure or underlying structure of the workflow diagram, that is, learn the entire workflow diagram to achieve a complete description of the workflow diagram. In the fine-tuning process, the pre-trained GPT is fine-tuned to adjust its parameters using training data for a specific task, specifically, by only inputting soft prompts and part of the hard prompts to complete a specific task to output a solution for the specific task, thereby optimizing the performance of the GPT for a specific task. The method for processing a workflow diagram according to the present disclosure effectively encodes the graph information of the workflow diagram into a language model so that the trained and fine-tuned GPT can be used to provide effective and organized customer service. The combination of the graph neural network and the language model can not only process a large number of complex workflow diagrams, but also generate natural language output.

[0005] In a first aspect of an embodiment of the present disclosure, a method for processing a workflow diagram is provided. The method includes: encoding structural information of a workflow diagram including multiple nodes and multiple edges using a graph neural network to obtain a vector representation of the structural information. The method also includes obtaining text description data about the workflow diagram at the node. The method also includes training a language model based on the obtained text description data and the obtained vector representation to obtain a pre-trained language model. The method also includes fine-tuning the pre-trained language model through training data of a specific task to obtain a fine-tuned language model.

[0006] In a second aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes at least one processor; and instructions coupled to the at least one processor and stored thereon, wherein the instructions, when executed by the at least one processor, cause the electronic device to perform the following work, wherein the action includes encoding the structural information of a workflow diagram including multiple nodes and multiple edges using a graph neural network to obtain a vector representation of the structural information. The action also includes obtaining text description data about the workflow diagram at a node. The action also includes training a language model based on the obtained text description data and the obtained vector representation to obtain a pre-trained language model. The action also includes fine-tuning the pre-trained language model through training data of a specific task to obtain a fine-tuned language model.

[0007] In a third aspect of an embodiment of the present disclosure, a computer program product is provided, which is tangibly stored on a non-volatile computer-readable medium and includes machine-executable instructions, which when executed cause a machine to perform the method according to the first aspect of the present disclosure.

[0008] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1A A schematic diagram illustrating an example environment in which various embodiments of the present disclosure can be implemented;

[0011] Figure 1B A schematic diagram showing a method for processing a workflow diagram according to some embodiments of the present disclosure is shown;

[0012] Figure 2 A schematic diagram showing a process for processing a workflow diagram according to some embodiments of the present disclosure;

[0013] Figure 3 A schematic diagram showing a portion of a workflow diagram according to some embodiments of the present disclosure;

[0014] Figure 4 A schematic diagram showing a process of encoding structural information of a workflow diagram to obtain a vector representation of the structural information according to some embodiments of the present disclosure;

[0015] Figure 5 A schematic diagram showing a process for generating text description data of a node according to some embodiments of the present disclosure;

[0016] Figure 6 A schematic diagram showing a flowchart for a specific task in a fine-tuning process according to some embodiments of the present disclosure; and

[0017] Figure 7 A block diagram of a computing device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] As mentioned above, the latest developments in text language processing and machine learning have produced powerful language models, such as GPT, which can generate questions similar to human language to guide customers to answer or select options. As mentioned above, there are a large number of workflow diagrams for failures of electronic products such as computers. In traditional technology, the workflow diagram is encoded into a picture that can show the steps and conditions. Customer service staff can provide consultation and services to customers in accordance with the steps and conditions shown on the picture. If GPT can be introduced to ask questions to customers and guide customers to find solutions based on their answers, this can greatly improve the efficiency of customer service and reduce after-sales service costs. However, this requires GPT to convert workflow diagrams containing large amounts of data and graphs into language models and provide services to customers, which will be a major challenge.

[0021] To this end, an embodiment of the present disclosure provides a method for generating a language model for processing a workflow diagram, through which a language model that can process a workflow diagram containing a large amount of data (for example, a workflow diagram for solutions to various problems, where the workflow diagram for each problem includes multiple steps, multiple branches, and multiple solutions) can be obtained. The method of the present application can encode the diagram information in a predetermined workflow diagram into a language model, and

[0022] Figure 1A A schematic diagram of an example environment 100 in which multiple embodiments of the present disclosure can be implemented is shown. The computing device 102 in the example environment 100 can be any device with computing capabilities. As a non-limiting example, the computing device 102 can be any type of fixed computing device, mobile computing device, or portable computing device, including but not limited to a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a multimedia computer, a mobile phone, etc.; in addition, all or part of the components of the computing device 102 can also be distributed in the cloud.

[0023] In the example environment 100, a user 101 asks a question to a computing device 102, which is arranged in the cloud, for example, the question may be "the computer cannot be turned on", "the computer automatically shuts down due to a blue screen", etc. The computing device 102 will ask a series of questions to the user 101, such as the service code of the computer, whether the computer has a maintenance record, the system version of the computer, the memory capacity of the computer, the processor model, the computer failure characteristics, etc. The user 101 answers these questions, and then the computing device 102 will predict the computer components that may appear and provide the user with maintenance suggestions. For example, if the user 101 adjusts or repairs the computer according to the maintenance suggestion, but the failure still exists, the computing device 102 can ask other questions, guide the user 101 to make a choice, and then give a new maintenance suggestion.

[0024] In order to enable the computing device 102 to ask a suitable question to the user 101, and after the user 101 has answered the question, ask a related second question, until finally enabling the user 101 to identify the faulty component of his computer.

[0025] The computing device 102 must process a large number of predetermined workflow diagrams and learn the workflow diagrams in order to ask appropriate questions and provide reasonable solutions to computer failures, such as suggesting that the user 101 upgrade the system, upgrade the BIOS, replace the memory stick, etc.

[0026] As described above, GPT can ask questions similar to human language and can communicate with humans, but in traditional technology, there is no solution for directly inputting a workflow diagram containing a large amount of data and diagrams into GPT and asking questions to customers.

[0027] In order to more clearly understand the method for obtaining a language model for processing a workflow diagram provided by an embodiment of the present disclosure, reference will be made to Figure 1B-Figure 7 To further describe the embodiments of the present disclosure.

[0028] Figure 1B A schematic diagram of a method for processing a workflow diagram according to some embodiments of the present disclosure is shown. Figure 1B As shown, the graph representation of the workflow diagram 10 is input into the graph neural network CNN 20 to encode the structural information of the graph representation to generate a soft prompt 30, which contains a complete encoding of the graph information and can be understood by a text-based language model (e.g., GPT) and can be directly input into the GPT. In addition, hard prompts 40 of some nodes of the flowchart are obtained based on the graph representation of the workflow diagram 10, and the hard prompts provide the necessary details for the text-based language model to complete the task. The hard prompts 40 and the soft prompts 30 are then combined to obtain a combined prompt 50. The combined prompt 50 can be understood by the GPT, and therefore will be input into the language model 60 to pre-train and fine-tune the language model 60.

[0029] When pre-training the language model GPT, the task is completed based on the graph, so that the language model 60 can understand the underlying structure of the workflow graph to achieve a complete description of the workflow graph. In addition, in the fine-tuning process, by inputting only soft prompts and some hard prompts to complete a specific task to output a solution for the specific task, the parameters of the language model 60 are fine-tuned to optimize the performance for the specific task.

[0030] By combining the advantages of both soft prompts and hard prompts, the method for processing a workflow diagram according to the present disclosure effectively encodes the graph information of the workflow diagram into a language model so that the trained and fine-tuned GPT can be used to provide effective and organized customer service. The combination of the graph neural network and the language model can not only process a large number of complex workflow diagrams, but also generate natural language output.

[0031] Figure 2 A process 200 for processing a workflow diagram according to some embodiments of the present disclosure is shown. The process 200 may be implemented by the computing device 102 of FIG1 . For ease of discussion, the process 200 will be described in conjunction with FIG1 .

[0032] In block 210, the computing device 102 encodes the structural information of the workflow diagram including multiple nodes and multiple edges using a graph neural network to obtain a vector representation of the structural information. Figure 3 A portion of a workflow diagram according to an example of the present disclosure is described below.

[0033] like Figure 3 As shown, the workflow diagram 300 only shows a part of a complete workflow diagram, and the complete workflow diagram also includes Figure 3 The workflow diagram 300 includes a plurality of nodes and a plurality of edges, each node is represented by a rectangular box and corresponds to a step in the flowchart, and each edge represents a connection line between two rectangular boxes. In other words, each node corresponds to each step of the workflow diagram, and the edge represents a connection line between rectangular boxes. The workflow diagram 300 has structural information. Figure 3 The flowchart for the memory failure is shown. For the memory failure, there are single memory case 1 and single memory case 2, and the processing flow is different.

[0034] Return to reference Figure 2 , when executing the steps shown in box 210, the workflow diagram can be first encoded into a picture, and the picture fully shows all the required workflow diagrams, such as workflow diagrams for all solutions to a certain problem. For example, there may be several solutions to the problem that the computer cannot be turned on, and each solution corresponds to a complete workflow diagram or a branch of a complete workflow diagram. The structural hierarchy or structural information placed in the workflow diagram can be independently designed by the service provider as needed, but those skilled in the art should understand that each workflow diagram has its own graph representation and structural information. Then, the graph (for example, a topological graph including nodes and edges) can be input into the graph neural network to encode the structural information contained in the graph representation of the graph to obtain a vector representation of the structural information. The vector representation can be a vector representation with a fixed length. Graph neural networks (GNNs) are very suitable for this task because the network is designed to operate on graph-structured data and can effectively encode graph information into vector representations.

[0035] In order to clearly explain how to encode the structural information of the workflow diagram to obtain the vector representation of the structural information, the following will refer to Figure 4 The encoding process 400 is described below.

[0036] The vector representation of structural information is also called soft hint. Obtaining the soft hint is a key step because it can encode the structural information of the graph into a compact and continuous vector representation, which can be used as the input of the language model. One way is to use a combination of graph convolutional network (GCN) and graph attention network (GAT) to process the graph to obtain the vector representation of its structural information, i.e. soft hint.

[0037] The graph convolutional network GCN operates on the graph adjacency matrix and the node features on the graph to calculate the continuous representation of each node. Because there is a connection between each node and the adjacent nodes, the closer the nodes are, the closer the relationship is. Therefore, when the graph convolutional network processes a certain node, in addition to considering its own node features, it also needs to consider the graph adjacency matrix.

[0038] Therefore, if Figure 4 As shown, in block 401, a graph convolutional network (GCN) is used to operate a node feature vector in a graph representation of a specific node among multiple nodes and a graph adjacency matrix for the specific node to determine a continuous representation of the specific node. Here, the specific node should be each node among all nodes contained in the workflow diagram, because it is necessary to know the structural information of the workflow diagram, so it is necessary to learn the characteristics of each node and the connection between two nodes. The graph or graph in the graph convolutional network is a topological graph in mathematics or graph theory that uses nodes and edges to establish adjacent relationships.

[0039] For example, let A∈R n×n is the adjacency matrix of the graph, where n is the number of nodes in the graph. The vector of the i-th node is denoted by x i ∈R d , where d represents the dimension of the node feature vector. Then the output of GCN for this node is represented by h i (i.e., continuous representation) can be expressed by the following formula (1):

[0040]

[0041] Among them, σ represents the activation function, A i，j represents the adjacency matrix between node i and node j, x j represents the node feature vector of the jth node, and W represents the learnable weight matrix.

[0042] like Figure 4As shown, in box 402, a graph attention network (GAT) is used to operate the node feature vector in the graph representation of a specific node to determine the weighted sum of the features of the neighboring nodes adjacent to the specific node. In the topological structure of the graph, a node can be adjacent to multiple nodes, so that there is a connection between these adjacent nodes. The graph attention network performs attention operations on the adjacent node j on the graph for each node i. The attention mechanism allows GAT to dynamically weight the contribution of these nodes to the specific node based on the importance of different adjacent nodes.

[0043] For node i, the similarity coefficient e between its adjacent node j and node j is calculated one by one using the following formula (2): i，j :

[0044] e i，j =LeakyReLUa T [Wx i ||Wx J ) (2)

[0045] Among them, the leaky rectified linear unit function (leakyReLU) is an activation function; a∈R 2dout is a learnable attention vector; W is a shared parameter whose linear mapping increases the dimension of the node features, which is a common feature enhancement method; || represents the concatenation of the transformed features of node i and node j.

[0046] Based on the similarity coefficient e calculated above i，j , and then calculate the attention score α between node i and node j according to the following formula (3): i，j :

[0047] α i，j =softmax(e i，j ) (3)

[0048] Among them, softmax is an activation function.

[0049] The function softmax can be expressed according to the following formula (4):

[0050]

[0051] Here, n represents the number of adjacent nodes.

[0052] After calculating the attention score α i，j Afterwards, the output of CAT (i.e., the weighted sum of the features of adjacent nodes) is calculated by the following formula (5):

[0053]

[0054] Among them, σ represents the activation function, W represents the shared parameters, and x j Represents the feature vector of the jth node.

[0055] Return to reference Figure 4 In block 403, a vector representation with a fixed length of structural information about the node is determined based on the weighted sum of the continuous representation and the features of the adjacent nodes. That is, based on the weighted sum of the continuous representation for the i-th node output by GCN and the features of the adjacent nodes for the i-th node output by GAT, a vector representation with a fixed length of structural information for the i-th node, i.e., a soft prompt for the node, can be obtained. The soft prompt can be directly input into the language model GPT, so that it can be understood by GPT.

[0056] The vector representation of CGN and GAT output is combined by the following formula (6):

[0057]

[0058] The combined vector representation can be used as a soft hint for training the language model GPT, which contains the structural information of all nodes in the workflow diagram. The soft hint is input into the language model GPT, which can understand the information of the diagram, such as the structural information, and learn the structural information of the diagram to understand the architectural information of the workflow diagram, so as to obtain a complete description of the workflow diagram.

[0059] In order to make good use of two types of graph neural networks (GNNs), a combined soft hint is generated by both GCN and GAT. This combined soft hint can be obtained by the following operations: first, use GCN to perform preliminary feature propagation, and then use GAT to perform attention-based weighting of multiple node features. The vector representation obtained by the combination of GCN and GAT can be used as a soft hint for the language model GPT.

[0060] Return to reference Figure 2 In box 202, the computing device 102 obtains text description data about the nodes in the workflow diagram. The text description data can also be called hard prompts that are directly input into the language model GPT. The nodes here do not need to be all nodes, but only some nodes. It can be several steps that are executed first in the flowchart, or steps that are necessary to perform a specific task. For example, if a customer needs a solution, then the required step can be: if a customer complains, provide a solution. In the method according to the present disclosure, only the text description data or hard prompt of this one step may be required.

[0061] Generating hard hints is the second key step of the method according to the present disclosure, by which the graphical representation of the workflow diagram can be encoded into a hint language that can be understood by the language model. Hard hints are text descriptions of the steps in the workflow diagram that provide additional information to the language model. The hard hints provide the necessary details for the language model GPT to complete the task. By taking the graphical representation of the workflow diagram as input, hard hints can be generated by extracting information from the node features and edge features of the graphical representation of the workflow diagram.

[0062] The following will refer to Figure 5 The process 500 of generating text description data of a node is described in detail.

[0063] like Figure 5 As shown, in block 501, a graph representation of a first specific node among multiple nodes is obtained. The first specific node can be one of a predetermined number of nodes. For example, the information represented by the specific node can be: if a customer complains, then a solution is generated.

[0064] In block 502, an edge label between a first specific node and a second specific node among the plurality of nodes is obtained. The second characteristic node is one of the plurality of nodes connected to the first specific node through an edge. For example, if two steps in a workflow diagram are connected by a connecting line, the connecting line between them may have an edge label.

[0065] In block 503 , the graph vector representation and the edge label of the first specific node are concatenated to determine text description data for the first specific node.

[0066] For example, the hard prompt h is calculated by the following formula (7): hard :

[0067] h hard =[x i ||l i，j ] (7)

[0068] Among them, x i represents the feature vector of the i-th node in the workflow diagram, l i，j Represents the edge label between node i and node j in the graph representation of the workflow diagram.

[0069] In such Figure 5 In the method shown, the edge label between node i and node j is extracted, and then the edge label is concatenated with the feature vector of node i to obtain a hard hint for node i. That is, the hard hint is generated by describing the relationship between multiple nodes represented by the graph. This method can better model the relationship between two nodes.

[0070] Although an exemplary method of obtaining a hard hint by describing the relationship between multiple nodes is described above, the hard hint may also be obtained in other ways.

[0071] For example, let x i represents the feature vector of the i-th node in the workflow diagram, where x i ∈R d , where d represents the dimension of the node feature vector. Hard prompts can also be generated by concatenating node representations or node feature vectors of multiple steps arranged in a predetermined order in the workflow diagram. For example, hard prompts can be generated by the following formula (8):

[0072] h hard =[x1||x2||x3||...||x k ] (8)

[0073] Among them, 1, 2, ...k represent k steps arranged in a predetermined order in the workflow diagram, and || represents splicing the features of the nodes.

[0074] Return to reference Figure 2 In box 203, the computing device 102 trains a language model based on the acquired text description data and the acquired vector representation to obtain a pre-trained language model. Pre-training is a key step in the method according to the present disclosure, which allows GPT to learn the basic structure of the diagram representation of the workflow diagram. When given the soft prompts generated in the above step 201 and the hard prompts generated in the above step 202 (if the generated hard prompts include hard prompts for other steps in addition to the necessary steps, then only the hard prompts for the necessary steps need to be provided), in the case of performing a diagram completion task, this step allows GPT to generate a complete description of the workflow diagram.

[0075] During pre-training, GPT performs a graph completion task with soft hints and partial hard hints (e.g., hard hints for necessary steps) as input, and then generates a complete text description of the workflow graph. The training progress can be quantified by a loss function that measures the difference between the generated text and the real text. The goal of this pre-training is to minimize the loss function, which can be achieved by adjusting the parameters of GPT using an optimization algorithm, such as the stochastic gradient descent (SGD) algorithm.

[0076] The loss function calculates the difference between the generated text and the real text. Let the input graph of the workflow diagram be represented as G, the soft prompt generated by the graph neural network (e.g., GCN and GAT) be S, the generated hard prompt be H, and the real text description of the workflow diagram be T. Then the loss function can be calculated by the following formula (9):

[0077] Loss = ||GPT(S,H)-T|| (9)

[0078] Here, GPT(S,H) is the text generated by GPT given a soft prompt S and a hard prompt H, and T is the actual text description of the workflow diagram.

[0079] The loss function can be further decomposed into two terms: reconstruction loss

[0080] ReconstructionLoss and AdversarialLoss. The reconstruction loss measures the difference between the generated text and the real text, while the adversarial loss forces the generated text to be similar to the real text in terms of language style, grammar, and vocabulary.

[0081] Therefore, considering the above two losses, the loss function can be calculated by the following formula (10):

[0082] Loss=ReconstructionLoss+AdversarialLoss (10)

[0083] The pre-training process can use unsupervised learning algorithms, such as autoencoder-based training, generative adversarial network (GAN) training, or variational autoencoder (VAE) training. The choice of training algorithm depends on the specific requirements of the task and the desired properties of the generated text.

[0084] The reconstruction loss measures the difference between the generated output and the true target. It is usually defined as the mean square error (MSE) between the generated output and the true target. The reconstruction loss can be calculated by the following formula (11):

[0085]

[0086] Where N is the number of instances in the training set, and y i Indicates the real goal, Represents the generated output.

[0087] Adversarial loss measures the confidence of the discriminator, which determines whether the generated output comes from the real data distribution. It is usually defined as the binary cross entropy loss between the predicted probability and the target label, where label 0 represents the real data and label 1 represents the generated data. The adversarial loss is calculated by the following formula (12):

[0088]

[0089] Among them, y adv,i represents the target label indicating whether the sample is a real sample or a generated sample, It should be noted that the parameters of graph neural networks (GCN and GAT) and GPT are pre-trained jointly, that is, their parameters can be adjusted.

[0090] Return to reference Figure 2 In box 204, the computing device 102 fine-tunes the pre-trained language model through the training data of the specific task to obtain a fine-tuned language model. After the language model is pre-trained, the pre-trained language model can be used to perform a specific task, so that the parameters of the pre-trained language model are further adjusted during the execution of the task to optimize the performance of the model in performing the specific task. Model fine-tuning is performed through the following process: input soft prompts, partial hard prompts, and specific tasks generated by GNN as training data, and then fine-tune the parameters of the GPT model to generate specific answers for specific tasks. This is the main difference between the pre-training process and the fine-tuning process. The fine-tuning process requires the input of a specific task (the specific task can be part of the input) and gives a specific answer to the task, while the pre-training process is the structural information of the workflow diagram contained in the learning diagram.

[0091] The process of model fine-tuning needs to provide the desired solution for the task. The user inputs the task into the pre-trained GPT, and then GPT asks some questions based on the task. Figure 6 To illustrate the problems that can be solved when fine-tuning a model for a specific task. Figure 6 As shown, in the first box 601, the pre-trained GPT asks "Is it win11?" If the user answers no, it continues to box 602, and asks "Update Windows?" in box 602. If the user answers yes to whether it is a win11 system, it continues to box 603, in which the pre-trained GPT asks "Meet the minimum requirements?".

[0092] If the user selects yes to the question in box 602, the process proceeds to box 603, otherwise the process proceeds to box 604, where a prompt "Continue to use win10 and keep updated" is displayed. If the user selects yes to the question in box 603, the process proceeds to box 605, where the pre-trained GPT asks "Are there any faults?". If the user selects no to the question in box 603, the process proceeds to box 606, where a prompt "Check compatibility and driver updates" is displayed.

[0093] If the user selects yes to the question in box 605, then proceed to box 607, where it is prompted "Are there any fault codes or information?". If the user selects no to the question in box 605, then proceed to box 608, where it is prompted "Enjoy the new system and new experience". If the user selects yes to the question in box 607, then proceed to box 609, where it is prompted "Provide solutions based on fault codes or information", then the pre-trained GPT will provide solutions based on the error codes or information. If the user selects no to the question in box 607, then proceed to box 610, where it is prompted "Run the fault checking software and record the results".

[0094] The objective function of model fine-tuning can be expressed by the following formula (13):

[0095]

[0096] Where N represents the number of samples, is the measurement of the true label y i and predicted labels The prediction loss of the previous error, and L Regularization (θ) is the regularization loss, which penalizes complex models and helps prevent overfitting, and λ is a hyperparameter that controls the trade-off between prediction loss and regularization loss.

[0097] The prediction loss can be defined as the cross entropy loss using the following formula (14):

[0098]

[0099] Where V represents the vocabulary size, y ij and are the true probability and predicted probability of the jth word in the vocabulary, respectively.

[0100] The regularization loss can be defined as the L2 regularization term by the following formula (15):

[0101]

[0102] Where k is the number of parameters in the model, θ k is the kth parameter.

[0103] The fine-tuning process can be performed using a gradient-based optimization algorithm, such as stochastic gradient descent (SGD) or adaptive moment estimation (Adam), by calculating the gradient of the objective function with respect to the parameters and updating the parameters in the direction of the negative gradient. The optimization process continues until the objective function converges or a predefined number of iterations is reached.

[0104] In the method according to the present disclosure, in the case of a known graph representation of a workflow diagram, a graph neural network (GNN, such as GCN and GAT) is used to encode the structural information of the graph representation to generate soft prompts, which contain a complete encoding of the graph information and can be understood by GPT and can be directly input into GPT. Hard prompts are text descriptions of specific steps of the workflow diagram, which are comprehensive and detailed descriptions of the specific steps of the workflow diagram. The method according to the present disclosure takes into account both soft prompts and hard prompts. By using the generated soft prompts and hard prompts (the hard prompts are only for some necessary steps, not for all steps) as input, the GPT is pre-trained using the graph completion task, so that the GPT can understand the basic structure or underlying structure of the workflow diagram, that is, learn the entire workflow diagram to achieve a complete description of the workflow diagram. In the fine-tuning process, the parameters of the GPT are fine-tuned, and the specific task is completed by inputting only soft prompts and part of the hard prompts to output a solution for the specific task, thereby optimizing the performance for the specific task. In the fine-tuning process, if the specific task is different, the parameters of the GPT that need to be fine-tuned are also different. Through this fine-tuning process, the execution performance of the pre-trained GPT for a specific task can be optimized, making the GPT perform tasks more effectively and providing more accurate solutions.

[0105] The method of processing workflow diagrams according to the present disclosure can have practical applications in providing effective and organized customer service in a variety of industrial scenarios. Compared with traditional methods, the method of processing workflow diagrams according to the present disclosure has significant improvements in accuracy and efficiency. Advantages of the method of processing workflow diagrams according to the present disclosure include providing improved customer service, which makes workflow tasks easier and more automated, enhances the ability to handle large and complex diagrams, and enhances the ability to effectively generate natural language output. Specifically, the method of processing workflow diagrams according to the present disclosure achieves an accuracy of 82% and reduces execution time by 30%.

[0106] The method for processing a workflow diagram according to the present disclosure can be applied to a variety of scenarios. For example, efficient and organized customer service can be provided for various industries. By encoding the information in a predefined workflow diagram into a language model, services such as after-sales service can be provided to customers through the language model. In addition, workflow diagram management can also be provided, and by encoding the workflow diagram into a graphical representation, it is easier to manage and track the progress of the workflow diagram. Further, knowledge management can also be provided, which can be used to encode and store company-specific knowledge in a text-based language model, making it easier to search and retrieve relevant information. In addition, a daily task automation service can also be provided, which automates daily tasks by encoding the information in a predefined workflow diagram into text-based language model information.

[0107] The above description is merely an example of some application scenarios of the method according to the present disclosure, and the solution can be adapted and used in various scenarios in other fields that require encoding a predefined workflow diagram into a text-based language model.

[0108] Example Device

[0109] Figure 7 A schematic block diagram of an example device 700 that can be used to implement an embodiment of the present disclosure is shown. Device 700 can be used to implement the computing device 102 of Figure 1. As shown, device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded from a storage unit 708 into a random access memory (RAM) 703. In RAM 703, various programs and data required for the operation of device 700 can also be stored. CPU 701, ROM 702 and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0110] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0111] The processing unit 701 performs the various methods and processes described above, such as any one of the processes 200, 400, 500, and 600. For example, in some embodiments, any one of the processes 200, 400, 500, and 600 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU 701, one or more steps of any one of the processes 200, 400, 500, and 600 described above may be performed. Alternatively, in other embodiments, the CPU 701 may be configured to perform any one of the processes 200, 400, 500, and 600 by any other suitable means (e.g., by means of firmware).

[0112] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0113] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the workflow diagram and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0114] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0116] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for processing a workflow diagram, comprising: Using a graph neural network to encode structural information of a workflow diagram including multiple nodes and multiple edges to obtain a vector representation of the structural information; Obtaining text description data about the workflow diagram at a node; Based on the acquired text description data and the acquired vector representation, training a language model to acquire a pre-trained language model; as well as The pre-trained language model is fine-tuned using training data of a specific task to obtain a fine-tuned language model.

2. The method according to claim 1, wherein based on the obtained text description data and the obtained vector representation, training a language model comprises: The vector representation of the structural information and the text description data of a predetermined number of nodes among the multiple nodes are used as inputs of the language model, and the language model is made to perform a graph completion task to pre-train the language model to generate a complete description of the workflow diagram.

3. The method according to claim 2, wherein the parameters of the language model are adjusted based on the difference between the real text for at least one specific node among the multiple nodes and the text for the at least one specific node generated by the language model according to the graph completion task to obtain a complete description of the workflow diagram.

4. The method according to claim 1, wherein encoding the structural information of a workflow diagram including a plurality of nodes and a plurality of edges using a graph neural network to obtain a vector representation of the structural information comprises: Using a graph convolutional network, operating a node feature vector in a graph representation of a specific node among the plurality of nodes and a graph adjacency matrix for the specific node to determine a continuous representation of the specific node, the specific node being each node among the plurality of nodes; Operating, using a graph attention network, the node feature vector in the graph representation of the specific node to determine a weighted sum of features of neighboring nodes adjacent to the specific node; as well as Based on the continuous representation and a weighted sum of the features of the neighboring nodes, a vector representation with a fixed length of structural information about the node is determined.

5. The method according to claim 1, wherein obtaining text description data about a predetermined number of nodes of the workflow diagram comprises: obtaining a graph representation of a first particular node among the plurality of nodes, wherein the first particular node comprises one of a predetermined number of nodes among the plurality of nodes; Obtaining an edge label between the first specific node and a second feature node among the multiple nodes; as well as The graph representation of the first specific node and the edge label are concatenated to determine text description data for the first specific node.

6. The method according to claim 1, wherein the text description data at the nodes of the workflow diagram includes text description data at a predetermined number of nodes, and the predetermined number of nodes correspond to a predetermined number of steps of the workflow diagram that are first executed or required to implement the specific task.

7. The method according to claim 1, wherein the workflow diagram comprises a plurality of branches, each branch comprises at least one node and at least one edge, and different branches correspond to different solutions for the specific task.

8. An electronic device comprising: at least one processor; as well as coupled to the at least one processor and having instructions stored thereon, the instructions, when executed by the at least one processor, causing the electronic device to perform actions, the actions comprising: Using a graph neural network to encode structural information of a workflow diagram including a plurality of nodes and a plurality of edges, to obtain a vector representation of the structural information; Obtaining text description data about the workflow diagram at a node; Based on the acquired text description data and the acquired vector representation, training a language model to acquire a pre-trained language model; and The pre-trained language model is fine-tuned using training data of a specific task to obtain a fine-tuned language model.

9. The electronic device according to claim 8, wherein based on the acquired text description data and the acquired vector representation, training a language model comprises: The vector representation of the structural information and the text description data of a predetermined number of nodes among the multiple nodes are used as inputs of the language model, and the language model is made to perform a graph completion task to pre-train the language model to generate a complete description of the workflow diagram.

10. An electronic device according to claim 9, wherein the parameters of the language model are adjusted based on the difference between the real text for at least one specific node among the multiple nodes and the text for the at least one specific node generated by the language model according to the task completed by the graph to obtain a complete description of the workflow diagram.

11. The electronic device according to claim 8, wherein encoding the structural information of a workflow diagram including a plurality of nodes and a plurality of edges using a graph neural network to obtain a vector representation of the structural information comprises: Using a graph convolutional network, operating a node feature vector in a graph representation of a specific node among the plurality of nodes and a graph adjacency matrix for the specific node to determine a continuous representation of the specific node, the specific node being each node among the plurality of nodes; Operating, using a graph attention network, the node feature vector in the graph representation of the specific node to determine a weighted sum of features of neighboring nodes adjacent to the specific node; as well as A vector with a fixed length of structural information about the node is determined based on the continuous representation and a weighted sum of the features of the neighboring nodes.

12. The electronic device according to claim 8, wherein acquiring text description data about a predetermined number of nodes of the workflow diagram comprises: obtaining a graph representation of a first particular node among the plurality of nodes, wherein the first particular node comprises one of a predetermined number of nodes among the plurality of nodes; Obtaining an edge label between the first specific node and a second feature node among the multiple nodes; as well as The graph representation of the first specific node and the edge label are concatenated to determine text description data for the first specific node.

13. An electronic device according to claim 8, wherein the text description data at the nodes of the workflow diagram includes text description data at a predetermined number of nodes, and the predetermined number of nodes correspond to a predetermined number of steps of the workflow diagram that are first executed or required to implement the specific task.

14. The electronic device according to claim 8, wherein the workflow diagram comprises a plurality of branches, each branch comprises at least one node and at least one edge, and different branches correspond to different solutions for the specific task.

15. A computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions that when executed cause a machine to perform actions comprising: Using a graph neural network to encode structural information of a workflow diagram including a plurality of nodes and a plurality of edges, to obtain a vector representation of the structural information; Obtaining text description data about the workflow diagram at a node; Based on the acquired text description data and the acquired vector representation, training a language model to acquire a pre-trained language model; as well as The pre-trained language model is fine-tuned using training data of a specific task to obtain a fine-tuned language model.

16. The computer program product according to claim 15, wherein based on the obtained text description data and the obtained vector representation, training a language model comprises: The vector representation of the structural information and the text description data of a predetermined number of nodes among the multiple nodes are used as inputs of the language model, and the language model is made to perform a graph completion task to pre-train the language model to generate a complete description of the workflow diagram.

17. A computer program product according to claim 16, wherein the parameters of the language model are adjusted based on the difference between the real text for at least one specific node among the multiple nodes and the text for the at least one specific node generated by the language model according to the graph completion task to obtain a complete description of the workflow diagram.

18. The computer program product according to claim 15, wherein encoding the structural information of a workflow diagram including a plurality of nodes and a plurality of edges using a graph neural network to obtain a vector representation of the structural information comprises: Using a graph convolutional network, operating a node feature vector in a graph representation of a specific node among the plurality of nodes and a graph adjacency matrix for the specific node to determine a continuous representation of the specific node, the specific node being each node among the plurality of nodes; Operating, using a graph attention network, the node feature vector in the graph representation of the specific node to determine a weighted sum of features of neighboring nodes adjacent to the specific node; as well as A vector with a fixed length of structural information about the node is determined based on the continuous representation and a weighted sum of the features of the neighboring nodes.

19. The computer program product of claim 15, wherein obtaining text description data about a predetermined number of nodes of the workflow diagram comprises: obtaining a graph representation of a first particular node among the plurality of nodes, wherein the first particular node comprises one of a predetermined number of nodes among the plurality of nodes; Obtaining an edge label between the first specific node and a second feature node among the multiple nodes; as well as The graph representation of the first specific node and the edge label are concatenated to determine text description data for the first specific node.

20. The computer program product of claim 15, wherein the text description data at the nodes of the workflow diagram includes text description data at a predetermined number of nodes, the predetermined number of nodes corresponding to a predetermined number of steps of the workflow diagram that are first executed or required to achieve the specific task.

Images