An intelligent assistance system for low-code development of machine learning applications

Abstract

The present invention realizes an intelligent assistance system for low-code development of machine learning applications through methods in the field of artificial intelligence. It includes two parts: a low-code development template recommendation method and a low-code development component recommendation method; the low-code development template recommendation method calculates the similarity between the workflow diagram and the text description through a neural network to recommend a low-code development template to the developer; the low-code development component recommendation method combines the TransRec model for common components and the MetaRec model for cold start components to provide developers with stable and accurate component recommendation services. The intelligent assistance system provided by the present invention can improve the development efficiency of machine learning applications developed using a low-code method, lower the development threshold, and allow public developers to use the low-code development platform to quickly develop standard and effective machine learning applications.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent auxiliary system for low-code development of machine learning applications. Background Art

[0002] As an important technology to promote the development of artificial intelligence, machine learning has been successfully applied to image recognition, machine translation and other fields. However, the development of machine learning application systems involves complex processes such as model design, training and deployment, and is highly dependent on the personal experience and manual customization of developers, making it difficult to meet the actual needs of various fields for the rapid construction of large-scale artificial intelligence applications. Therefore, improving the development efficiency of machine learning application systems has become an important issue.

[0003] In response to the above problems, the industry and academia have conducted a lot of exploration in the fields of domain-specific languages ​​(DSL), automatic machine learning (AutoML), and low-code development. In particular, the low-code development method has received widespread attention. Its basic idea is to encapsulate the code into reusable components and quickly implement application development through a visual development interface, thereby reducing the development difficulty of machine learning applications and improving development efficiency. However, the existing low-code development environment lacks support for recommendation systems, resulting in developers using low-code development environments for application development still facing the problems of high learning barriers and low development efficiency.

[0004] At present, there are some research works trying to solve the efficiency problems faced by developing machine learning applications using low-code methods. In order to improve development efficiency, existing research uses template recommendation technology to solve the problem. In order to further lower the development threshold and allow public developers, that is, developers with little experience, to quickly develop machine learning applications, existing research uses AutoML and recommendation technology to solve the problem.

[0005] Although existing methods have made some progress in helping developers quickly develop machine learning applications, existing recommendation technologies are mostly aimed at traditional development methods and are not suitable for low-code development methods. In addition, in terms of lowering the development threshold, existing technologies can only provide auxiliary support for specific machine learning applications or are not suitable for the development of machine learning applications, which has certain limitations.

[0006] In response to the problems of low development efficiency and high development threshold of machine learning applications in the existing low-code development environment for machine learning applications, the present invention aims to propose an intelligent auxiliary system for low-code development of machine learning applications, improve the development efficiency of machine learning applications using a low-code approach, and lower the development threshold.

[0007] Specifically, the main problems to be solved are: to improve development efficiency, design and implement development template recommendation technology; to lower the development threshold so that public developers can also quickly develop standard and effective machine learning applications, design and implement development component recommendation technology. Summary of the invention

[0008] To this end, the present invention first proposes an intelligent auxiliary system for low-code development of machine learning applications, the system comprising an input module, an output module and an operation unit;

[0009] The input module takes a partially completed process as input and processes it through the computing unit in two development methods: one is to use a process template and apply the recommended method of low-code development templates; the other is to develop a new machine learning application process from scratch, which corresponds to the recommended method of low-code development components. The two development methods respectively assist in forming a complete low-code development system;

[0010] The low-code development template recommendation method takes two parts as input: one part is the process information, workflow diagram and text information of the recommended process template; the other part is the text description of the new project, and the similarity of the two parts of input is calculated through a neural network;

[0011] The low-code development component recommendation method includes two parts: a Transformer-based TransRec model for recommending common components, and a meta-learning-based MetaRec model for recommending components with few historical interactions (i.e., cold start components). The scores of components that appear in a large number of samples are given by TransRec, and the scores of components that appear only in a few samples are given by MetaRec. The two are combined to obtain a series of prediction results;

[0012] The output module sorts the prediction results according to similarity, and recommends the top five historical projects or five components to the developer.

[0013] The similarity calculation is completed through the WNN model, which consists of three parts, namely, the Workflow Embedding neural network for embedding workflows, the Text Embedding neural network for embedding natural language descriptions, and the Similarity module for measuring similarity. The workflow process information of the process template and the description text corresponding to the project are extracted from the historical projects to form a training set. The WNN model is trained so that it can encode the two process information and description texts, and encode similar information in the same area. In the deployment stage, when a new project is created, the most matching project is found from the historical projects according to the task description for developers to learn from or reuse. The deployment stage uses both the workflow information and the process description information of the process. The two parts use WNN to obtain vectors respectively, and then the final score is obtained by calculating the Cosine similarity between the two.

[0014] The Workflow Embedding neural network is used to convert workflow information in low-code applications into a vector neural network. The input of a process information is A=[T, Graph(V, E), D], where is the character sequence after the process name and label are concatenated, k, k+m=n, which are the character lengths of the label and name respectively. Label is the keyword label that the developer adds to the process. Graph(V, E) is the workflow graph information. Each workflow graph is a directed graph. V represents the node set and E represents the edge set. Each node v∈V has its own ID attribute; each edge e∈E has two attributes: the start point and the end point. D is the natural language description information of the process.

[0015] First, use the neural network to embed each part, and use BiLSTM to embed T

[0016] e t =Embedding(w t )

[0017] f t =σ(W f ·[h t-1 , e t ]+b f )

[0018] i t =σ(W i ·[h t-1 , e t ]+b i )

[0019]

[0020]

[0021] o t =σ(W o ·[h t-1 , w t ]+b o )

[0022] h t =o t *tanh(C t )

[0023] in Yes t The embedding vector, w t is the tth character in T, represents the concatenation of two vectors, is the matrix of trainable parameters in BiLSTM, tanh and σ are the activation functions of BiLSTM, and σ represents the Sigmoid function. Finally, T is embedded as an n×d dimensional vector:

[0024] h=[h1,...,h n ]=BiLSTM(T)

[0025] For the Graph (V, E) part, since it is format feature information, the GAT network is used to embed the Graph (V, E). The GAT calculation formula includes the information construction stage and the domain aggregation stage. The information construction stage constructs the input representation for the GAT network of each layer. The specific formula is:

[0026]

[0027]

[0028] in Indicates the representation of the network at the beginning. In the WENN network, the node ID is used as the input of the graph and is embedded in the network through the ID Embedding network. Convert it into a d-dimensional vector;

[0029] The domain aggregation stage aggregates the information to be transmitted by each domain node. The domain represents all nodes adjacent to the current node. The specific formula for neighborhood aggregation is as follows:

[0030]

[0031]

[0032] Where Nu is the neighborhood set of node u,

[0033] Then the two features are fused through the Cross Attention module and the two features are concatenated:

[0034] w ij =(W[h j , f i ])

[0035]

[0036]

[0037]

[0038]

[0039]

[0040] o=[o h , o f ]

[0041] in, is a trainable parameter. Finally, the two vectors are concatenated and fused, and the output vector o is the final vector representation of the process information.

[0042] The Text Embedding neural network for embedding natural language description is a neural network for converting natural language description information into vectors. Assume that the input of a natural language description information is D=w1,...,w m , is a text sequence of length m. BiLSTM is used to embed the text sequence D, and the maximum pooling method maxpool is used to transform its dimension.

[0043] d = maxpool(BiLSTM(D)).

[0044] The Similarity module calculates the similarity between the historical project workflow embedding vector obtained by the Workflow Embedding neural network for embedding workflows and the Text Embedding neural network for embedding natural language descriptions and the project text description embedding vector encoded from the bidirectional RNN network, and uses Cosine similarity to measure the similarity between the two vectors:

[0045]

[0046] Where o and d are the vectors of workflow and text description respectively, and ||d|| represents the norm of vector d.

[0047] The TransRec model of the low-code development component recommendation method consists of three parts, namely, the Embedding layer, the Transformer layer and the prediction layer. The Embedding layer is responsible for embedding the original data into the corresponding vector representation, and the Transformer layer is responsible for modeling the correlation between sequences. It mines the correlation between the two without considering the distance between the sequence pairs. This modeling method is superior to the RNN-based modeling method when modeling time series data. Finally, the final result is obtained through the prediction layer.

[0048] The Embedding layer first needs to preprocess the input and convert the behavior sequence <S1,S2,...,S |S| >Convert to a fixed length sequence S= <S1,S2,...,S n >, where n is the maximum length that the model can handle. If the length of the sequence is greater than n, the most recent n actions are taken as input. If the length of the sequence is less than n, the sequence is completed, that is, the front end of the sequence is added. <padding>, until the sequence length is equal to n, create the component embedding matrix Where d is the dimension of the latent vector, |I| is the number of components, <padding>The corresponding vector is set to a full 0 vector;

[0049] Use Position Embedding to integrate sequence information into the input vector by embedding:

[0050]

[0051] where e i ∈E is an S with dimension d i Embedded, p i ∈P is the index embedding corresponding to position i of dimension d, g(i) is the decomposition of the component's Bigram vector using the matrix decomposition method, using the learnable position embedding Embedding, the position embedding matrix

[0052] The Transformer layer will Stacked together to form a matrix It contains two sublayers, a multi-head self-attention sublayer and a Position-wise feedforward network; the multi-head self-attention mechanism captures the dependencies between representation pairs regardless of their distance in the sequence. First, H is transformed into l Linearly project to the subspace, and then apply the Attention function to generate the re-projected vector representation in parallel:

[0053]

[0054] The projection matrix of each head and is a learnable network parameter. The formula of the Scaled Dot-Product attention mechanism is as follows:

[0055]

[0056] Where Q, K, V are H l The transformation is obtained by generating the re-projected vector representation in parallel through the Attention function. This is to avoid extremely small gradients and produce a softer attention distribution;

[0057] The Position-wise feed-forward network transforms its result, which consists of two affine transformations and uses LeakyReLU as the activation function:

[0058]

[0059] FFN(x)=LeayReLU(W 1 x+b 1 )W 2 +b 2

[0060] in, It is a learnable network layer parameter and is shared by all positions in the same layer, but not between layers. At the same time, in order to increase the expressive power of the model, Transformer stacks multiple layers of self-attention mechanisms to form a self-attention layer.

[0061] H l =Transformer(H l-1 )

[0062] Transformer(H l-1 )=LN(A l-1 +Dropout(PFFN(A l-1 )))

[0063] A l-1 =LN(H l-1 +Dropout(MH(H l-1 )))

[0064] Use residual connection between two sub-layers, then perform layer normalization (LN), and apply dropout to the output of each sub-layer, which is then LNed again to stabilize and speed up network training;

[0065] The prediction layer masks the output term o at time step t t , then according to Prediction shielding item t , a two-layer feedforward network is used with ReLU as the activation function in the middle to obtain the probability distribution of the final target component:

[0066]

[0067] Where W P , b P , b O are trainable network parameters, Embedding matrix for components. Use shared item embedding matrix in input and output layers to reduce overfitting and model size.

[0068] In the model training process, define O = {o1, ..., o n } is the true label of the output sequence, S = {S1, ..., S n } is the input label of the behavior sequence:

[0069]

[0070] Where t∈[1,...,n] represents the position of the label in the sequence. Using negative sampling technology, a negative example is prepared for each positive example, and Binary Cross Entropy is used as the loss function to train the network:

[0071]

[0072] Among them, n is the sequence length, S is the input sequence, and o t For the correct label, is a randomly selected negative example label. During the calculation process, the model will ignore o t = <padding>The loss generated by the position and use Adam as the optimizer.

[0073] The MetaRec model input is {S1, ..., S n-1 }, the output is S n The cold start problem refers to the situation where there are only a few training examples for a component. To simulate the cold start scenario, we assume that in the current task, any component only appears in K sequences, K∈[1,3,5]; the MetaRec model includes a data preprocessing process. First, we define I as the set of all components, which is divided into I according to the ratio of 7:1:2. train , I valid , I test , and there is no intersection between them; following the standard meta-learning setting, according to a set of training tasks Train MetaRec model for any task I train Extract N components as the basis, build Support Set and Query Set for these components, Support Set contains K sequences containing the same ending components, the sequence length is not less than 3, Query Set = {Q1, ..., Q N }, Q i represents a sequence ending with the i-th component and the length of the sequence is not less than 3, Q i The end component of is removed as a label. Such a pair of Support Set and Query Set constitutes a task T i Input;

[0074] The MetaRec model first uses Sequence gncoder to encode the input into a representation vector, and then uses Attention Block to aggregate K sequences in Support Set to generate Support Set vector representation. Finally, the matching network MatchNet is used to match the fused Support Set latent vector representation with the Query Set latent vector representation to obtain the prediction result.

[0075] The Sequence Encoder is specifically: The task of the sequence encoder is to encode each behavior sequence into a separate vector representation, that is, Set i ={S i，1 , S i，2 , ..., S i，l }, where S i，1 , S i，2 , ..., S i，l For different components, l is Set i The length of the sequence is , this step performs the previous padding operation on the sequence to unify the length under the same input. The specific formula of the sequence encoder is as follows:

[0076] h i =f(Set i )

[0077]

[0078]

[0079] e j =W p (W last g(S i，l-1 )+W cur g(S i，j )+W αvg S i，avg +b)

[0080]

[0081] in, b are all trainable network parameters, g is the Embedding layer, which converts the components into their corresponding vector representations, and then h i With g(S i，l ) are concatenated and the final output is obtained through an m-layer fully connected network. In order to make training easier, the model adds skip connections. and Add together to get the final vector representation

[0082] The Attention Block is specifically as follows: The task of the attention module is to fuse the h corresponding to the K behavior sequences. In order to make the final latent vector h contain more key features related to the Query Set, the following formula is used for fusion:

[0083]

[0084] Where W Q , W K , W V is a trainable network parameter, q is the vector representation of the Query Set after Sequence Encoder, and h is the representation of the Support Set after encoding;

[0085] The MatchNet is specifically: the main task of matching is to obtain the The final probability distribution is obtained by matching with q, and MatchNet is used as the final matching layer. The specific formula is as follows:

[0086]

[0087] Score=σ(W2(ReLU(W1F+b1))+b2)

[0088] in is a trainable network parameter, σ is a Sigmoid function, and the model is trained using the Meta-Train process commonly used in meta-learning.

[0089] The technical effects to be achieved by the present invention are:

[0090] Improve development efficiency, design and implement development template recommendation technology;

[0091] Lower the development threshold so that public developers can quickly develop standard and effective machine learning applications, and design and implement development component recommendation technology BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 Overall framework diagram;

[0093] Figure 2 Workflow Neural Network;

[0094] Figure 3 Recommendation of low-code development templates based on joint embedding;

[0095] Figure 4 Component recommendation based on meta-learning enhancement;

[0096] Figure 5 TransRec model;

[0097] Figure 6 MetaRec model structure;

[0098] Figure 7 List of machine learning applications for application scenarios of the embodiment;

[0099] Figure 8 Project description information;

[0100] Fig. 9 Process Information Example;

[0101] Fig.10 Example of recommended results;

[0102] Fig.11 This application system component library;

[0103] Fig.12 Task input;

[0104] Fig.13 Task output; DETAILED DESCRIPTION

[0105] The following is a preferred embodiment of the present invention and combined with the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to this embodiment.

[0106] The present invention proposes an intelligent auxiliary system for low-code development of machine learning applications.

[0107] 1. Overall structure:

[0108] The present invention proposes an intelligent auxiliary system for low-code development of machine learning applications based on the development template recommendation and development component recommendation steps. The overall framework is as follows: Figure 1 shown.

[0109] The intelligent auxiliary system for low-code development of machine learning applications uses recommendation technology to lower the development threshold of machine learning applications. This approach has been widely studied and discussed in academia. Auxiliary development functions such as recommendation technology are crucial. The first recommendation technology that needs to be studied is low-code development template recommendation. The development template recommendation method can match similar historical tasks based on the developer's description of the task, and recommend the development results of similar historical tasks to the developer as development templates. The low-code development template recommendation method can provide developers with a good development starting point, allowing developers to reuse part or all of the development results of historical tasks, and conduct secondary development on this basis, thereby improving development efficiency. In addition to providing developers with a good development starting point, providing developers with suggestions during the development process is also the key to improving development efficiency and quality. Therefore, a component recommendation method that recommends possible component selections to developers based on the current application development context (i.e., the built process) is also indispensable.

[0110] The low-code development template recommendation method and the low-code development component recommendation method proposed in the present invention are introduced in detail below.

[0111] 2. Recommendation of low-code development template based on joint embedding:

[0112] The development template recommendation task for low-code development has two input parts: one is the process information, workflow diagram and text information (project label, name, etc.) of the recommended process template; the other is the text description of the new project. The similarity between the two is calculated by neural network. The present invention designs and implements Workflow NeuralNetwork to complete the similarity calculation.

[0113] Figure 2 The overall structure of the WNN network is shown in Figure 1. The WNN model consists of three parts: the Workflow Embedding function for embedding workflow diagram information, the Text Embedding function for embedding natural language description information, and the Similarity function for evaluating similarity. Because deep neural networks perform very well in feature engineering, this article uses deep neural networks to implement two Embedding functions.

[0114] In summary, the WNN model consists of three modules: the Workflow Embedding Neural Network (WENN) for embedding workflows, the Text Embedding Neural Network (TENN) for embedding natural language descriptions, and the Similarity module for measuring similarity.

[0115] (1) WENN module

[0116] WENN is a neural network used to convert workflow information in low-code applications into vectors. Workflow information is not simple text information, it contains component information, component call order, labels, and names. These features are directly helpful for recommendations.

[0117] Assume that the input of a process information is A = [T, Graph (V, E), D], where is the character sequence after the process name and label are concatenated, k, k+m=n, respectively, are the length of the label and name. The label is the keyword label that the developer adds to the process, such as machine learning, classification tasks, etc. Graph(V, E) is the workflow graph information. Each workflow graph is a directed graph. V represents the node set and E represents the edge set. Each node v∈V has its own ID attribute; each edge e∈E has two attributes: the start point and the end point. D is the natural language description information of the process.

[0118] D is the natural language description of the process. In order to embed this information into Embedding, the WENN module first uses a neural network to embed each part. WENN uses BiLSTM to embed T

[0119] e t =Embedding(w t ) (1)

[0120] f t =σ(W f ·[h t-1 , e t ]+b f ) (2)

[0121] i t =σ(W i ·[h t-1 , e t ]+b i ) (3)

[0122]

[0123]

[0124] o t =σ(W o ·[h t-1 , w t ]+b o ) (6)

[0125] h t =o t *tanh(C t ) (7)

[0126] in Yes t The embedding vector, w t is the tth character in T, represents the concatenation of two vectors, is the matrix of trainable parameters in BiLSTM, tanh and σ are the activation functions of BiLSTM, and σ represents the Sigmoid function. Finally, T is embedded as an n×d dimensional vector:

[0127] h=[h1,...,h n ]=BiLSTM(T)

[0128] Similarly, for the Graph (V, E) part, since it is format feature information, the GAT network that is often used to process graph data is used here to embed the Graph (V, E).

[0129] The formula for GAT calculation mainly includes two stages: information construction and domain convergence. The information construction stage mainly constructs input representation for each layer of the GAT network. The specific formula is as follows:

[0130]

[0131]

[0132] in Indicates the representation of the network at the beginning. In the WENN network, the node ID is used as the input of the graph and is embedded in the network through the ID Embedding network. Convert it into a d-dimensional vector. Formula (10) indicates that the input of other layers is the output of the GAT network of the previous layer.

[0133] The domain aggregation stage aggregates the information to be transmitted by each domain node. The domain represents all nodes adjacent to the current node. The specific formula for neighborhood aggregation is as follows:

[0134]

[0135]

[0136] Where N u is the neighborhood set of node u, is a learnable network parameter. The GAT network calculates the attention score based on the information gathered from the neighborhood according to formula 12, and performs weighted summation to obtain the final result. Here, the number of layers of GAT is 2.

[0137] After obtaining the representation of the two types of information, because the two types of information do not exist independently, there is an implicit relationship between them. In order to mine the feature information of the relationship between the two features, the two parts of the features are fused through the Cross Attention module and the obtained two parts of the features are spliced:

[0138] w ij =(W[h j , f i ]) (13)

[0139]

[0140]

[0141]

[0142]

[0143]

[0144] o=[o h , o f ] (19)

[0145] in, is a trainable parameter. Finally, the two vectors are concatenated and fused, and the output vector o is the final vector representation of the process information.

[0146] (2)TENN module

[0147] TENN is a neural network used to convert natural language description information into vectors. Assume that the input of a natural language description information is D = w1, ..., w m , is a text sequence of length m. Similar to WENN, BiLSTM is used to embed the text sequence D, and the maximum pooling method maxpool is used to transform its dimension.

[0148] d = maxpool(BiLSTM(D)) (20)

[0149] (3)Similarity module

[0150] The vector obtained above is used to calculate the similarity with the project description encoded from the bidirectional RNN network. The above two sections have described the transformation of mapping process information and description to vectors (i.e., o and d). Since we want the vectors of process information and description to be jointly embedded, we can use Cosine similarity to measure the similarity between the two vectors:

[0151]

[0152] Where o and d are the vectors of process information and text description, respectively, and ||d|| represents the norm of vector d. The higher the similarity, the greater the correlation between the two. In summary, WNN<workflow,description> Take pairs as input and calculate their cosine similarity cos(o, d).

[0153] (4) Model training

[0154] During the training phase, process information and text descriptions are separated from historical projects, and the network is trained using the MSE loss function as the target. After deployment, when a new project is built, its task description information is used to calculate the similarity with the process template of the project in the library, and recommendations are made based on the similarity, so as to obtain the historical project that is most similar to the current task description for developers to refer to and reuse.

[0155] (5) Overall process

[0156] The recommended process for overall template development is as follows: Figure 3 As shown in the figure, the process information, workflow diagram (workflow) and corresponding description text of the process template are extracted from the historical projects to form a training set, and the Workflow Neural Network is trained to enable it to encode the process information and description text, and encode similar information in the same area. In the deployment phase, when a new project is created, the most matching project is found from the historical projects according to the task description for developers to learn from or reuse. The deployment phase uses both the workflow diagram information and the process description information of the process. The two parts use WNN to obtain vectors respectively, and then the final score is obtained by calculating the Cosine similarity between the two.

[0157] 3. Component recommendation based on meta-learning enhancement:

[0158] The component recommendation method based on meta-learning consists of two parts: the Transformer-based TransRec model for recommending common components and the meta-learning-based MetaRec model for recommending components with few historical interactions, i.e., cold start components. The relationship between the two is as follows: Figure 4 shown.

[0159] When a recommendation query arrives, both TransRec and MetaRec models are used for prediction. Since MetaRec performs better in cold start scenarios but worse than TransRec in normal scenarios, the scores of non-cold start components in the prediction results, that is, components that appear in a large number of samples, are given by TransRec, while the scores of cold start components, that is, components that appear in only a few samples, are given by MetaRec. The two are combined to obtain the final prediction result.

[0160] Here we first introduce the technology related to component recommendation. In the component recommendation task, the dataset will give the developer's behavior sequence S= <S1,S2,...,S n >, the model needs to predict the next component that the developer will call. During the training phase, the Transformer model predicts the t+1th component at time step t based on the first t components. The input of the model is <S1,S2,...,S n-1 >, and requires the output to be the behavior sequence after the offset <S2,S3,...,S n >.

[0161] Transformer-based component recommendation model:

[0162] The TransRec model consists of three parts: the Embedding layer, the Transformer layer, and the prediction layer. The Embedding layer is responsible for embedding the original data into the corresponding vector representation, and the Transformer layer is responsible for modeling the correlation between sequences. In particular, the Attention mechanism can mine the correlation between the two sequences without considering the distance between the sequence pairs. This modeling method is better than the RNN-based modeling method when modeling time series data. Finally, the final result is obtained through the prediction layer. The following is a detailed introduction to these three parts.

[0163] (1) Embedding layer

[0164] The Embedding layer first needs to preprocess the input and convert the behavior sequence <S1,S2,...,S |S| >Convert to a fixed length sequence S= <S1,S2,...,S n >, where n is the maximum length that the model can handle. If the length of the sequence is greater than n, the most recent n actions are taken as input. If the length of the sequence is less than n, the sequence is completed, that is, the front end of the sequence is added <padding>, until the sequence length is equal to n. Create component embedding matrix Where d is the dimension of the latent vector and |I| is the number of components. <padding>The corresponding vector is set to an all-zero vector.

[0165] Since the Transformer model does not contain sequence order information and cannot perceive the order information of components, Position Embedding is used here to embed the sequence information into the input vector:

[0166]

[0167] where e i ∈E is an S with dimension d i Embedded, p i ∈P is the index embedding corresponding to position i of dimension d. g(i) is the decomposition of the component's Bigram vector using matrix decomposition method. In the present invention, learnable position embedding is used, which can achieve better performance. Position embedding matrix

[0168] (2) Transformer layer

[0169] like Figure 5 As shown in Figure 2, we apply the Transformer module to the component recommendation task. Here, we Stacked together to form a matrix This helps us calculate the attention function at all positions simultaneously in practice. The Transformer module consists of two sublayers, a multi-head self-attention sublayer and a Position-wise feedforward network.

[0170] Multi-head self-attention mechanism: The attention mechanism is able to capture the dependencies between representation pairs without considering their distance in the sequence. Studies have shown that capturing information from different representation subspaces from different perspectives is beneficial for the final prediction. Therefore, this paper adopts a multi-head self-attention mechanism here instead of implementing a single-head attention mechanism. Specifically, H is first transformed into l Linearly project to the subspace, and then apply the Attention function to generate the re-projected vector representation in parallel:

[0171]

[0172] The projection matrix of each head and is a learnable network parameter. The formula of Scaled Dot-Product attention mechanism is as follows:

[0173]

[0174] Where Q, K, V are H l After transformation as in formula (23), The purpose is to avoid extremely small gradients and produce a softer attention distribution.

[0175] Position-wise Feed-Forward network layer: After getting the output of the self-attention mechanism, although it can represent the component sequence as a single fusion vector, it is still essentially just a linear change layer. In order to make the model nonlinear and have interactive information between different dimensions, the Transformer network uses the Position-wise Feed-Forward network to transform its results, which consists of two affine transformations and uses LeakyReLU as the activation function:

[0176]

[0177] FFN(x)=LeayReLU(W 1 x+b 1 )W 2 +b 2 (26)

[0178] in, It is a learnable network layer parameter and is shared by all positions in the same layer, but not between layers. At the same time, in order to increase the expressive power of the model, Transformer stacks multiple layers of self-attention mechanisms to form a self-attention layer.

[0179] H l = Transformer(H l-1 ) (27)

[0180] Transformer(H l-1 )=LN(A l-1 +Dropout(PFFN(A l-1 ))) (28)

[0181] A l-1 =LN(H l-1 +Dropout(MH(H l-1 ))) (29)

[0182] In order to make the training smoother, the invention uses residual connection between two sub-layers and then performs layer normalization (LN). In addition, we also apply dropout to the output of each sub-layer and then perform LN on it to stabilize and accelerate network training. Based on the actual data situation, a 2-layer 2-head Transformer is used here.

[0183] (3) Prediction layer

[0184] After the Transformer network layer, the final output vector H L It already contains all the information of the historical behavior sequence. Suppose we mask the output item o at time step t t , then according to Prediction shielding item t Specifically, we use a two-layer feed-forward network with ReLU as the activation function in the middle to obtain the probability distribution of the final target component:

[0185]

[0186] Where W P , b P , b O are trainable network parameters, is the embedding matrix of the component. We use shared item embedding matrices in the input and output layers to reduce overfitting and model size.

[0187] (4) Model training

[0188] In the sequence recommendation task, the next item of each component is usually predicted as a training task. Therefore, during the training process, we define O = {o1, ..., o n } is the true label of the output sequence, S = {S1, ..., S n } is the input label of the behavior sequence:

[0189]

[0190] Where t∈[1, ..., n] represents the position of the label in the sequence. In order to make the model more accurate in prediction, we hope that the model always scores the components that are more likely to score higher than other components. Therefore, the present invention uses negative sampling technology to prepare a negative example for each positive example, and uses Binary Cross Entropy as the loss function to train the network:

[0191]

[0192] Among them, n is the sequence length, S is the input sequence, and o t For the correct label, is a randomly selected negative example label. During the calculation process, the model will ignore o t = <padding>The loss generated by the position is calculated and Adam is used as the optimizer. Compared with SGD, Adam sets a sparse learning rate that decreases as the number of iterations increases, making model training smoother.

[0193] Recommendation model based on meta-learning enhancement:

[0194] TransRec uses deep learning to improve the accuracy of recommendations, but it also inherits the requirements of deep learning for data volume. However, due to the rapid development and iteration of machine learning technology, low-code component libraries often need to expand new algorithm components to meet new development needs. There is little historical usage data related to new components, which leads to the cold start problem in component recommendation models. Some research has now emerged to solve the cold start problem, such as using auxiliary information to help cold start components complete recommendations. However, since there is no auxiliary information for low-code sequence recommendation tasks and only the sequence information in the process is available, these methods are not suitable for component recommendation. Meta-learning is regarded as a method to solve the cold start problem in many scenarios. Unlike traditional methods based on Learning to Rank, meta-learning completes the model update through subtasks, each subtask is a recommendation task, and the candidate set of the recommendation task is randomly selected from the overall candidate set. This application draws on the idea of ​​meta-learning to convert the recommendation task into a clustering task, classify the query through the clustering model, and the final classification label is the component to be recommended. The recommendation model needs to find implicit knowledge information from a large amount of historical data to complete the recommendation, while the clustering model only needs to determine which category in the query and candidate set is closer to give a classification result. This method reduces the model's dependence on data and can alleviate the cold start problem in the absence of auxiliary information. However, since the existing model does not model the interaction between the query and the candidate set, the model cannot fully utilize the information provided by the candidate set. Therefore, the present invention proposes a cold start sequential recommendation method (MetaRec) based on meta-learning to alleviate the cold start problem of items in sequential recommendation. At the same time, MetaRec can effectively extract user preferences from limited interactions and learn to match target cold start items with potential users. In addition, the MetaRec framework can be easily integrated with the TransRec mentioned above.

[0195] (1) Problem definition

[0196] Recommend the next component for the input sequence {S1, ..., S n-1 }, the output is S n The cold start problem refers to the fact that there are only a few training examples for a component. Therefore, to simulate the cold start scenario, assume that in the current task, any component only appears in K sequences, K∈[1, 3, 5]. To ensure the fairness of the results, the components in the test set do not appear in the training set.

[0197] (2) Data preprocessing

[0198] Since MetaRec combines meta-learning technology to solve the cold start problem, it has different prediction methods and also handles the data set differently. First, define I as the set of all components, which are divided into I in a ratio of 7:1:2. train , I valid , I test , and there is no intersection between them. Following the standard meta-learning setting, according to a set of training tasks Train MetaRec model for any task I train Extract N components as the basis, and build Support Set and Query Set for these components. Support Set is shown in Table 1, which consists of K sequences containing the same ending components, and the sequence length is not less than 3. Similarly, Query Set = {Q1, ..., Q N }, Q i represents a sequence ending with the i-th component and the length of the sequence is not less than 3. In order to ensure that no information is leaked, Q i The end component of is removed as a label, as shown in the example given in Table 1. Such a pair of Support Set and Query Set constitutes a task T i The role of Support Set and Query Set in the task is to provide information for classification. Since Mecos transforms the recommendation task into a classification task, the candidate set is all components. In order to ensure the accuracy of classification, it is necessary to extract information from historical data and build a portrait for the component. Support Set i is the data used to build the portrait of the i-th component. Query Set is the data used to build the query vector.

[0199] Table 1 Support Set and Query Set Examples

[0200]

[0201] (3) Model construction

[0202] In this section, we will introduce MetaRec in detail. The overall structure is as follows Figure 6 MetaRec first uses Sequence Encoder to encode the input into a representation vector, and then uses Attention Block to aggregate K sequences in Support Set to generate Support Set vector representation. Finally, MatchNet is used to match the fused Support Set latent vector representation with the Query Set latent vector representation to obtain the prediction result.

[0203] Sequence Encoder: The task of the sequence encoder is to encode each behavior sequence into a separate vector representation, that is, Set i ={S i，1 , S i，2 , ..., S i，l }, where S i，1 , S i，2 , ..., S i，l For different components, l is Set i In order to ensure that the input meets the requirements of the network, the previous padding operation is performed to unify the length of the same input. The specific formula is as follows:

[0204] h i =f(Set i ) (33)

[0205]

[0206]

[0207] e j =W p (W last g(S i，l-1 )+W cur g(S i，j )+W avg S i，avg +b) (36)

[0208]

[0209] in, b are trainable network parameters, g is the Embedding layer, which converts the components into their corresponding vector representations. Then h i With g(S i，l ) are concatenated and the final output is obtained through an m-layer fully connected network. In order to make training easier, the model adds skip connections. and Add together to get the final vector representation

[0210] Attention Block: The task of the attention module is to fuse the h corresponding to the K behavior sequences. In order to make the final latent vector h contain more key features related to the Query Set, the following formula is used for fusion:

[0211]

[0212] The Attention formula is shown in formula (24), where W Q , W K , W V is a trainable network parameter, q is the vector representation of the Query Set after SequenceEncoder, and h is the representation of the Support Set after encoding.

[0213] MatchNet: The main task of matching is to obtain the The final probability distribution is obtained by matching with q, and MatchNet is used as the final matching layer. The specific formula is as follows:

[0214]

[0215] Score=σ(W2(ReLU(W1F+b1))+b2) (40)

[0216] in is a trainable network parameter, σ is a Sigmoid function, and the model is trained using the Meta-Train process commonly used in meta-learning.

[0217] The actual application scenario of this solution is a low-code development system for machine learning applications. Figure 7-9 As shown; the development template recommendation part uses the description information of these projects and the process information developed by developers as input during training, and outputs the similarity between the two parts (a probability value of [0, 1]). The training goal is to allow the network to maximize the similarity between the description information and process information belonging to the same project, while minimizing other confusing examples. The input involved in actual application is the description information and process information of these historical projects (Part 1), and the description information of the new project (Part 2). The output result is the similarity between Part 1 and Part 2. The higher the similarity, the greater the correlation between the new project and the old project, and the more likely it is to be used as a development template by the new project. Finally, we will sort these similarities and recommend the top 5 historical projects to developers. The recommendation results are in Fig.10 Through the recommendation list, users can directly click to jump to the corresponding historical item and copy it.

[0218] Component recommendation refers to the engineering problem of recommending the node type that a developer should choose next based on the process nodes he has already developed when he is developing.

[0219] In other words, this application is a machine learning low-code development system designed based on such engineering problems. The purpose of low-code is to help developers complete application development by writing little or no code. The specific method is to complete the development by dragging the preset components of the system and then connecting the components to design the logic. The input of the task is Fig.12 When the developer is halfway through developing the process, the system will give the most appropriate node type from the 26 types of nodes and Fig.13 Therefore, the system of the present application has a component library of 26 components, such as Fig.11 shown.

[0220] This task also uses historical projects during the training process. We split the historical project dataset. For example, the original process is start node, missing value filling, dataset segmentation, CNN, end node. We will expand the process into the following set: Example 1: Input is [start node, missing value filling, dataset segmentation], output is CNN; Example 2: Input is [start node, missing value filling, dataset segmentation, CNN], output is end node; Example 3: Input is [start node, missing value filling, dataset segmentation, CNN, end node], output is 0. So the input of this task is a node sequence, and the output is a node type.< / padding> < / padding> < / padding> < / padding> < / padding> < / padding>

Claims

1. An intelligent assistance system for low-code development of machine learning applications, characterized by: The system includes an input module, an output module and an operation unit; The input module takes a partially completed process as input and processes it through the computing unit in two development methods: one is to use a process template and apply the recommended method of low-code development templates; the other is to develop a new machine learning application process from scratch, which corresponds to the recommended method of low-code development components. The two development methods respectively assist in forming a complete low-code development system; The low-code development template recommendation method inputs two parts of content, one part is the process information, workflow diagram and text information of the recommended process template; the other part is the text description of the new project, and the similarity of the two parts of input is calculated through a neural network; The low-code development component recommendation method takes two parts as input: a Transformer-based TransRec model for recommending common components, and a meta-learning-based MetaRec model for recommending components with few historical interactions. The scores of components that appear in a large number of samples are given by TransRec, and the scores of components that appear only in a few samples are given by MetaRec. The two are combined to obtain a series of prediction results. The output module sorts the prediction results according to similarity, and recommends the top five historical projects or five components to the developer.

2. An intelligent assistance system for low-code development of machine learning applications as claimed in claim 1, characterized in that: The similarity calculation is completed through the WNN model, which consists of three parts, namely, the Workflow Embedding neural network for embedding workflows, the Text Embedding neural network for embedding natural language descriptions, and the Similarity module for measuring similarity. The workflow diagram process information of the process template and the description text corresponding to the project are extracted from the historical projects to form a training set. The WNN model is trained to enable it to encode the process information and the description text, and similar information is encoded in the same area. In the deployment stage, when a new project is created, the most matching project is found from the historical projects according to the task description for developers to learn from or reuse. The deployment stage uses both the workflow diagram information and the process description information of the process. The two parts use WNN to obtain vectors respectively, and then the Cosine similarity of the two is calculated to obtain the final score.

3. An intelligent assistance system for low-code development of machine learning applications as claimed in claim 2, characterized in that: The Workflow Embedding neural network is used to convert workflow information in low-code applications into a vector neural network. The input of a process information is A = [T, Graph (V, E), D], where It is the character sequence after the process name and label are concatenated. k+m=n, which are the length of the label and name respectively; the label is the keyword label that the developer adds to the process, and Graph(V,E) is the workflow graph information. Each workflow graph is a directed In the graph, V represents the node set, E represents the edge set, each node v∈V has its own ID attribute; each edge e∈E has two attributes: the start point and the end point; D is the natural language description information of the process; First, use the neural network to embed each part, and use BiLSTM to embed T e t =Embedding(w t ) f t =σ(W f ·[h t-1 ,e t ]+b f ) I t =σ(W i ·[h t-1 ,e t ]+b i ) o t =σ(W o ·[h t-1 ,w t ]+b o ) h t =o t *tanh(C t ) in Yes t The embedding vector, w t is the tth character in T, represents the concatenation of two vectors, is the matrix of trainable parameters in BiLSTM, tanh and σ are the activation functions of BiLSTM, and σ represents the Sigmoid function; finally, T is embedded as an n×d dimensional vector: h=[h1,...,h n ]=BiLSTM(T) For the Graph (V, E) part, since it is format feature information, the GAT network is used to embed the Graph (V, E). The GAT calculation formula includes the information construction stage and the domain aggregation stage. The information construction stage constructs the input representation for the GAT network of each layer. The specific formula is: in Indicates the representation of the network at the beginning. In the WENN network, the node ID is used as the input of the graph and is embedded in the network through the ID Embedding network. Convert it into a d-dimensional vector; The domain aggregation stage aggregates the information to be transmitted by each domain node. The domain represents all nodes adjacent to the current node. The specific formula for neighborhood aggregation is as follows: Where N u is the neighborhood set of node u, is a trainable parameter; Then the two features are fused through the Cross Attention module and the two features are concatenated: w ij =(W[h j ,f i ]) o=[o h ,o f ] in, is a trainable parameter; finally, the two vectors are concatenated and fused, and the output vector o is the final vector representation of the process information.

4. An intelligent assistance system for low-code development of machine learning applications as claimed in claim 3, characterized in that: The Text Embedding neural network for embedding natural language description is used to convert natural language description information into a vector neural network. Assume that the input of a natural language description information is D=w1,…,w m , is a text sequence of length m. BiLSTM is used to embed the text sequence D, and the maximum pooling method maxpool is used to transform its dimension. d = maxpool(BiLSTM(D)).

5. An intelligent assistance system for low-code development of machine learning applications as claimed in claim 4, characterized in that: The Similarity module calculates the similarity between the historical project workflow embedding vector obtained by the Workflow Embedding neural network for embedding workflows and the Text Embedding neural network for embedding natural language descriptions and the project text description embedding vector encoded from the bidirectional RNN network, and uses Cosine similarity to measure the similarity between the two vectors: Where o and d are the vectors of workflow and text description respectively, and ||d|| represents the norm of vector d.

6. An intelligent assistance system for low-code development of machine learning applications as claimed in claim 5, characterized in that: The TransRec model of the low-code development component recommendation method consists of three parts, namely, the Embedding layer, the Transformer layer and the prediction layer. The Embedding layer is responsible for embedding the original data into the corresponding vector representation, and the Transformer layer is responsible for modeling the correlation between sequences. It mines the correlation between the two without considering the distance between the sequence pairs. This modeling method is superior to the RNN-based modeling method when modeling time series data. Finally, the final result is obtained through the prediction layer.

7. An intelligent assistance system for low-code development of machine learning applications as claimed in claim 6, characterized in that: The Embedding layer first needs to preprocess the input and convert the behavior sequence <S1,S2,…,S |S| >Convert to a fixed length sequence S= <S1,S2,…,S n >, where n is the maximum length that the model can handle. If the length of the sequence is greater than n, the most recent n actions are taken as input. If the length of the sequence is less than n, the sequence is completed, that is, the front end of the sequence is added. <padding>, until the sequence length is equal to n, create the component embedding matrix Where d is the dimension of the latent vector, |I| is the number of components, <padding> The corresponding vector is set to a full 0 vector;< / padding> < / padding> Use Position Embedding to integrate sequence information into the input vector by embedding: where e i ∈E is an S with dimension d i Embedded, p i ∈P is the index embedding corresponding to position i of dimension d, g(i) is the decomposition of the component's Bigram vector using the matrix decomposition method, using the learnable position embedding Embedding, the position embedding matrix The Transformer layer will Stacked together to form a matrix It contains two sublayers, a multi-head self-attention sublayer and a Position-wise feedforward network; the multi-head self-attention mechanism captures the dependencies between representation pairs without considering their distance in the sequence; first, H is transformed into l Linearly project to the subspace, and then apply the Attention function to generate the re-projected vector representation in parallel: The projection matrix of each head and is a learnable network parameter, and the formula of the Scaled Dot-Product attention mechanism is as follows: Where Q, K, V are H l It is obtained by applying the Attention function in parallel to generate the re-projected vector representation transformation. The purpose is to avoid extremely small gradients to produce a softer attention distribution; The Position-wise feed-forward network transforms its result, which consists of two affine transformations and uses LeakyReLU as the activation function: FFN(x)=LeayReLU(W 1 x+b 1 )W 2 +b 2 in, It is a learnable network layer parameter and is shared by all positions in the same layer, but not between layers. At the same time, in order to increase the expressive power of the model, Transformer stacks multiple layers of self-attention mechanisms to form a self-attention layer. H l =Transformer(H l-1 ) Transformer(H l-1 )=LN(A l-1 +Dropout(PFFN(A l-1 ))) A l-1 =LN(H l-1 +Dropout(MH(H l-1 ))) A residual connection is used between the two sub-layers, followed by layer normalization (LN), and Dropout is applied to the output of each sub-layer; finally, LN is performed again on the resulting variables to stabilize and accelerate network training; The prediction layer masks the output term o at time step t t , then according to Prediction shielding item t , a two-layer feedforward network is used with ReLU as the activation function in the middle to obtain the probability distribution of the final target component: Where W P , b P , b O are trainable network parameters, Embedding matrix for components. Use shared item embedding matrix in input and output layers to reduce overfitting and model size. In the model training process, define O = {o1, ..., o n } is the true label of the output sequence, S = {S1, ..., S n } is the input label of the behavior sequence: Where t∈[1,…,n] represents the position of the label in the sequence. Using negative sampling technology, a negative example is prepared for each positive example, and Binary Cross Entropy is used as the loss function to train the network: Among them, n is the sequence length, S is the input sequence, and o t For the correct label, is a randomly selected negative example label. During the calculation process, the model will ignore o t = <padding> The loss generated by the position and use Adam as the optimizer.< / padding> 8. An intelligent assistance system for low-code development of machine learning applications as claimed in claim 7, characterized in that: The MetaRec model input is {S1,…,S n-1 }, the output is S n ,The cold start problem refers to that there are only a small number of training examples for a component.,To simulate the cold start scenario, it is assumed that in the current task, any component only appears in K sequences, K∈[1,3,5];,The MetaRec model includes a data preprocessing process; First, define I as the set of all components, divided into I according to the ratio of 7:1:2 train ,I valid ,I test , and there is no intersection between them; following the standard meta-learning setting, according to a set of training tasks Train MetaRec model; any task I train Extract N components as the basis, build Support Set and Query Set for these components, Support Set contains K sequences containing the same ending components, the sequence length is not less than 3, Query Set = {Q1, ..., Q N }, Q i represents a sequence ending with the i-th component and the length of the sequence is not less than 3, Q i The end component of is removed as a label. Such a pair of Support Set and Query Set constitutes a task T i Input; The MetaRec model first uses Sequence Encoder to encode the input into a representation vector; then uses Attention Block to aggregate K sequences in the Support Set to generate a Support Set vector representation; finally, uses the matching network MatchNet to match the fused Support Set latent vector representation with the Query Set latent vector representation to obtain the prediction result.

9. An intelligent assistance system for low-code development of machine learning applications as claimed in claim 8, characterized in that: The Sequence Encoder is specifically: The task of the sequence encoder is to encode each behavior sequence into a separate vector representation, that is, Where S i,1 ,S i,2 ,…,S i,l For different components, l is Set i The length of the sequence is , and this step performs the previous padding operation on the sequence to unify the length under the same input; the specific formula of the sequence encoder is as follows: h i =f(Set i ) e j =W p (W last g(S i，l-1 )+W cur g(S i，j )+W avg S i，avg +b) in, b are all trainable network parameters, g is the Embedding layer, which converts the components into their corresponding vector representations, and then h i With g(S i，l ) are concatenated and the final output is obtained through an m-layer fully connected network; in order to make training easier, skip connections are added to the model; finally and Add together to get the final vector representation The Attention Block is specifically as follows: The task of the attention module is to fuse the h corresponding to the K behavior sequences. In order to make the final latent vector h contain more key features related to the Query Set, the following formula is used for fusion: Where W Q , W K , W V is a trainable network parameter, q is the vector representation of the Query Set after Sequence Encoder, and h is the representation of the Support Set after encoding; The MatchNet is specifically: the main task of matching is to obtain the The final probability distribution is obtained by matching with q, and MatchNet is used as the final matching layer. The specific formula is as follows: Score=σ(W2(ReLU(W1F+b1))+b2) in is a trainable network parameter, σ is a Sigmoid function, and the model is trained using the Meta-Train process commonly used in meta-learning.

Images