Programming support device and programming support method

Abstract

To reduce the burden on a user in adding a node and an edge during visual programming.SOLUTION: A programming support device has: a visual program storage unit that stores a visual program created in the past; a learning data generation unit that edits the stored visual program and generates data for leaning including a pair of input data to a learning model including a directed graph of the visual program and a correct answer data corresponding to the input data to be training data for learning; a graph learning unit that learns a graph learning model having, in the input data, the characteristics of nodes of the visual program and the connection relationship between the nodes as input and the likelihood of the characteristics of nodes of the visual program to be added as output; and an additional node selection unit that calculates the likelihood of the characteristics of the nodes for the learning model learned in the graph learning unit with data of the visual program as input data, and selects nodes to be added based on the likelihood.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a programming support device and a programming support method that use a visual programming interface. [Background technology]

[0002] Methods for implementing program or modeling logic without writing source code include visual programming, visual scripting, procedural modeling, etc. (hereinafter referred to as "visual programming"). In visual programming, the processing content of a program is expressed in a directed graph with nodes that represent processing content and variables, and edges that connect nodes and represent processing flow and variable references.

[0003] Application developers (hereafter referred to as "users") add nodes corresponding to the processing they want to execute, assign properties to the nodes that specify the processing content, connect the added nodes with edges, and define the processing content. By adopting visual programming, users can develop applications without the skills to write source code.

[0004] In general, in application development, development time is shortened by using a pre-prepared program as a template and changing only the necessary parts. Similarly, in visual programming, when developing a program, you select a visual program template that includes the desired program, load the template, and add or modify the template. This eliminates the effort of building a visual program for the parts defined in the template, thereby making application development more efficient.

[0005] In this way, visual programming using templates makes it possible to omit the selection of nodes to add and the definition of edges that connect nodes, which is expected to make application development more efficient.

[0006] Here, in the technology described in Patent Document 1, when supporting the development of visual programming using templates, if the program to be developed includes communication processing with external services, the technology works in conjunction with a deployment manager for deploying external services, automatically deploying the external services when the template is downloaded and automatically setting endpoint information such as the address of the external service in the external service call node, thereby reducing the burden on the developer. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent application No. 2019-24935 Summary of the Invention [Problem to be solved by the invention]

[0008] However, in order to provide visual programming support using templates, as in Patent Document 1, it is necessary to manage and organize the templates, for example by manually defining and registering frequently used programs, which creates a burden to appropriately design and register the templates.There are many items that must be considered when using templates, such as searching for an appropriate template, combining multiple templates, or using only part of a template.

[0009] Although Patent Document 1 considers a method for effectively utilizing templates, it does not consider a method for facilitating template registration and management. Therefore, the technology in Patent Document 1 has a problem in that the more templates there are, the more burden there is in defining, managing, and utilizing the templates.

[0010] Therefore, an object of the present invention is to improve the efficiency of application development without the burden of manually defining and managing templates. [Means for solving the problem]

[0011] In order to solve the above problems, the programming support device of the present invention is a programming support device having a visual program storage unit that stores visual programs created in the past; a learning data generation unit that edits the visual program stored in the visual program storage unit and generates learning data that includes input data for a learning model including information about the directed graph of the visual program and pairs of correct answer data corresponding to the input data that serve as teacher data for learning; a graph learning unit that takes as input the node features and node connection relationships of the visual program from the input data and learns a graph learning model that outputs the likelihood of the node features of the visual program to be added; and an additional node selection unit that uses the visual program data as input data for the learning model learned by the graph learning unit, calculates the likelihood of the node features, and selects a node to be added based on the likelihood. [Effects of the Invention]

[0012] According to the programming support device and programming support method of the present invention, application development can be made more efficient without the burden of manually defining and managing templates. [Brief explanation of the drawings]

[0013] [Figure 1] 1 shows an example of a system configuration of a programming support device according to an embodiment. [Figure 2] An example of the GUI for the visual programming section. [Figure 3A] An example of visual program data. [Figure 3B] An example of a visual program expressed in JSON format. [Figure 4]10 is an example of a processing flow in a learning data generation unit. [Figure 5] 10 shows an example of a processing flow in a feature amount calculation unit. [Figure 6] 10 is an example of a processing flow in a graph learning unit. [Figure 7] 10 shows an example of a processing flow in an additional node selection unit. [Figure 8] An example of a recommendation in a visual program section using an additional node selection section. DETAILED DESCRIPTION OF THE INVENTION

[0014] Instead of manually defining and managing templates, the programming support device of the present invention statistically learns the node and edge patterns that frequently appear in visual programs, thereby generating commonly used visual program patterns that were previously prepared in the form of templates using a statistical model. The programming support device of the present invention also collects and stores existing visual programs in a database, accumulates the relationships between existing visual programs and the nodes and edges to be added, statistically learns the relationships between the graph structure of a visual program in the process of being created and the nodes and edges to be added, and presents to the user the nodes and edges to be added to the visual program in the process of being created. Because there are patterns of frequently used nodes and their connection relationships, statistically revealing the regularity between the nodes to be added and the graph structure of the visual program in the process of being created makes it possible to present the above-mentioned information to improve the efficiency of template-based visual programming. Below, a programming support device 100 according to one embodiment of the present invention is described with reference to the drawings.

[0015] 1 shows an example of the system configuration of a programming support device 100 according to an embodiment of the present invention. This system collects and stores visual programs created on a user terminal 1. By statistically learning the relationship between what visual program nodes are added to a visual program in the process of being created, the system can recommend which visual program nodes the user should add when developing a visual program.

[0016] The programming support device 100 of this embodiment is a computer equipped with hardware such as a central processing unit (CPU) or a graphics processing unit (GPU), a main storage device such as a semiconductor memory, an auxiliary storage device such as a hard disk or a solid-state drive (SSD), and a communication device. The computing device (CPU, GPU) executes a desired program while referencing various databases to realize each of the functional units described below. However, all of the functional units do not necessarily need to be realized on a single computer. For example, the functional units above the dashed-dotted line AA in FIG. 1 may be realized on a computer directly used by the user, and the functional units below may be realized on a cloud server. Alternatively, the functional units above the dashed-dotted line BB in FIG. 1 may be realized on a computer directly used by the user, and the functional units below may be realized on a cloud server. Each of the functional units in FIG. 1 will be described below.

[0017] <User Device 1> The user terminal 1 is a computer and physical interface that allows a user to operate the visual program unit 2, which will be described later. This physical interface is a display device (such as a display) for displaying the visual program GUI (Graphical User Interface) generated by the visual program unit 2, and an operation device (such as a mouse, keyboard, or touch panel) for operating the GUI generated by the visual program unit 2. The user operates the GUI generated by the visual program unit 2 via the display device and operation device of the user terminal 1 to create a visual program.

[0018] <Visual Program Section 2> The visual program unit 2 is a functional unit used by the user to create a visual program, and has the functions of presenting a GUI for the visual program to the user, creating and updating data for the visual program in response to the user's operation on the user terminal, and updating the GUI display. Furthermore, in order to create and update data for the visual program, the visual program unit 2 adds, modifies, or deletes nodes N or edges E of the visual program, and updates metadata D including tag information consisting of the name of the visual program and multiple keyword strings. M It has functions such as inputting the information of the created visual program, storing it in memory, hard disk, SSD, object storage on the network, and retrieving data including definition information such as the names of nodes N and edges E that can be added in the visual program, types of functions and variables, input / output property names, and input / output property values.

[0019] <Visual Program Storage Unit 3> The visual program storage unit 3 is a functional unit that stores the visual program data created by the visual program unit 2 together with the identifier. This visual program storage unit 3 has the function of storing the visual program data contained in the command according to the command of an external program including the visual program unit 2 by the user, or outputting the corresponding visual program according to the identifier requested in the command, and listing, selecting, and sorting the identifiers of the stored visual programs and the information contained in the visual program data.

[0020] <Learning Data Generation Unit 4> The learning data generation unit 4 is a functional unit that acquires multiple visual programs stored in the visual program storage unit 3 and generates learning data including pairs of teacher data and correct answer data using the acquired visual programs. This learning data generation unit 4 is a functional unit that is executed at the request of an administrator or periodically. The learning data generated by the learning data generation unit 4 consists of explanatory variable data that serves as input data to the learning model, and objective variable data that corresponds to the explanatory variables and serves as correct answer data when training the learning model.

[0021] Here, the learning model is a learning model handled by the graph learning unit 6 and the learning model storage unit 7, which will be described later. The learning data generation unit 4 creates a connected subgraph by omitting some nodes N and edges E from the directed graph of the acquired visual program, and combines the subgraph with the metadata D of the visual program. M The explanatory variables are the information including the above, and the objective variable is the information of the node N and edge E that were connected starting from the subgraph in the original directed graph. The explanatory variables correspond to the information of the visual program that is trying to add node N and edge E, and the objective variable corresponds to the information of the node N and edge E that were actually added to the visual program.

[0022] <Feature Calculation Unit 5> The feature calculation unit 5 calculates the metadata D of the visual program of the explanatory variable data or the objective variable data output from the learning data generation unit 4 or the additional node selection unit 8. M , and the information on the nodes N and edges E of the visual program's directed graph are converted into metadata features F represented as a numerical vector so that they can be applied to a statistical learning model in the graph learning unit 6. M , node feature F N , edge feature F E This is a functional unit that converts

[0023] For the data of the directed graph of a visual program, the node information and edge information are converted into numerical vectors and then converted into features. The connectivity information between nodes in the directed graph is expressed numerically using a dense or sparse adjacency matrix, and the node feature F N , edge feature F E When characterizing the information of node N and edge E, the type, name, and property value of node N, and the input and output item names of edge E are used.

[0024] <Graph Learning Section 6> The graph learning unit 6 is a functional unit that performs learning of a learning model using explanatory variables and objective variables of the visual program that have been characterized by the aforementioned feature calculation unit 5. When implementing the graph learning unit 6, it is preferable to use a GPU, which is a computing device dedicated to image processing, rather than a CPU, which is a general-purpose computing device.

[0025] The learning model described above uses the metadata feature F M , node feature F N , edge feature F EThe graph learning unit 6 takes the connectivity information of the directed graph as input and outputs the likelihood of information including the node type, node name, node properties, edge type, and input / output item name of the node N included in the objective variable. Furthermore, the learning model described above is trained using the explanatory variables and objective variable data input to the graph learning unit 6 to improve the prediction and classification accuracy of the objective variable. In other words, learning is performed so that the node N and edge E added to the visual program of the explanatory variables, which are the objective variables, can be predicted from the information in the visual program, which is the explanatory variable.

[0026] As described in the Background Art section, visual programs have patterns (templates) of frequently used combinations of nodes N and edges E. Therefore, if part of a visual program matches the aforementioned pattern, there is a high probability that the nodes N and edges E that follow that visual program will also follow that pattern. Therefore, by converting the information of the visual program in the process of being created into a numerical vector as faithfully as possible in the aforementioned feature calculation unit 5, and then using the visual program in the process of being created to train the problem of predicting the information of nodes N and edges E that have actually been added, the aforementioned patterns are statistically learned, making it possible to predict the likelihood of the nodes N and edges E that have been added from the visual program in the process of being created.

[0027] <Learning model storage unit 7> The learning model storage unit 7 is a memory device that stores data including the architecture, weights, learning conditions, and accuracy indicators of the learning model learned by the graph learning unit 6, and is a functional unit that stores learning models and provides learning models from among the stored learning models in accordance with a request from the additional node selection unit 8.

[0028] <Additional node selection section 8> The additional node selection unit 8 is a functional unit that recommends candidates for additional nodes to the visual program unit 2. Specifically, in response to a recommendation request from the visual program unit 2, the additional node selection unit 8 receives information about the visual program being created by the visual program unit 2, inputs the information about the visual program being created to the feature calculation unit 5, calculates the corresponding feature F, retrieves the learning model stored in the learning model storage unit 7, and, based on the likelihoods of the nodes N and edges E obtained by inputting the calculated feature F into the learning model, creates a ranking of candidates for nodes N and edges E to be added to the visual program being created and presented to the user by the visual program unit 2 using sorting using the likelihood, etc., and sends the ranking to the visual program unit 2, where it is displayed on the user interface generated by the visual program unit 2.

[0029] For example, when a user performs an operation to add a node to a visual program that is being created using the visual program unit 2, the visual program that is being created is sent to the added node selection unit 8, and in response, the added node selection unit 8 sends a ranking of candidates for the node N and edge E to be added to the visual program unit 2.The candidate added nodes are then displayed to the user in a ranked order, or the node N and edge E with the highest likelihood is automatically added.

[0030] <An example of the GUI for Visual Program Section 2> Figure 2 shows an example of the GUI of the visual program unit 2. A list L of nodes N is displayed on the left side of Figure 2, and a visual program created by the user operating the user terminal 1 is shown on the right side.

[0031] As shown here, the created visual program has multiple nodes N that define data and processing content, and the nodes are connected to each other by directed edges E. For example, an edge E connects the output O1 of node N of the upper function f1 to the input I1 of node N of function f2, and similarly, an edge E connects the output O1 of node N of the lower function f1 to the input I2 of node N of function f2. Note that the points where edges E can be connected (output O, input I of node N) will be referred to as "handles" hereafter.

[0032] In addition to connecting nodes N with edges E, it is also possible to define the value of a property P that represents data or processing content for a node N. Also, a node N that represents a data structure such as a constant or a structure can be defined. D There are also nodes N other than functions with inputs and outputs, such as those listed above. There are also other nodes N that define the order and flow of processing, such as conditional branches and loops. The user creates a visual program by operating the user terminal 1 to press the add button to add nodes N and edges E, or to delete, edit (change parameters, etc.), or move nodes N and edges E.

[0033] Nodes N, which represent processes including functions, constants, and branches that are components of a visual program, are predefined, and the desired one can be selected from the list L of nodes N displayed on the left side of the GUI in Figure 2. The definition information of each node N, such as function f1 and function f2, including what names of properties P can be defined and whether handles to which edges E can be connected are defined, is predefined in the visual program section 2.

[0034] <Example of visual program data> Figure 3A shows the visual program data D P This is an example of visual program data D P contains the visual program metadata D M and the graph data D of a visual program defined as a directed graph. G Includes:

[0035] In Figure 3A, metadata D M Here is an example written in JSON (JavaScript (registered trademark) Object Notation), but M The metadata can be stored in a format that includes a hierarchical data structure such as XML or binary. M contains information including a program name string, creator, save date, and tag information consisting of multiple keyword strings.

[0036] On the other hand, the graph data of the visual program G The node information I that can reproduce the visual program created in the visual program section 2 is N , Edge Information I E This graph data D G An example of the graph data D of a visual program is shown in Figure 3B. G Node information I N and edge information I E It is expressed as an array of

[0037] Node information I shown by dashed lines N The array contains information including the identifier of node N, data, function, conditional branch, etc., a node type string indicating the type of node N, a name string indicating the name of the function, etc., the name of property P that the user can directly input for node N, the input value, and the date and time when node N was added.

[0038] On the other hand, edge information I shown by the dashed line E The array defines information including the edge identifier, the identifiers of the edge source and destination nodes N, and the handle names of the source and destination nodes N to which the edge E is connected.

[0039] <Processing flow of the learning data generation unit 4> 4 is an example of a processing flow in the learning data generation unit 4. When the learning data generation process is started, first, in step S41, the learning data generation unit 4 reads the visual program stored in the visual program storage unit 3. Either the entire visual program is read, or the metadata D M The programs are then filtered based on the data, including the creation date and tag information, and the subsequent processing is performed on each of the multiple visual programs that have been loaded.

[0040] Next, in step S42, the learning data generation unit 4 extracts directed graph data describing the program from the read visual program data.

[0041] In step S43, the learning data generation unit 4 selects a pair of a node NA and a node NB connected to it from the directed graph data of the visual program. At the time of selection, the nodes NA and NB are selected randomly, or the nodes NA and NB are selected from the metadata D of the visual program. M A possible method is to select nodes NA and NB when node NB is added immediately after node NA is created, using the editing history of the visual program, such as the creation date and time included in node N.

[0042] Next, in step S44, the learning data generation unit 4 creates a subgraph by deleting node N and edge E from the directed graph of the visual program so that node NA becomes a terminal node (there is no node with node NA as a connection source). Methods for deleting node N and edge E include deleting nodes N and edges E created after the creation of node NA using the above-mentioned editing history, deleting the minimum number of nodes N and edges E so that node NA becomes a terminal node, or deleting nodes N and edges E randomly so that node NA becomes a terminal node.

[0043] In step S45, the learning data generation unit 4 generates the subgraph, the information on the node NA that was the basis for creating the subgraph, and the metadata D of the visual program. M are defined as explanatory variables. Information on the node NB and the edge E between the node NB and the node NA is defined as objective variables. The explanatory variables and objective variables defined here are output to the feature calculation unit 5.

[0044] In step S46, the learning data generation unit 4 performs the processes of steps S43 to S45 one or more times for each visual program by changing the selected nodes NA and NB, and as a result of the processing, one or more pairs of explanatory variables and target variables are obtained and output from one visual program.

[0045] <Processing flow of the feature amount calculation unit 5> 5 shows an example of a processing flow in the feature calculation unit 5. The feature calculation process in the feature calculation unit 5 is started in response to an external request for feature calculation, including a request from the learning data generation unit 4 or the recommendation unit 8.

[0046] First, in step S51, the feature calculation unit 5 receives data on explanatory variables and target variables from the learning data generation unit 4, or receives only explanatory variables from the additional node selection unit 8. In the latter case, subsequent processing is performed only on the explanatory variables. The explanatory variable data includes the metadata D of the visual program. M It contains information on:

[0047] In step S52, the metadata feature calculation unit 5a of the feature calculation unit 5 calculates the metadata feature F M Specifically, the metadata D M Metadata D of explanatory variables, including tag information, which is an array of keyword strings. MThe string information in the text is converted into a numerical vector. When converting to a numerical vector, it is tokenized by segmenting the text according to the characteristics of the language, and then vectorized using methods including language models using co-occurrence relationships and deep learning, such as word2vec, seq2vec, Transformer, and BERT. The vector generated here is used as the metadata feature F M It is called.

[0048] Next, in step S53, the node feature calculation unit 5b of the feature calculation unit 5 calculates the node feature F N Specifically, for node N of the directed graph of the visual program included in the explanatory variables and the objective variables, quantities including data on the type of node N (function, constant, conditional branch, etc.), name (function name, etc.), property value data of node N in key-value format, and whether or not the node is NA in the learning data generation unit 4 are converted into a numerical vector. During conversion, the type and name of node N are converted into a vector as categorical values ​​using a conversion method including One Hot Encoding.

[0049] The property values ​​of node N are converted into vectors using an appropriate conversion method (numeric vector, one-hot, language model, etc.) depending on the data type of the property value (numeric, categorical value, string, etc.). As mentioned above, after each component of node N is expressed as a numeric vector, these vectors are converted into fixed-length features F using aggregation methods including combination, addition, average, variance, maximum, multilayer perceptron attention mechanism, etc. This feature is called the node feature F. N It is called.

[0050] In step S54, the edge feature amount calculation unit 5c of the feature amount calculation unit 5 calculates the edge feature amount F ESpecifically, the information of the edge E of the directed graph of the visual program, which is included in the explanatory variables and objective variables, including the type of edge E (function input / output, execution order definition, data flow, type of data sent / received from node N through edge E, etc.), the handle of the destination node N, and the handle identifier of the source node N, is converted into a numerical vector. To convert into a numerical vector, the edge type is converted into a numerical vector using one-hot encoding, and the handle information of the destination and source node N is converted into a numerical vector using one-hot encoding, a language model of property names, and other methods including feature quantification. The numerical vector of the aforementioned envelope is converted into a fixed-length feature using aggregation methods including combination, addition, average, variance, maximum, multilayer perceptron, attention mechanism, etc. This feature is converted into an edge feature F E It is called.

[0051] Next, in step S55, the feature calculation unit 5 calculates a plurality of node features F N , edge feature F E The feature F is a column vector, and the combined edge feature F E For each row of the table, the source and destination nodes N of the corresponding edge E are expressed as the combined node features F N It checks which row the edge E is in, creates a pair of integer row indices for the destination and source node features, and creates a matrix of indices for the node N to which the edge E is connected. At this time, the matrix of indices for node N is used to generate an adjacency matrix for the directed graph in a sparse or dense data structure, as necessary. Hereinafter, the matrix of indices for node N, or the sparse or dense adjacency matrix, will be referred to as an adjacency matrix.

[0052] Finally, in step S56, the feature calculation unit 5 calculates the node feature F N , edge feature F E , graph feature F G , and outputs data including the adjacency matrix to the graph learning unit 6.

[0053] <Processing flow of graph learning unit 6> FIG. 6 shows an example of the processing flow of graph learning unit 6.

[0054] First, in step S61, the graph learning unit 6 calculates the node feature F N , edge feature F E , graph feature F G Receives data on explanatory variables and objective variables characterized by features including adjacency matrices.

[0055] Next, in step S62, the graph learning unit 6 calculates the node feature F N , edge feature F E , data that holds structural information of a directed graph consisting of an adjacency matrix, and metadata D M The graph feature F G We build a graph learning model that can simultaneously input

[0056] The graph learning model constructed here uses the node feature F N , edge feature F E ,The graph learning-based learning model takes the adjacency matrix as input and,graph feature F, G It is a multimodal learning model that combines a learning model that takes vectors as input with graph kernel learning, graph convolution learning, message passing, graph transformer, and other deep learning graph learning models.

[0057] In addition, the metadata feature F M For this, we apply machine learning methods that take vectors as input, such as linear models and multilayer perceptrons. N , edge feature F E , and the learning architecture using the adjacency matrix as input and the metadata feature F MThe output vectors of the learning architecture that takes the input as input are further combined, and a method including a multi-layer perceptron is used to construct a learning model architecture that finally outputs the likelihood and predicted values ​​of the type, name, property value, origin and destination handle of the node N, which correspond to the objective variable. Note that multimodal learning is not adopted, and the node feature F N , edge feature F E Alternatively, a learning architecture that outputs likelihood and predicted values ​​may be constructed using only a learning architecture that takes the adjacency matrix as input.

[0058] Next, in step S63, the graph learning unit 6 divides the explanatory variable and objective variable data, which are data for training the aforementioned learning model, into a training dataset to be used in the learning process to optimize the weights of the learning model, and an evaluation dataset to evaluate the accuracy of the trained learning model. The division in this step is performed randomly or using criteria including the time of creation of the visual program, and the division pattern is one, or multiple division patterns of datasets are generated to later perform K-fold cross validation and leave-one-out cross validation.

[0059] In step S64, the graph learning unit 6 performs learning using the aforementioned learning dataset. The loss function used in this step is the weighted sum of the cross-entropy loss and squared error of the type, name, and property value of node N, the source and destination handle information of edge E, and the likelihood and predicted value predicted by the learning model, as well as the objective function's node type, name, property value, and source and destination handle information of edge E. Depending on the learning method, a regularization term or the like is added to the loss function, and stochastic gradient descent or the like is used for learning.

[0060] Next, in step S65, the graph learning unit 6 evaluates the accuracy of the graph learning model trained in step S64 using the evaluation dataset prepared in step S63. Evaluation metrics include least squares error, cross-entropy, AUC, Top N accuracy rate, f1 score, etc. When multiple dataset division patterns are defined in the dataset division in step S63, such as K-fold cross-validation, in step S65, accuracy metrics are calculated for each division pattern and the accuracy is evaluated by averaging them. In some cases, to improve accuracy, optimization methods such as Bayesian optimization and neural network architecture search may be used to tune hyperparameters, including the learning architecture, learning rate, batch size, and loss function weights, by repeating the learning (S64) and evaluation (S65).

[0061] Finally, in step S65, the graph learning unit 6 outputs the learning results, including the architecture of the learning model, learned data such as weights, hyperparameters used during learning, data to be learned, accuracy index, and creation date, to the learning model storage unit 7 and stores them.

[0062] <Processing flow of the additional node selection unit 8> FIG. 7 shows an example of a processing flow in the additional node selection unit 8.

[0063] First, in step S81, the additional node selection unit 8 acquires a selection request for an additional node from the visual program unit 2. This selection request is a request sent from the visual program unit 2 to the additional node selection unit 8 when the user operates the node addition button (see FIG. 2) on the visual program GUI, or when the visual program unit 2 attempts to automatically add a node N regardless of user operation.

[0064] This selection request is made through a method including process communication between programs or an API (Application Programming Interface) through a network, and is transmitted to the metadata D of the visual program being created in the visual program section 2. M The information includes the information of the node N that is the connection source of the node N to which the visual program is to be added, as well as data of the directed graph of the visual program. If available, the information of the node N that is the connection source of the node N to which the visual program is to be added is also included. The information of this node N is treated as the node NA by the feature calculation unit 5.

[0065] In step S82, the additional node selection unit 8 outputs the data included in the selection request (information about the visual program being created by the user) as an explanatory variable to the feature amount calculation unit 5, and calculates the metadata feature amount F M , node feature F N , edge feature F E , and the numerical vector of the adjacency matrix is ​​calculated.

[0066] Next, in step S83, the additional node selection unit 8 selects and acquires one of the learning models stored in the learning model storage unit 7 in response to the selection request, i.e., taking into consideration the accuracy, creation date, and data including the learning target data.

[0067] In step S84, the additional node selection unit 8 inputs the feature F calculated in step S82 into the learning model acquired in step S83, and obtains as output the type, name, property value, likelihood of the source and destination handles of edge E, and predicted value of node N.

[0068] If the selection request acquired in step S81 does not contain information about node NA (see steps S43 to S45 in Figure 4), consider the case where each terminal node in the directed graph corresponds to node NA, and calculate the feature F (S82) and the likelihood-prediction value (S84) for each using the same learning model, and average the outputs of the learning model for all of the above cases to adopt the likelihood-prediction value.

[0069] Finally, in step S85, the additional node selection unit 8 transmits information such as likelihoods and predicted values ​​calculated using the learning model to the visual program unit 2. At this time, the likelihoods and predicted value information is output with semantic information (including the likelihood of the node type, the likelihood of the node name, the likelihood of the handle names of the connection source and connection destination of edge E, etc.) added to it and transmitted to the visual program unit 2.

[0070] <An example of a node recommendation screen> Next, Fig. 8 shows an example of a recommendation screen for a node N to be added, which is presented on the GUI of the visual program unit 2 when the node addition button in Fig. 2 is operated, etc. For simplicity, Fig. 8 omits the illustration of the information that was displayed to the left of the node for function f2 in Fig. 2.

[0071] When a user operates the user terminal 1 to add a new node N downstream of the node of the function f3, the visual program unit 2 sends an additional node selection request (specifically, the information of the directed graph currently being created, enclosed by the dashed line, and, if specified by the user, the information of the node of the function f3 to which the additional node is connected, and the metadata D of the visual program). M The information (FIG. 3A) is transmitted to the additional node selection unit 8.

[0072] 7 is executed in the additional node unit 8, and the final output is information including the likelihood of the node type, the likelihood of the node name, and the likelihood of the handle names of the connection source and connection destination of edge E. When handling likelihood information, various pieces of information are sorted in descending order of likelihood.

[0073] When the visual program unit 2 receives the output of the additional node selection unit 8, the GUI of the visual program unit 2 displays a node recommendation screen downstream of the directed graph currently being created, indicated by the dashed-dotted line. The node recommendation screen consists of a recommended node list screen at the top and an application example screen at the bottom. The recommended node list screen lists nodes in descending order of likelihood. When the user selects a node N to add from the list screen, or a node whose likelihood exceeds a certain threshold is automatically selected, the node N recommended by the additional node selection unit 8 is added downstream of the visual program currently being created. In this way, the user can easily create a visual program by sequentially selecting the recommended nodes. When adding a node, the additional node selection unit 8 may use the predicted value of the property information and the likelihood and predicted value of the handle information of the connection source and connection destination of the edge E output from the additional node selection unit 8 to automatically add the corresponding edge E or property P if the likelihood exceeds a threshold.

[0074] In the example GUI screen of Figure 8, if a visual program to which each node in the list of recommended candidates has been added matches, partially matches, or approximately matches the directed graph of an existing visual program in the visual program storage unit 3 around the added node, one or more visual programs with the highest degree of match are read from the visual program storage unit 3 and displayed as examples of adding node N, starting with the one with the highest degree of match.

[0075] To determine whether the visual program in the visual program storage unit 3 matches the added node, partial matches are determined using algorithms such as subgraph matching, approximate subgraph matching, and frequent pattern matching. When displaying the added node, a graph of the visual program around the added node and metadata D of the visual program are displayed. M Present information including:

[0076] As described above, according to this embodiment, the following effects can be obtained. (1) When users create visual programs, the system recommends nodes and edges to add, reducing the user's burden in selecting appropriate nodes. (2) By replacing templates with statistical models that learn from data, we can present frequently occurring patterns of visual programs by collecting performance data of visual programs without the burden of defining and managing templates. (3) The more actual data there is, the more patterns can be selected to recommend optimal nodes and edges, improving recommendation accuracy. (4) The know-how of visual programs contained in past performance data will be reflected in recommendations.

[0077] Therefore, according to the programming support device of this embodiment, by using a statistical model to learn and collect visual program templates, frequently occurring visual program patterns can be presented to the user during application development, without the burden of manually defining and managing templates, thereby making application development more efficient.

[0078] The present disclosure is not limited to the above-described examples and includes various modifications. The above-described examples have been described in detail to clearly explain the present disclosure, and are not necessarily limited to those including all of the described configurations. It is possible to replace part of the configuration of one example with the configuration of another example, and it is also possible to add the configuration of another example to the configuration of one example. Furthermore, it is also possible to add, delete, or replace part of the configuration of each example with other configurations.

[0079] The above-described configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware such as an integrated circuit. The above-described configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or on a storage medium such as a flash memory card or a DVD (Digital Versatile Disk).

[0080] In each embodiment, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0081] 100 programming support device, 1 user terminal, 2 visual program unit, 3 visual program storage unit, 4 learning data generation unit, 5 feature calculation unit, 6 graph learning unit, 7 learning model storage unit, 8 additional node selection unit

Claims

1. A programming support device that supports visual programming, a visual program storage unit for storing previously created visual programs; a learning data generation unit that edits the visual program stored in the visual program storage unit and generates learning data including input data to a learning model containing information about a directed graph of the visual program, and correct answer data corresponding to the input data serving as training data for learning; a graph learning unit that learns a graph learning model that receives inputs of the node features and node connection relationships of the visual program among the input data and outputs likelihoods of the node features of the visual program to be added; an additional node selection unit that calculates likelihoods of node features using visual program data as input data for the learning model learned by the graph learning unit, and selects nodes to be added based on the likelihoods; A programming support device comprising:

2. 2. The programming support device according to claim 1, The system has a node feature calculation unit that converts node data, including node types, node names, and node property values, into feature quantities; the node feature calculation unit calculates node features from the input data and outputs the calculated features to the graph learning unit; The programming support device is characterized in that the graph learning unit learns a graph learning model using data including the node features as input.

3. 2. The programming support device according to claim 1, The method has an edge feature amount calculation unit that converts data including handle information of edge connection destinations and connection sources and edge types, which are included in data of edges connecting nodes in a visual program, into feature amounts; the edge feature calculation unit calculates an edge feature from the input data and outputs it to the graph learning unit; The programming support device is characterized in that the graph learning unit learns a graph learning model using data including the edge feature as input.

4. 2. The programming support device according to claim 1, a metadata feature amount calculation unit that converts metadata including tag information including a name associated with a visual program and an array of keyword strings into feature amounts; the metadata feature calculation unit calculates metadata features from the input data and outputs the metadata features to the graph learning unit; The programming support device is characterized in that the graph learning unit learns a graph learning model using data including the metadata feature as input.

5. 2. The programming support device according to claim 1, The learning data generation unit sets a partial graph of a visual program stored in the visual program storage unit as input data, and sets some of the nodes and edges connected to the partial graph as correct data.

6. 2. The programming support device according to claim 1, The learning data generation unit sets a visual program in the middle of being created at a certain point in time as input data based on the creation history of the visual program stored in the visual program storage unit, and sets nodes and edges added to the visual program in the middle of being created as correct answer data.

7. A programming support method for supporting visual programming, comprising: (a) collecting and storing previously created visual programs; a step (b) of generating learning data including pairs of input data to a learning model including information on a directed graph of a visual program and correct answer data corresponding to the input data serving as training data for learning; a step (c) of learning a graph learning model that uses the node features and node connection relationships of the visual program among the input data as inputs and outputs the likelihood of the node features of the visual program to be added; Step (d) of calculating likelihood of node features in the trained graph learning model using data of the visual program being created by the user as input data; (e) outputting a node to add based on the likelihood; A programming support method comprising:

8. 8. The programming support method according to claim 7, In the step (d), when a user performs an operation to add a node to a visual program, the likelihood of the node's features is calculated using the data of the visual program at the time the operation was performed as input data.

9. 9. A programming support method according to claim 8, comprising: The programming support method further comprises a step (f) of sorting based on the likelihood calculated by the graph learning model, selecting one or more nodes with high likelihood, and presenting the one or more nodes with high likelihood to the user as candidates for nodes to be added.

10. 10. The programming support method according to claim 9, In the step (f), a degree of similarity between the directed graph of the visual program when the presented candidate node is added to the visual program and a visual program created in the past is calculated, and if there is a past visual program whose degree of similarity exceeds a threshold, one or more matching visual programs are presented to the user.

Images