Visual programming system

Abstract

A visual programming system and intelligent agent are provided. The visual programming system comprises a storage medium to store one or more predefined visual programming elements, a user interface c

Description

TECHNICAL FIELD The present disclosure relates to methods and a system for visual programming. BACKGROUND In recent decades, data science has transformed business and industry. Data science has been applied to optimization, supply chain management, resource allocation and many other areas to improve efficiency, reduce waste and increase profitability. One domain which has benefitted greatly from these advances is condition monitoring. Traditionally, condition monitoring systems measured parameters such as vibration and temperature via single sensors attached to a machine. In recent years more data-oriented approaches have been developed which leverage machine learning and data analytics methods. For example, predictive maintenance systems use data analytics on historical time series data and real-time sensor data to identify patterns and anomalies and pre-emptively alert users to potential impending problems. This reduces equipment failures which, in turn, helps to reduce downtime and extend the lifespan of critical assets. In data analytics systems raw data is transformed through various stages in order to prepare the data for more sophisticated algorithms. For example, data normalization may be used to re-scale data into a uniform range, statistical analysis may be used to remove erroneous values and data aggregation may be used to collect, combine and organize data into a complete dataset. Different tools and systems have been developed to facilitate data transformation. Visual programming systems are one example of such a tool. Visual programming systems are software development systems which give a user the ability to create programs using graphical elements rather than text-based elements. These systems make programming more accessible by providing a more intuitive and visually oriented approach. Visual programming systems may comprise a graphical user interface (GUI) such as the GUI 100 depicted in Figure I. The GUI 100 comprises a workspace I 10 which provides an area in for a user to place and connect visual programming elements such as the element 120 which represent individual components of programming logic. The interconnections 130 between the elements may represent the flow of data between data and nodes 140 may represent data inputs and outputs. In the example shown in Figure I, the GUI 100 comprises a library 150 of pre-defined graphical elements 160 for use in the workspace. These pre-defined elements 160 may represent commonly used programming elements including variables, arithmetic functions such as binary and unary operations, statements including conditionals and iterators, and vector and scalar operations. Visual programming systems offer a variety of benefits which, in principle, make them an attractive choice for programming data transformations. For example, in comparison with conventional code, visual programming interfaces provide visualization of data flows. Many visual programming systems also enable re-use of previously generated components providing modularity. In the case of data transformation many of the operations are based on commonly occurring methods and functions which are replicated across systems. The easy-to-understand intuitive nature of visual programming off-loads a lot of the work which might otherwise be required to write a program. Data engineers and scientists are usually more interested in the result of downstream analytics operations and therefore any system which simplifies the process of data transformation is valuable in these applications. However, in practice complexities arise in programming data transformation which diminish the benefits of using visual programming. All but the simplest data transformations are inefficient to program visually and difficult to interpret. For more complex data transformations, the user is forced to choose between complex procedures using a small number of primitives or simple-looking procedures using many complex primitives. In the former case, the high visual complexity is hard for a user to understand. In the latter case, the complexity is hidden behind black boxes and opaque to the user. In the limit, there is an explosion of either combinations of elements or the number of elements. This complexity is illustrated with the following example: A user wishes to identify threshold violations in a time series. Accordingly they require a method of comparing data to a threshold. This data could be the most recent data, or a window of recent data. If using a window the window could be a sliding window that covers a certain number of data points or a sliding window covering a time period. Alternatively the window could be a “hopping window” which covers data from a fixed point in time. Once the data is windowed, the user needs to decide how to do the comparison with the threshold. For example, they may require that the most recent N points exceed the threshold, that N points within the window exceed it, that a fraction of points exceed it, or that a statistical measure like the median is above the threshold. A user may find it difficult to assemble windowing functions and vector-scalar comparators from simple primitives. Therefore, the system may need to provide predefined blocks for these different windowing schemes and comparators. However, if the system provides these predefined blocks, the user still needs to know what these blocks do and this may be non-obvious. As the example illustrates, a task which appears simple gives rise to considerable complexity and this complexity only multiplies when other tasks are considered in a similar way. Nevertheless, despite these drawbacks, visual programming systems still have the persistent benefit of being more approachable for users. Many users who are unwilling to engage with regular programming languages, will willingly engage with a visual programming system. There is therefore a need for improving visual programming systems in order to increase useability and improve the user experience. SUMMARY It is an object of the invention to provide an improved visual programming system which supports data transformation. The foregoing and other objects are achieved by the features of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures. According to a first aspect, a visual programming system is provided. The visual programming system comprises a storage medium to store one or more predefined visual programming elements; a user interface comprising a workspace to perform visual programming based on the one or more predefined visual programming elements; an execution module to translate a configuration of visual programming elements in the workspace into executable instructions; and an intelligent agent comprising: a trained language model; and one or more modules to facilitate interaction between the trained language model and the visual programming system. In response to an input comprising a natural language query, the language model evaluates the natural language query and interacts with the visual programming system using the one or more modules, based on the content of the query. In a first implementation form the system is arranged to store the configuration in the storage medium. In a second implementation form the user interface comprises a user input feature to enable a user to assign labels to configurations of visual programming elements. In a third implementation form the user interface comprises a user input feature to enable a user to input dialogue to the language model. In a fourth implementation form each of the one or more modules is associated with a set of functions and each of the modules is arranged to perform one or more selected functions from its associated set. In a fifth implementation form a set of functions comprises functions to: a) embed a label of a configuration of visual programming elements as a vector in an embedding space; b) embed a natural language description of a program as a vector in the embedding space; c) embed an abstract syntax tree of a configuration of visual programming elements as a vector in the embedding space; d) compare vectors in the embedding space; or e) identify stored configurations of visual programming elements using natural language descriptions of programs, based on the comparison of corresponding vectors in the embedding space. In a sixth implementation form a set of functions comprises functions to: a) interpret a configuration of visual programming elements; b) evaluate a program executed by a configuration of visual programming elements; c) identify inconsistencies in a configuration of visual programming elements; or d) reconfigure a configuration of visual programming elements. In a seventh implementation form a set of functions comprises functions to: a) generate a configuration of visual programming elements based on a natural language description of a program; b) modify a configuration of visual programming elements based on a natural language query; or c) provide a natural language description of a modification to a configuration of visual programming elements. In an eighth implementation form a set of functions comprises functions to a) translate a configuration of visual programming elements into code in a target programming language; b) translate code in the target programming language into a configuration of visual programming elements in the workspace; or c) augment visual programming elements in the workspace with code in the target programming language. These and other aspects of the invention will be apparent from the embodiment(s) described below. BRIEF DESCRIPTION OF THE DRAWINGS For a more complete understanding of the present disclosure, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: Figure I shows a user interface, according to an example. Figure 2 shows a visual programming system, according to an example. Figure 3 shows a user interface, according to an example. Figure 4 shows a flow diagram of a computer-implemented method for interacting with a visual programming system, according to an example. Figure 5 shows a condition monitoring system, according to an example. DETAILED DESCRIPTION Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein. Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate. The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and “the” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof. Unless otherwise defined, all terms including technical and scientific terms used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealized or overly formal sense unless expressly so defined herein. The methods and systems described herein enhance visual programming using language models. Visual programming systems enable users to create programming by manipulating visual programming elements. The visual programming system described herein is particularly suited to applications which use data transformation such as data analytics. However, the system and methods may also be used in conjunction with any application of visual programming. Figure 2 is a block diagram of a visual programming system 200, according to an example. The visual programming system 200 comprises a data processor 210, a storage medium 220, an execution module 230, an intelligent agent 240 and user interface 250. The storage medium 220, execution module 230, intelligent agent 240, and user interface 250 are communicatively coupled to the data processor 210. The data processor 210 is arranged to manage the flow of data and control communication between components of the visual programming system 200. The arrangement depicted in Figure 2 is one possible arrangement of components that may be used to implement the visual programming system described herein. The skilled person would understand that the visual programming system 200 may be realized by other hardware and software configurations. For example, components such as data processor 210, execution module 230 and intelligent agent 240 and their corresponding functions may be combined and executed by a general-purpose processor. Similarly, components may be realized on different types of hardware and software platforms. This includes, but is not limited to, cloud-based implementations, virtual machines, embedded systems, and specialized hardware such as GPUs or FPGAs for accelerated processing tasks. Returning to Figure 2, the storage medium 220 is arranged to store one or more predefined visual programming elements. The predefined visual programming elements may include programming elements such as variables, conditional statements, iterative statements, arithmetic and logic operations and any other common programming elements. In some examples, the storage medium 220 also stores predefined configurations of visual programming elements from the user interface 250. For example, in some cases users are able to store their own configurations of visual programming elements in the storage medium 220 or retrieve configurations generated by other users. The execution module 230 is arranged to translate a configuration of visual programming elements from the user interface 250, into executable instructions. In some cases the execution module 230 may convert a configuration of visual programming elements into an intermediate representation such as an abstract syntax tree (AST). The intelligent agent 240 is arranged to facilitate interaction with the other components of the visual programming system 200, via a trained language model 260. Language models are powerful machine learning models which exhibit a diverse range of capabilities. Language models, such as those based on transformer architectures, are built on deep neural networks which are trained on broad datasets, enabling them to understand nuanced natural language statements. The input to a language model may be structured via prompt engineering, so that instructions are interpreted by the language model in a particular way in order to control the output. In addition, language models may be trained to interpret and output computer code, translate between programming languages, generate code based on natural language descriptions and explain the purpose of code. Some language models also exhibit multimodal capabilities enabling them to convert from one format such as image data to a second format such as text data. Other intelligent agent systems may incorporate forms of machine learning which are not based on natural language queries and prompt engineering. Such intelligent agent systems may exhibit general intelligence capabilities in a similar manner as a trained language model. The skilled person would understand that these systems could be used as an alternative to the intelligent agent 240 and could be configured to perform a similar role as the trained language model 260. The intelligent agent 240 further comprises one or modules 270 which facilitate interaction with the language model 260 and other components of the visual programming system 200. In the example shown in Figure 2 the intelligent agent 240 comprises four modules. In other examples, more or less modules may be present. Each module may be preconfigured to execute selected functions from a set of functions, in conjunction with the language model 260. A user interacts with the visual programming system 200 via the user interface 250. The user interface 250 may comprise a GUI. Figure 3 shows an example 300 of such a GUI. Similarly to the GUI 100 shown in Figure I, the GUI 300 comprises a workspace 310 which provides an area for a user to perform visual programming, by connecting visual programming elements, and an area 320 displaying a library of the pre-defined visual programming elements, which correspond to elements stored in the storage medium 210. The interfaces described herein may comprise additional visual features such as drop-down menus, toolbars, windows and tabs. Such features may provide the user with a wide variety of options and tools to perform various functions within the interface. The manipulation of visual programming elements may be performed via, for example, a user input device connected to the visual programming system 200. The user input device may include pointing devices such as a mouse, touchpad, touch screen, or similar. In other examples, the user input device may include audio or voice recognition systems or similar. The GUI 300 further comprises a dialogue window 330. The dialogue window 330 provides an input space in which a user may engage in a text-based dialogue session with the language model 260, via prompt engineering. The dialogue window may also display text outputs generated by the language model. In some cases the dialogue window 330 may include features to communicate voice commands to the language model, or upload data in other formats such as image data. In some cases, the output may be displayed in separate window from dialogue window 330, or even split across multiple user interfaces. Referring again to Figure 2, one or more of the modules 270 may be arranged to analyse programs represented by configurations of visual programming elements in the workspace, using the language model 260. This may include using the language model 260 to interpret a configuration of visual programming elements, evaluate a program executed by a configuration of elements, identify inconsistencies in a configuration and reconfigure a configuration of visual programming elements, for example, in response to identifying inconsistencies. In one example, one of the modules 270 may be configured to compose programs using visual programming elements in the workspace. This may include using the language model 260 to generate a configuration of visual programming elements based on a natural language description of a program, provided, for example, by a user. In another example, the module may use the language model 260 to modify a configuration of visual programming elements based on a natural language query. In another example, the module may use the language module 26- to generate a natural language description of a modification or alteration to a configuration of visual programming elements in the workspace, either as a result of an edit made by a user or as a result of the intelligent agent making an alteration in the workspace. This process is strongly facilitated by providing a user with immediate feedback on the behaviour of the program represented by the configuration. In a further example, one of the modules 270 may be configured to perform translation functions. For example, the module may be configured to use the language model 260 to translate from a configuration of visual programming elements into code in a target programming language. In another example, the module may translate code in the target programming language into a configuration of visual programming elements in the workspace. In a further example, the module may be configured to augment visual programming elements in the workspace with code in the target programming language. These functions may be combined so that the whole configuration of visual programming elements is translated into code in a target programming language, edited in the target programming language, then converted back to a reconfigured configuration of visual programming elements. In a further example, one of the modules 270 may be configured to perform recommendations. A user-generated descriptive label, describing the intent or purpose of a configuration of visual programming elements may be embedded by the module as a vector in an embedding space. In response to a natural language query, comprising a description of a desired intent or purpose of a program, the module may similarly represent the query as a vector in the embedding space, and compare the vector with previous vectors. This may be used to identify stored configurations of visual programming elements which have similar functionality to the desired functionality. In addition to recommending configurations for a user to select, the module may prompt the language model such that when the language model generates new configurations, for example, in response to user instructions, the new configurations are consistent with existing configurations. This may help with future understandability and reduce disorder in configurations. The modules described herein, and the functions described in connection with these modules serve to illustrate the different kinds of functions that may be performed in the visual programming system 200 in conjunction with the trained language model 270. The skilled person would understand that a trained large language model can perform a wide variety of functions and, in particular, may be configured to perform other kinds of functions from those described. For example, a language model may be configured to execute functions which are specific to a particular use-case or application of a visual programming system. Figure 4 is a block diagram of a method 400 for interacting with the visual programming system 200 described herein. At block 410 the method 400 comprises displaying, in the workspace of the user interface, a configuration of visual programming elements. At block 420, an input comprising a natural language query is received, for example, in a dialogue window of the GUI. At block 430, the language model evaluates the natural language query. At block 440, the intelligent agent interacts with the visual programming system using the one or more modules, based on the content of the query. The visual programming system described herein may be used in conjunction with condition monitoring systems. Figure 5 is a diagram showing an example of a condition monitoring system 500. Figure 5 depicts a collection of assets 5 10. The assets in the collection of assets 5 10 may include different kinds of industrial asset as used in various industries including, but not limited to: machine tools such as lathes, milling tools, drilling tools; industrial robots such as welding robots, inspection robots, testing and validation robots; production line equipment such as belt conveyers, roller conveyers, packaging machines, pick and place machines, sorting machines, or supervisory systems and control systems such as distributed control systems, supervisory and data acquisition (SCADA) systems, Programmable logic control (PLC) systems, robotic control systems. An asset may also refer to a part of a device or system, such as an electronic panel, a transformer, a drum, a filter, a generator, pump, a belt, a solar panel, a rotary feeder, a scale, a water jacket, a compressor, a gearbox, a car, a bearing, a lubrication system, asset exterior, a power supply, a clamping unit or any other component which commonly occurs as part of a device or system in an industrial setting. Each asset in the collection 5 10 is monitored via the condition monitoring system 500. Data is obtained via monitored data streams from the assets 510. In some cases, a data stream may comprise time series data for a continuous variable measured via one or more sensors connected to the asset. Sensors may include: temperature, pressure, humidity, optical, motion sensors or any other types of sensor. Data may also be obtained from Internet of Things (loT) devices such as smart devices or other remote monitoring systems. In some cases a data stream may represent a discrete variable such as a mode of operation of an asset, a program, a type of part being manufactured by an asset, or machine state data such as on / off. The data obtained from assets 510 may be communicated over local networks within, for example, an industrial environment, before being communicated over an external network 520. For example, data may be communicated locally over a Local Area Network (LAN), wireless sensor networks, industrial ethernet or Internet of Things (loT) network, before being communicated to the external network 520, for example, via a server (not shown in Figure 5). The network 520 may be the internet, or another wide area network (WAN), wireless LAN, cellular network or any other kind of network. In Figure 5, the collection of assets 510 is monitored remotely by server 530, via network 520. The server 530 may comprise a network interface to facilitate communication via network 520 with the collection of assets 510. Data may be received via network interface from the from monitored data streams associated with the assets 510. The server 530 may comprise one or more data processing units to process data received via the network interface and a memory to store instructions that may be implemented by the data processing unit(s). The server 530 is communicatively coupled to data storage 540. Data from monitored data streams and other information related to the assets 5 10 may be stored in the data storage 540 for subsequent use by the server 530. A client device 550 connects with the computing system 530, via network 520. The client device 550 may be a user device such as a desktop, laptop, tablet, smartphone, thin client or similar. In examples described , the visual programming system 200 may be implemented on client device 550 in conjunction with the server 530. In particular the user may be able to interact with user interface 250 through the client device 550 using an application hosted on server 530, that implements that visual programming system 200. The server 530 may execute instructions output by the visual programming system to transform data for data analytics operations on data received from assets 510, for example. The present disclosure is described with reference to flow charts and / or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. In some examples, some blocks of the flow diagrams may not be necessary and / or additional blocks may be added. The present inventions can be embodied in other specific apparatus and / or methods. The described embodiments are to be considered in all respects as illustrative and not restrictive. In particular, the scope of the invention is indicated by the appended claims rather than by the description and figures herein. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

I. A visual programming system, comprising:a storage medium to store one or more predefined visual programming elements;a user interface comprising a workspace to perform visual programming based on the one or more predefined visual programming elements;an execution module to translate a configuration of visual programming elements in the workspace into executable instructions; andan intelligent agent comprising:a trained language model; andone or more modules to facilitate interaction between the trained language model and the visual programming system,wherein, responsive to an input comprising a natural language query, the language model evaluates the natural language query and interacts with the visual programming system using the one or more modules, based on the content of the query.

2. The system of claim I, wherein the system is arranged to store the configuration in the storage medium.

3. The system of claim I or 2, wherein the user interface comprises a user input feature to enable a user to assign labels to configurations of visual programming elements.

4. The system of claim I or 2, wherein the user interface comprises a user input feature to enable a user to input dialogue to the language model.

5. The system of claim I to 4, wherein each of the one or more modules is associated with a set of functions, and, wherein each of the modules is arranged to perform one or more selected functions from its associated set.

6. The system of claim 5 wherein a set of functions comprises functions to:a) embed a label of a configuration of visual programming elements as a vector in an embedding space;b) embed a natural language description of a program as a vector in the embedding space;c) embed an abstract syntax tree of a configuration of visual programming elements as a vector in the embedding space;d) compare vectors in the embedding space; ore) identify stored configurations of visual programming elements usingnatural language descriptions of programs, based on the comparison of corresponding vectors in the embedding space.

7. The system of claim 5 wherein a set of functions comprises functions to:a) interpret a configuration of visual programming elements;b) evaluate a program executed by a configuration of visual programming elements;c) identify inconsistencies in a configuration of visual programming elements; ord) reconfigure a configuration of visual programming elements.

8. The system of claim 5 wherein a set of functions comprises functions to:a) generate a configuration of visual programming elements based on a natural language description of a program;b) modify a configuration of visual programming elements based on a natural language query; orc) provide a natural language description of a modification to a configuration of visual programming elements.

9. The system of claim 5 wherein a set of functions comprises functions to:a) translate a configuration of visual programming elements into code in a target programming language;b) translate code in the target programming language into a configuration of visual programming elements in the workspace; orc) augment visual programming elements in the workspace with code in the target programming language.

10. A condition monitoring system, comprising:a data processing system arranged to monitor a plurality of data streams associated with one or more industrial asset;a data storage device arranged to store data received by the data processing system from the one or more assets; andthe visual programming system according to any one of claims I to 9,wherein the data processing system is arranged to access data stored on the data storage device and execute instructions output by the visual programming system on the data.II. A computer-implemented method for interacting with the visual programming system of any one of claims I to 9, the method comprising:displaying, in the workspace of the graphical user interface, a configuration of visual programming elements;receiving, an input comprising a natural language query;evaluating, by the language model, the natural language query; andfacilitating interaction, via the intelligent agent, with the visual programming system using the one or more modules, based on the content of the query.

12. A computer program product comprising instructions to cause the system ofclaims I to 9 to execute the steps of the method of claim I I.

Images