Device, system and method for providing visualization program for machine vision system

Abstract

The technology includes a system, a computerized method, and a computer-readable medium for creating a graphical program in a graphical program development environment. In the graphical program, a spreadsheet node having an input terminal is instantiated. The spreadsheet node is associated with a spreadsheet that specifies a list of functions to be executed in a computing device; and the input terminal is connected to a first terminal of a first node, which indicates a data connection between the first terminal of the first node and the input terminal of the spreadsheet node. The input terminal of the spreadsheet node is associated with a first cell in the spreadsheet, which indicates that the first cell in the spreadsheet is filled with any data received by the input terminal. A human-readable file is generated, specifying the graphical program including the spreadsheet node.

Description

[0001] Divisional application

[0002] This application is a divisional application of application number 2017107128743, application date August 18, 2017, and entitled "Device, system and method for providing visualization program for machine vision system". Technical Field

[0003] Disclosed herein is an apparatus, system, and method involving programming a machine vision system using a graphical program. Background Art

[0004] Machine vision systems are becoming a popular tool to replace human vision in a wide range of applications such as manufacturing operations. Machine vision systems typically provide automated, computer-based image acquisition and analysis capabilities that can be used for tasks such as measurement and inspection of parts or materials. For such tasks, machine vision systems are typically configured with a camera for acquiring images of an object of interest (e.g., a part being produced) and also configured with processing functionality that processes the acquired images and generates information about the object. The camera and / or processing functionality can be programmed to adapt the operation of the machine vision system (including the camera and / or processing functionality) to the application of interest.

[0005] As machine vision systems become more common in a variety of applications, it is often desirable that machine vision systems be usable and programmable by users with little (or no) programming expertise. Therefore, it is desirable to provide a program development environment for machine vision systems that is suitable for a wide range of users. Summary of the invention

[0006] According to the disclosed subject matter, an apparatus, system, and method for programming a machine vision system using a graphical program are provided.

[0007] Some embodiments include a computer-implemented method for creating a graphical program in a graphical program development environment. The method includes providing a graphical program in a graphical program development environment, wherein the graphical program includes a first node, the first node having a first terminal. In the graphical program, instantiating a spreadsheet node, the spreadsheet node having an input terminal, wherein the spreadsheet node is associated with a spreadsheet that specifies a list of functions to be executed in a computing device; and connecting the input terminal to a first terminal of the first node, which indicates a data connection between the first terminal of the first node and the input terminal of the spreadsheet node. The method includes associating the input terminal of the spreadsheet node with a first cell in the spreadsheet, which indicates that the first cell in the spreadsheet is filled with any data received by the input terminal. The method includes generating a human-readable file specifying a graphical program including a spreadsheet node.

[0008] Some embodiments include a system for creating a graphical program in a graphical program development environment. The system includes a processor in communication with a memory, wherein the processor is configured to run a computer program stored in the memory, the computer program being configured to: provide a graphical program in a graphical program development environment, wherein the graphical program includes a first node, the first node having a first terminal. The program is configured to instantiate a spreadsheet node in the graphical program, the spreadsheet node having an input terminal, wherein the spreadsheet node is associated with a spreadsheet specifying a list of functions to be executed in a computing device; and connect the input terminal to a first terminal of the first node, which indicates a data connection between the first terminal of the first node and the input terminal of the spreadsheet node. The program is configured to associate the input terminal of the spreadsheet node with a first cell in the spreadsheet, which indicates that the first cell in the spreadsheet is filled with any data received by the input terminal. The program is configured to generate a human-readable file specifying a graphical program including a spreadsheet node.

[0009] Some embodiments include a non-transitory computer-readable medium having executable instructions, the medium being associated with a system for creating a graphical program in a graphical program development environment. The instructions are operable to cause the system to provide a graphical program in the graphical program development environment, wherein the graphical program includes a first node, the first node having a first terminal. The instructions are operable to cause the system to configure the program to instantiate a spreadsheet node in the graphical program, the spreadsheet node having an input terminal, wherein the spreadsheet node is associated with a spreadsheet specifying a list of functions to be executed in a computing device; and connect the input terminal to a first terminal of the first node, which indicates a data connection between the first terminal of the first node and the input terminal of the spreadsheet node. The instructions are operable to cause the system to associate the input terminal of the spreadsheet node with a first cell in the spreadsheet, which indicates that the first cell in the spreadsheet is populated with any data received by the input terminal. The instructions are operable to cause the system to generate a human-readable file specifying a graphical program including a spreadsheet node.

[0010] In some embodiments, a human-readable file is sent to a controller, and the controller is configured to execute a graphical program specified in the human-readable file. An execution attribute of a spreadsheet node may be received, which indicates a computing device designated to execute computer-readable instructions of the spreadsheet node. An execution attribute of a first node may be received, which indicates a computing device designated to execute computer-readable instructions of the first node. Based on the execution attribute of the spreadsheet node, a determination may be made at the controller to designate a computing device to execute the computer-readable instructions of the spreadsheet node; and the computer-readable instructions of the spreadsheet node may be transmitted to the computing device to execute the computer-readable instructions of the spreadsheet node.

[0011] In some embodiments, upon receiving a request to view a spreadsheet associated with a spreadsheet node, displaying the spreadsheet in a graphical program development environment; receiving first data associated with a second cell in the spreadsheet; and populating the second cell in the spreadsheet with the first data. The computing device may be a camera. The graphical program may be configured to perform a machine vision task. The computer-readable instructions of the spreadsheet node may be designed to perform a machine vision analysis of an image. An input terminal of the spreadsheet node may be configured to receive an image from a first terminal of the first node.

[0012] In some embodiments, a second spreadsheet node is provided in the graphical program, wherein the second spreadsheet node is associated with a second spreadsheet designed to be executed at another computing device. The spreadsheet node may include an output terminal, wherein the output terminal is associated with a second cell in the spreadsheet, which represents an output of the computer-readable instructions of the spreadsheet node. The output terminal of the spreadsheet node may be connected to a terminal of the second node of the graphical program to provide output to the terminal of the second node of the graphical program.

[0013] In some embodiments, a visualization interface having a graphical element is provided, and the graphical element is associated with a second cell in the electronic table to display the content of the second cell at the graphical element. The content of the second cell may include an image, and wherein associating the graphical element with the second cell includes assigning a pointer to the graphical element that references the image. A portion of the electronic table may be displayed in the graphical element of the visualization interface. A request to modify a cell of the electronic table may be received using the visualization interface.

[0014] In some embodiments, computer-readable instructions of a spreadsheet node are executed at a computing device, wherein executing the computer-readable instructions includes: receiving a pointer to first data to be processed by the computing device; determining whether the computing device maintains the first data locally at the computing device; when it is determined that the first data is maintained locally at the computing device, processing the first data, and when it is determined that the first data is not maintained locally at the computing device, retrieving the first data from a storage medium referenced by the pointer and processing the first data. The computing device may include a plurality of computing modules, and wherein the method further includes automatically selecting one or more computing modules to execute the computer-readable instructions of the spreadsheet node. Automatically selecting the one or more computing modules may include determining a computing load of the one or more computing modules.

[0015] Thus, there has been outlined rather broadly the features of the disclosed subject matter in order that the detailed description that follows may be better understood, and in order that the present contribution to the art may be better appreciated. Of course, additional features of the disclosed subject matter will be described hereinafter, and will form the subject matter of the appended claims. It should be understood that the phraseology and terminology used herein are for descriptive purposes only and should not be regarded as limiting. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Various objects, features, and advantages of the disclosed subject matter may be more fully understood by referring to the detailed description of the disclosed subject matter considered in conjunction with the following drawings, wherein like reference numerals represent like elements.

[0017] Figure 1 An exemplary visual program development environment for a machine vision system according to some embodiments is shown;

[0018] Figure 2 An exemplary visual program development environment in a graphical environment is shown according to some embodiments;

[0019] Figure 3 shows an exemplary graphical interface for programming a visualization program according to some embodiments;

[0020] Figure 4 shows a spreadsheet interface shown in a graphical interface according to some embodiments;

[0021] Figure 5 shows a spreadsheet interface shown in a graphical interface according to some embodiments;

[0022] Figure 6 illustrates an exemplary program panel of a graphical interface for a visualization program with multiple cameras according to some embodiments;

[0023] Figure 7 An exemplary program panel of a graphical interface for visualizing multiple nodes (each using a spreadsheet) for a single task for the same camera is shown according to some embodiments;

[0024] Figure 8 An exemplary program panel of a graphical interface for visualizing a plurality of nodes (each using a spreadsheet) for a plurality of cameras according to some embodiments is shown;

[0025] Fig. 9 shows an exemplary menu for publishing input and / or output of a spreadsheet according to some embodiments;

[0026] Figures 10A-10C illustrates an exemplary object of a spreadsheet node according to some embodiments; and

[0027] Fig.11 is an exemplary list of objects corresponding to cells in a spreadsheet according to some embodiments. DETAILED DESCRIPTION

[0028] In the following description, many specific details about the systems and methods of the disclosed subject matter and the environments in which these systems and methods can operate are set forth in order to provide a thorough understanding of the disclosed subject matter. However, it will be apparent to those skilled in the art that the disclosed subject matter can be practiced without such specific details, and certain features known in the art are not described in detail to avoid complicating the disclosed subject matter. In addition, it should be understood that the examples provided below are exemplary, and it is contemplated that other systems and methods within the scope of the disclosed subject matter exist.

[0029] As machine vision systems become more common in a variety of applications, it is often desirable that machine vision systems be used and programmed by users who do not have programming expertise. Therefore, it is desirable to provide a program development environment for machine vision systems that is applicable to a large number of users.

[0030] One such exemplary program development environment is a visual program development (VPD) environment. A VPD environment allows a user to develop a portion of a computer program, or even an entire computer program, by graphically manipulating visual elements rather than by specifying them textually.

[0031] Visualization programs developed by the VPD environment can graphically represent data flows by using various nodes and the interconnections between these nodes. In the visualization program, nodes can be represented by boxes and / or other shapes. In the visualization program, interconnections can be represented by lines. Each node can represent, for example, an element of a machine vision system (e.g., a camera) or an operation performed by the machine vision system (e.g., an acquisition action).

[0032] The visualization program can configure inputs and outputs for each node. For example, each node can include one or more input terminals to configure data input to the node, and / or one or more output terminals to configure data output from the node. The inputs and outputs can be represented graphically in the visualization program, such as by using visual objects extending from one side of the node (such as Figure 2 and Figure 3 As shown, for example, small circles are connected to nodes by small lines).

[0033] Interconnects can be used to connect the inputs and outputs of various nodes of a graphical program to construct a graphical program to indicate the flow of data between nodes. For example, an output terminal of a first node can be connected to an input terminal of a second node via a first interconnect to configure the program so that the second node receives (as input) output data from the output terminal of the first node. Similarly, an output terminal of a second node can be connected to an input terminal of a third node via a second interconnect to configure the graphical program so that the second node provides data to the input terminal of the third node.

[0034] A user can assemble a visual program by placing one or more nodes into the VPD environment and connecting one or more nodes using interconnections. For example, if a user wants to perform the addition of two numbers, the user can select a node corresponding to the addition operation and place the node into the VPD environment. The "addition" node has two input terminals and one output terminal, the two input terminals are configured to receive two numbers that should be added at the "addition node" (also called an operand), and the output terminal provides the result of the addition operation. Therefore, the user can use the interconnection to add the two numbers at the input terminals of the "addition node". The output terminal will contain the result of the addition. In the graphical program, the user can use the output of the "addition node" to further process the result of the addition.

[0035] A potential weakness of the VPD environment is that when it is necessary to design complex applications, such as machine vision applications, the various operations that nodes can perform are limited. For example, when a user wants to perform a specific operation and the VPD environment does not provide that operation, it may be the case that the VPD environment can only be configured to provide a limited number of node types that are dedicated to a limited variety of operations, such as addition, subtraction, and multiplication. Therefore, the flexibility and customizability of developing programs in the VPD environment may be unintentionally limited.

[0036] Some VPD environments attempt to increase flexibility by providing nodes that represent custom text programs. For example, a node, sometimes called a "script" node, can be associated with a script program designed to perform a specific operation. Thus, when a user wants to deploy a node that can perform a specific operation that is not provided by the VPD environment, the user can develop an actual software (or script) program for that specific operation and associate the script program with a script node so that the script node can perform the specific custom operation.

[0037] Unfortunately, since using the script node requires the ability to write scripts, most novice users cannot take advantage of the flexibility that the script node provides. The purpose of the VPD environment is to provide a program development environment that allows you to develop programs even without specialized knowledge of writing programs. In a sense, using the script node defeats the purpose of the VPD environment.

[0038] The technology described herein provides a new programming mechanism for increasing the flexibility of a VPD environment. In some embodiments, a VPD environment may include a spreadsheet node. As further described herein, a spreadsheet node may be associated with a spreadsheet that includes one or more cells configured to perform various operations. For example, a spreadsheet node may be used to specify a specific way to process an input image; as another example, a spreadsheet node may be used to specify a specific way to manipulate an input matrix to create an output matrix.

[0039] As opposed to script nodes, the benefit of using a spreadsheet node is that the spreadsheet associated with the spreadsheet node can be easily programmed. For example, a user can use a spreadsheet program to build a spreadsheet for performing a specific function. As described herein, the spreadsheet program of the spreadsheet node can be designed to be easy to learn and simple to operate, so that even novice computer users can easily program the spreadsheet. In addition, the spreadsheet program can provide built-in functions that can be easily combined or instantiated to perform the desired operations in the spreadsheet. Therefore, compared to script nodes, which typically require significant programming experience, spreadsheet nodes can achieve a better balance between the flexibility of the node and the simplicity of use or accessibility of the node.

[0040] Figure 1 An exemplary computing system capable of using a VPD environment in accordance with some embodiments is shown. System 100 may include a host device 102, a central controller 104, one or more computing devices 106, and one or more display devices 108.

[0041] In some embodiments, the host device 102 may include a processor 110, a memory device 112, and a visual program development (VPD) environment module 114. The processor 104 may execute instructions, and one or more memory devices 106 may store instructions and / or data. The memory device 106 may be a non-temporary computer-readable medium, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a disk drive, an optical drive, a programmable read-only memory (PROM), a read-only memory (ROM), or any other memory or combination of memories. The memory device 106 may be used for temporary storage of data. The memory device 106 may also be used for long-term data storage. The processor 104 and the memory device 106 may be supplemented and / or incorporated into a dedicated logic circuit.

[0042] As further described herein, the VPD environment module 114 can be configured to provide a graphical environment in which a user can develop a visualization program. Thus, a user can use the graphical environment of the VPD environment module 114 to develop a visualization program for the computing system 100. Once the user defines the visualization program, the visualization program can be launched in the computing system 100. For example, the VPD environment module 114 can package the visualization program as a project file and provide the project file to the central controller 104 via the interface 116 so that the central controller 104 can launch or execute the visualization program in the project file.

[0043] In some embodiments, the central controller 104 may include a processor 118, a memory device 120, and a visual program execution (VPE) module 122. The VPE module 122 may be configured to "decompress" a project file to retrieve a visual program and prepare to execute the visual program. To this end, as further described below, the VPD module 122 may retrieve a pre-existing machine-readable program as defined in the visual program. Based on the definition of the visual program, the central controller 104 may assign one or more operations to one or more computing devices 106. The computing device 106 may be, for example, a camera and / or a smart camera. The smart camera may include, for example, a processor and a memory so that the smart camera may be configured to execute one or more portions of a program designed using the VPD environment 114. In addition, based on the definition of the visual program, the central controller 104 may cause one or more display devices 108 to display data. In some embodiments, the central controller 104 may be part of the host device 102. For example, the VPE module 122 may be part of the host device 102.

[0044] In some embodiments, interface 116 may be implemented in hardware to send and receive signals in a variety of media (eg, optical, copper, and / or wireless interfaces) and in a number of different protocols, some of which may be non-transitory.

[0045] In some embodiments, the VPD environment module 114 and the VPE module 112 may be implemented in software. The software may be run on a processor 104 or processor 118 that is capable of executing computer instructions or computer code. The processor 104 and processor 118 may be implemented in hardware by using an application specific integrated circuit (ASIC), a programmable logic array (PLA), a digital signal processor (DSP), a field programmable gate array (FPGA), or any other integrated circuit. Processors 104 and processors 118 suitable for executing computer programs include, for example, general and special purpose microprocessors, digital signal processors, and any one or more processors of any kind of digital computer. Typically, the processor 104 and processor 118 receive instructions and data from a read-only memory or a random access memory or both.

[0046] The VPD environment module 114 and the VPE module 112 may be implemented in digital electronic circuitry or in computer hardware, firmware, software, or a combination thereof. The implementation may be a computer program product (e.g., a computer program tangibly embodied in a machine-readable storage device) for performing or controlling operations of a data processing apparatus (e.g., a programmable processor, a computer, and / or multiple computers). The computer program may be written in any form of computer or programming language, including source code, compiled code, interpreted code, and / or machine code, and the computer program may be deployed in any form, including as a stand-alone program or subroutine, component, or other unit suitable for a computing environment. The computer program may be deployed to execute on one computer or on multiple computers at one or more sites.

[0047] Figure 2 An exemplary visualization program 200 development environment is shown in accordance with some embodiments in the graphical environment of the VPD environment module 114. As described herein, a visualization program may include one or more nodes connected by one or more interconnects. Figure 2 The exemplary visualization program 200 shown includes three nodes: an image acquisition node 202, an image processing node 204, and a display node 206. As described herein, each node may include one or more input terminals for receiving data and one or more output terminals for outputting data. For example, the image processing node 204 includes an input terminal 208 and an output terminal 210.

[0048] A node may not have any input terminals. For example, the image acquisition node 202 has only an output terminal 214, for example, because the data processed by the image acquisition node 202 can be defined by the node itself rather than by one or more inputs. For example, the image acquisition node 202 can acquire an image using a camera, or the image acquisition node 202 can retrieve data from a storage device. Similarly, a node may not have any output terminals. For example, the display node 206 has only an input terminal 216. In this case, for example, the data manipulated by the display node 206 can be displayed on a display device and / or stored in a storage device. For example, the display node 206 can cause the data received at the input terminal 216 to be displayed on a display device such as Figure 1 108. The nodes may be connected by interconnects (eg, interconnects 218, 220) to identify data flows between the nodes.

[0049] In some embodiments, the visualization program is executed according to the data flow identified by the node and the interconnection. For example, when the image acquisition node 202 receives or creates data (e.g., by defining a program inside the node 202, such as capturing an image by a camera or a smart camera), the image acquisition node 202 can process the data and provide the first output data to the output terminal 214. Then, the output terminal 214 provides the first output data to the input terminal of another node, such as the input terminal 210 of the image processing node 204, through the interconnection 218. The image processing node 204 can process the first output data (e.g., capture an image) received at the input terminal 210 to generate the second output data (e.g., processed image data). The image processing node 204 can then provide the second output data to the output terminal 212, which is then received by the input terminal 216 of the display node 206. Since the function of the visualization program is defined by the function of the node and the interconnection between the nodes, the visualization program can intuitively convey the function of the program during the development process.

[0050] Figure 2 A simple example of a visualization program is shown. A person skilled in the art must understand that a visualization program can be more Figure 2 The visualization programs shown are much more complex, such that they include a large number of nodes and interconnections. In addition, other aspects of the visualization program can be configured, such as whether one or more nodes are to run in parallel with other nodes, if the nodes are to be executed sequentially (e.g., designed to be from left to right in the visualization program), and / or the like. For example, in some embodiments, data transmission between nodes can be asynchronous: a node can transmit data to another node at any time (e.g., as soon as the data is available). In other embodiments, data transmission between nodes can be synchronous. For example, at predetermined instances of a clock cycle, a node can transmit data to another node.

[0051] Figure 3A graphical interface of an exemplary VPD environment module for programming a visualization program according to some embodiments is shown. The graphical interface 300 may include a program panel 302, in which a user designs a graphical program by instantiating various nodes and / or interconnections. The graphical interface 300 may also include an explorer selection bar 304, which has a tree of device and / or feature selection for the visualization program, including subsystems in the VPD environment. These components in the explorer selection bar 304 include: script blocks (e.g., blocks that allow users to create custom scripts), hardware devices (e.g., cameras, smart cameras), recipes (e.g., a collection of tag data that can be loaded to customize applications for inspecting similar components with different requirements), tag managers (e.g., a collection of unique global customization values ​​for sharing data between all subsystems in an application at runtime), tasks (e.g., a graphical representation of a sequence of interconnected blocks), and web pages (e.g., a development user interface). For example, the device component in the explorer selection bar 304 displays the currently available hardware devices. For example, the device component can display which cameras are physically connected to the system, and whether the input / output (IO) hardware is recognized and started. The device component may also provide access to communication facilities such as TCP / IP, discrete IO, and / or the like.

[0052] When a user selects an element from the Explorer portion, the VPD can be configured to cause the VPD environment to change to correspond to the content provided by the subsystem. The user can select an item in the Explorer selection bar 304 and right-click in the program panel 302 to create a corresponding block in the graphics program. For example, the user can click on a task in the Explorer selection bar 304 and right-click in the program panel 302 to create a new task block for the graphics program. When the VPD environment creates a new task block, the toolbox 306 will change to tools suitable for the newly created task. For example, the user can click on a task in the Explorer selection bar 304 and right-click in the program panel 302 to create a new task block for the graphics program. This will change the toolbox 306 to display tools suitable for the web page.

[0053] In some examples, for certain components and / or devices in the explorer selection bar 304, the user can double-click the component to change the associated properties of the component. For example, if the user selects "Camera 0", the user can double-click it to configure any acquisition properties. In some embodiments, if the user double-clicks an item in the explorer selection bar 304, the user is taken to the corresponding tab 305 in the editor to allow the user to edit the selected item.

[0054] The graphical interface 300 may also include a toolbox 306 having a graphical tree for selecting one or more nodes and / or structures for the program panel 302. In some embodiments, the toolbox 306 is populated with appropriate tools given the component or subsystem being edited in the program panel 302. For example, if the user is editing a task (e.g., Figure 3 If the user is editing a web page (e.g., by selecting the web page tab 305B), the toolbox 306 is populated with controls that can be placed into a web page for creating a custom HMI.

[0055] For example, Figure 3 The nodes shown may include an acquire node 308 and an In-Sight Vision Tool Block node 310, which may be selected to be added to the program panel 302 using the toolbox 306. The program panel 302 also includes an interconnect 316 between the output of the acquire node 308 and the input of the In-Sight Tool Block 310. In addition, the toolbox 306 may allow the user to perform other functions, such as adding notes. The program panel 302 includes three annotations: Annotation 312A indicates that the visualization program is for a single camera scene; Annotation 312B indicates that the acquire node 308 (or block) performs data acquisition from the camera; Annotation 312C indicates that the In-Sight Tool Block node 310 (or block) performs vision on the camera (e.g., runs various vision algorithms for the acquired or input images). For example, the In-Sight Tool Block is a node that represents the execution of a spreadsheet. As further explained in detail herein, it may have an input terminal connected to an input cell, and it may have an output cell connected to an output terminal.

[0056] As another example, toolbox 306 allows the user to select parallel blocks. In some embodiments, any block located inside the parallel block in program panel 302 will cause the internal blocks to be executed in parallel with each other. Without parallel blocks, in some embodiments, the blocks in the task sequence diagram are executed continuously from left to right. Therefore, parallel blocks can improve the performance of the system by making the blocks run simultaneously in different threads. For example, using parallel blocks can make the entire task sequence complete faster.

[0057] Graphical program interface 300 may also include a property panel 314 for modifying the properties of an object selected in program panel 302. Figure 3302, annotation 312C is selected, such as by a darker outline around annotation 312C than the rest of the object outline. By selecting annotation 312C, the graphical interface allows the user to modify the properties of annotation 312C, i.e., to set the annotation to "this block performs vision on camera." Thus, graphical interface 300 may enable a user to add, configure, and modify multiple nodes and interconnections between nodes to create a visualization program.

[0058] In some embodiments, graphical interface 300 may enable a user to specify a computing device for performing operations associated with a node. For example, when a user selects a node, a property pane may be provided to the user in which the user may specify a computing device for performing the operations associated with the node.

[0059] In some embodiments, the graphical interface 300 may provide a spreadsheet node. A spreadsheet node is a node associated with a spreadsheet such that the node is configured to perform one or more operations as defined in the spreadsheet. Thus, a spreadsheet node may be used to incorporate spreadsheet functionality into a graphical programming environment. Specific operations of a spreadsheet may be defined by a user using a spreadsheet interface displayed to the user. U.S. Patent No. 7,107,519, issued on September 12, 2006, entitled "Spread-Sheet Based User Interface Creation," has been assigned to Cognex Corporation and describes an exemplary spreadsheet interface for programming a spreadsheet, the disclosure of which is incorporated herein by reference in its entirety.

[0060] A spreadsheet interface associated with a spreadsheet node can display the cell rows and columns of a spreadsheet. The contents of the cells can be specified by the user during an option specification process and can include, for example, adding or specifying functions, mathematical expressions, data labels, and / or data to be processed based on the desired operation. In addition, input and / or output functions can be placed in cells, such as input functions reading information from hardware devices and output functions interacting with specialized hardware devices. Together, cell expressions in the spreadsheet define portions of the visualization program associated with the spreadsheet node. Due to the flexibility of the spreadsheet program, the spreadsheet interface can be used to specify and / or develop custom functionality for a machine vision system that is not provided by a graphical programming environment.

[0061] In some embodiments, when a user selects a spreadsheet node (e.g., a user double-clicks a spreadsheet node, such as from Figure 3 When the introspection tool block 310 is displayed, the graphical interface of the VPD environment module 114 can provide a spreadsheet interface. Figure 4An example of a spreadsheet interface 402 shown in the graphical interface 300 of the VPD environment module 114 according to some embodiments is shown. As shown, the spreadsheet interface 402 may include a plurality of cells arranged in rows and columns. Figure 4 Columns A to S and rows 0 to 20 are shown.

[0062] As further described herein, the electronic form interface 402 can allow a user to configure inputs and outputs for associated nodes in a VPD environment. For example, any cell in the electronic form can be configured as an input or output of an associated node in the program panel 302. For example, a user can configure the electronic form to designate any cell as its input image, which the user can further process in the electronic form (e.g., by defining or using one or more functions in the electronic form). The electronic form can allow multiple inputs and / or multiple outputs. In addition, as further described herein, the electronic form can be configured to publish the value of any cell (e.g., the value set by the process and / or function in the electronic form) to the HMI to easily bind the HMI control to the cell in the electronic form.

[0063] The spreadsheet interface may be pre-configured with predetermined functions to assist in programming the spreadsheet. For example, U.S. Patent No. 9,123,093, issued on September 1, 2015, and entitled “Vision inspection programming method and apparatus,” assigned to Cognex Corporation, describes a spreadsheet interface that may include programming functions, the disclosure of which is incorporated herein by reference in its entirety. Toolbox 306 of graphical interface 300 may be configured to allow a user to select one or more predefined functions for a particular cell of spreadsheet interface 402. For example, Figure 4 The illustrated cell 3A includes a blob detection (DetectBlobs) function from the toolbox 306, which detects blobs in an image and sorts them by region, returning a blob structure, as explained in the comment section of the toolbox 306. A user may configure the inputs and outputs of the blob detection function by identifying cells in the spreadsheet interface 402. In another example, as further described herein, the spreadsheet interface may allow a user to add a script (e.g., JavaScript code) to a cell so that the script is executed when the spreadsheet is executed.

[0064] In some embodiments, an input terminal of a spreadsheet node may be used to provide cell content to a cell associated with the input terminal. Figure 4As shown, when the spreadsheet node receives data at the input terminal, the received data is provided as the contents of cell A0 associated with the input terminal. The spreadsheet can use the contents of cell A0 to process the data using the operation specified in the spreadsheet. For example, functions and operations can be added to the spreadsheet that use A0 as a parameter.

[0065] In some embodiments, the output terminal of the spreadsheet node can be used to retrieve cell contents from the spreadsheet for use by other nodes in the visualization program. Figure 4 As shown, when the spreadsheet provides the output of the operation to the cell associated with the output terminal, the spreadsheet node provides the output to the output terminal so that other nodes connected to the output terminal of the spreadsheet node can use the output.

[0066] In some embodiments, a single spreadsheet may include one or more cells designated as input terminals and / or one or more cells designated as output terminals of a spreadsheet node. For example, by right-clicking a cell in 402, a user can add entries for input and / or output to the property panel 314. By specifying a cell, the VPD environment can allow the user to select a menu item (e.g., a pop-up context menu pops up) to designate the cell as an input or output. In some embodiments, if a cell is designed to be an input or output to a spreadsheet, the associated cell will obtain a decoration (e.g., a small shadow in the lower right corner of the cell box, such as shown in cells A0 and E9 in the table), and the property panel 314 will automatically update the pin name and related information (e.g., input or output).

[0067] The program panel can display one or more input terminals and one or more output terminals of the spreadsheet node. For example, the property panel 314 displays a list of inputs and outputs of the spreadsheet. The image in cell A0 is defined as the input to the spreadsheet, and the output of one or more functions created in the spreadsheet interface 402 is filled into the spreadsheet in cell E9 as a string data type. By graphically linking to the output of the spreadsheet node (which will be filled into the value of cell E9 at runtime), subsequent nodes added to the graphical program can use the output on E9. The ability to define the input and / or output of a particular spreadsheet node through the cells of the spreadsheet creates a link between the spreadsheet and the graphical programming environment.

[0068] The data types of input and / or output may be predefined by the graphical programming environment. In some examples, the graphical programming environment may allow simple data objects to be used as input and output such as digital images, string data types, integer data types, double precision data types, etc. In some examples, the graphical programming environment may support complex data types, such as combinations of primitive data types organized in a certain manner. Complex data types may include, for example, lists of (x, y) data points, calibration information (e.g., matrix information, nonlinear data), and / or the like.

[0069] Although Figure 4 Only one input and output are shown, Figure 5 A spreadsheet node is shown that includes two input terminals and two output terminals according to some embodiments. As shown in the properties panel 314, the spreadsheet includes two inputs: (1) cell A0 corresponding to an image as an input to the spreadsheet, and (2) cell A15 corresponding to a double precision integer data type as an input to the spreadsheet. The spreadsheet also includes two outputs: (1) a string data type mapped to cell E9 for output, and (2) a double precision data type mapped to cell B3 for output.

[0070] In some embodiments, the graphical interface can automatically display and / or update the input and / or output terminals attached to a spreadsheet node (e.g., in program panel 302) based on cells identified as input terminals or output terminals in the spreadsheet interface. Thus, by designating cells as inputs or outputs, creation of input and output pins can be accomplished through spreadsheet interface 402. Once created through spreadsheet interface 402, the visual programming development environment can automatically update the display to display the input and output pins in program panel 302. For example, assuming that a user configures a spreadsheet to be associated with a spreadsheet node having one input terminal and one output terminal (e.g., as Figure 4 When the user returns to the program panel 302, the program panel 302 may graphically show one input terminal and one output terminal coupled to the spreadsheet node (e.g., conversely, the program panel 302 may not display any input or output prior to configuration).

[0071] The electronic form interface 402 may provide various techniques for inserting inputs and outputs. For example, a right-click menu may be provided to allow a user to publish an input pin and / or an output pin. Fig. 9An exemplary menu 900 for publishing inputs and / or outputs of a spreadsheet according to some embodiments is shown. A user can invoke the menu 900 by right-clicking a cell in a spreadsheet interface. The menu 900 includes a plurality of options for a particular cell in the spreadsheet interface to set the cell as an input pin or an output pin, and / or manipulate the cell. The option Publish to HMI 902 can select to publish the cell to the HMI (e.g., with the value of the cell displayed in the HMI as further explained herein). The option Publish as Input Pin 904 can select to publish the cell as an input pin (e.g., display the input pin on a corresponding block in a block diagram associated with the cell). The option Publish as Output Pin 906 can select to publish the cell as an output pin (e.g., display the output pin on a corresponding block in a block diagram associated with the cell). The option Unpublish 908 can select to unpublish the cell (e.g., delete the pin displayed on the block).

[0072] Each cell in the spreadsheet interface 402 may include one or more properties, such as for name, input, and output (e.g., as shown in the properties panel 314). In some embodiments, when the user publishes a cell as an input or output, the program selects a default name for the pin based on the cell properties and updates the cell's properties in the camera. If necessary, the "change" event signals that the corresponding block in the properties panel 314 and the program panel 302 is updated. For example, if the program determines that the block is a new input or output cell, the program publishes the corresponding pin of the block in the graphical diagram in the program panel 302.

[0073] Once the VPD environment module 114 receives the visualization program, the VPD environment module 114 may package the visualization program into a project file and provide the project file to the central controller 116. The central controller 116 may then open the visualization program (e.g., traverse the file and generate objects in memory according to the types specified in the file). In some embodiments, the central controller 116 generates an object for each node that knows how to execute based on the parameters of the object. For example, for Figure 2 The central controller 116 can determine the objects of each node 202, 204, 206 and arrange the objects of the nodes 202, 204, 206 so that Figure 2 Execute in sequence as shown.

[0074] When a node in the visual program is associated with a predetermined machine-readable program (e.g., the node is a script node), the central controller 116 can simply retrieve the predetermined machine-readable program. When a node in the visual program is not associated with a predetermined machine-readable program (e.g., the node is a spreadsheet node), the central controller 116 can organize the operation defined by the node into a human-readable representation of the portion of the program for execution.

[0075] For example, a human-readable representation of a visualization program may be created using JavaScript Object Notation (JSON), an open standard format that uses human-readable text. Figures 10A-10C An exemplary object of a spreadsheet node is shown in accordance with some embodiments. Figures 10A-10C A SheetBlock object 1002 is shown. The SheetBlock includes a "SpreadsheetData" entry 1004, which contains the data contained in the spreadsheet. For illustrative purposes, the placeholder " <data>" instead of actual spreadsheet data. In some embodiments, the spreadsheet data can be stored in a base64-encoded binary format. In some embodiments, the spreadsheet data (e.g., contained in a "SpreadsheetData" entry) includes cells with names, cells associated with input flags, and / or cells associated with output flags. In some embodiments, the names of cells in the spreadsheet data are matched with NamedCell objects. The SheetBlock object 1002 can also include information specifying the inputs and outputs of the spreadsheet. For example, the NamedCell entry 1004 includes the following information: it is an image ("DataType": "Image"), it is an input to the spreadsheet ("IsInput": true).

[0076] Fig.11 is an exemplary object list corresponding to cells in a spreadsheet according to some embodiments. The list includes a list of entries for each cell for explaining the cell contents. In this example, the first entry 1104 shows cell A0 of the spreadsheet (pointed to by arrow 1106), containing a function call InputImage (pointed to by arrow 1108). The second entry 1110 shows that cell B5 contains a label (represented by "expression": "threshold") and does not contain the function shown in entry 1104.

[0077] like Figures 10A-10C and Fig.11 As shown in the example in , the central controller has a list of nodes defined by the visual programming environment. The central controller converts each node into an object in memory. The central controller figures out which nodes to run based on left-to-right order (or another order if defined by the user) when it is created, and then executes the object. For example, if one of the nodes is a spreadsheet, the central controller can switch that data to the camera for execution. As another example, if one of the nodes is a script, the central controller can compile and execute the script.

[0078] For example, when the node is a spreadsheet node, the central controller 116 can determine the dependencies of the cell. For example, if the cell is an input, then one or more cells need to process the input before it can act on the input. Another example, if the cell is an output, then the cell that fills the output value needs to be executed before the output is returned.

[0079] In some embodiments, a single spreadsheet node may be used to set up and configure multiple cameras (e.g., one or more or all remotely from a central controller). The cameras may be, for example, smart cameras that include a processor and memory that allows the cameras to execute the spreadsheet and / or the functions defined by the spreadsheet. The user may specify the target camera directly in the spreadsheet so that the user may specify the remoteness of the graphical program. The controller (e.g., Figure 1 The central controller 104 shown, such as an embedded controller, can be configured to coordinate a spreadsheet executing on the entire set of cameras, for example, to transfer data between remote cameras.

[0080] In some embodiments, the programming environment generates a list of operations that need to be performed by the spreadsheet, which can be used to create a dependency graph representing all operations and data. For example, the controller can transmit a block of information representing the spreadsheet to the camera for execution. The camera can open the spreadsheet and determine the dependencies and order of operations that should be performed. The camera can process the dependency graph to determine, for example, where the data is, where it needs to go, and what operation the camera performs.

[0081] In some embodiments, the techniques described herein can use a process of serialization to create a human-readable file. Serialization can include, for example, configuring cells in a spreadsheet, configuring inputs to the spreadsheet, configuring outputs of the spreadsheet, and packaging the spreadsheet information to save it into a readable project file. The host device can send the project file to a controller (e.g., Figure 1 A central controller 104 is shown, such as an embedded controller. The central controller may be configured to coordinate the execution of the spreadsheet to one or more associated cameras, while the functions defined by other blocks of the graphics program may be executed on the controller itself.

[0082] In some embodiments, data (e.g., image data) acquired by the smart camera can be transmitted back to the controller in an abbreviated form. For example, because the image data can be large, the system can be configured so that the controller can access the image on the camera via a URL through the controller's network stack, rather than copying the image from the camera to the controller.

[0083] In some embodiments, when a node of a visual program is a display node, the node may be configured to display data on a display device 108. If there are multiple display devices 108, the display type node may specify a particular display device 108 on which to display the data.

[0084] In some embodiments, the display node can be associated with a specific human-machine interface (HMI). For example, the HMI can include a graphical interface designed to display data on the display device 108. In some embodiments, the HMI can be a network-based HMI. The HMI can be custom designed by the user so that the user can specify the type of data to be displayed on the HMI and the display mode of the data on the display device 108. The technology described here can transfer information from a spreadsheet to the HMI. For example, the host device can provide a graphical user interface builder with a tool tray (pallet), which allows the user to drag and drop a button / text box onto a canvas to graphically design a user interface (HMI). When a specific graphic item is dragged onto the canvas, the properties of the specific graphic item can be configured to point to one or more cells of a spreadsheet. For example, the text of a text box can be configured to display the value filled into a specific spreadsheet cell during the execution of the spreadsheet.

[0085] Although Figure 3 While an example of using a single spreadsheet (associated with introspection tool block 310 ) for a single camera (eg, a camera executing acquisition node 308 ) is shown, the techniques described herein may be used with multiple nodes including multiple cameras and / or supporting spreadsheets. Figure 6 A graphical interface for a visualization program with multiple cameras according to some embodiments is shown (e.g., Figure 3 300 in FIG. 1 . The program panel includes an acquisition node 602 configured to perform image acquisition from two different cameras simultaneously as shown in annotation 604. Images acquired from one camera are processed by an introspection tool block node 606. Images acquired from another camera are processed by an introspection tool block node 608. Thus, as shown in annotation 610, each introspection tool block 606, 608 performs visual processing on data from an associated camera used by the acquisition node 602.

[0086] Figure 7 An exemplary program panel 700 of a graphical interface for visualizing multiple nodes (each using a spreadsheet) for a single task for the same camera is shown in accordance with some embodiments. As noted in note 704, acquisition node 702 is configured to perform image acquisition using a single camera. The output of acquisition node 702 is input to an intraview tool block 706, which is configured to perform visual processing on the data acquired by the camera, as noted in note 708. As noted in note 712, the output of intraview tool block 706 is fed into intraview tool block 1 710, which runs after intraview tool block 706. Each intraview tool block 706 and intraview tool block 1 710 is configured using a different associated spreadsheet.

[0087] Figure 8 An exemplary program panel 800 of a graphical interface according to some embodiments is shown, which is used to visualize a plurality of nodes (each using a spreadsheet) for a plurality of cameras. The acquisition node 802 uses four different cameras to acquire image data. The data acquired by each camera is processed by a different node. The output of one camera is processed by an internal view tool block 806, and the output of the internal view tool block 806 is processed by an internal view tool block 808. The output of two cameras is processed by an internal view tool block 3 810, and the output of the internal view tool block 3 810 is processed by the internal view tool block 2 812. The output of the third camera is processed by the internal view tool block 5 814, and the output of the internal view tool block 5 814 is processed by the internal view tool block 4 816. The output of the fourth camera is processed by the internal view tool block 7 818, and the output of the internal view tool block 7 818 is processed by the internal view tool block 6 820. Each internal view tool block is configured using a different associated spreadsheet. The internal view tool blocks are contained in a parallel block 822. Thus, each of the introspection tool blocks 806, 810, 814, and 818 simultaneously performs visual calculations on data output from the corresponding camera. Similarly, the introspection tool blocks 808, 812, 816, and 820 are run after the first set of introspection tool blocks.

[0088] It should be understood that the disclosed subject matter is not limited in its application to the details of the structure and the arrangement of the parts set forth in the following description or shown in the accompanying drawings. The disclosed subject matter is capable of other embodiments and of being practiced and executed in various ways. In addition, it should be understood that the words and terms used herein are for descriptive purposes and should not be considered limiting.

[0089] Therefore, those skilled in the art will understand that the concepts upon which the present disclosure is based can be easily used as a basis for the design of other structures, methods and devices that achieve the several purposes of the disclosed subject matter. Therefore, it is important that the claims be considered to include these equivalent structures as long as they do not depart from the spirit and scope of the disclosed subject matter. For example, some of the disclosed embodiments involve one or more variables. This relationship can be expressed by a mathematical equation. However, a person of ordinary skill in the art can also use a different mathematical equation to express the same relationship between one or more variables by transforming the disclosed mathematical equation. It is important to regard the claims as including such an equivalent relationship between one or more variables.

[0090] While the disclosed subject matter has been described and illustrated in the foregoing exemplary embodiments, it should be understood that the present disclosure is made by way of example only and that numerous changes in implementation details of the disclosed subject matter may be made without departing from the spirit and scope of the disclosed subject matter.< / data>

Claims

1. A computer-implemented method for editing a graphics program using a graphics programming development environment, the method include: In the graphical programming development environment, select the Spreadsheet node where: The spreadsheet node graphically represents a portion of a graphical program; and The electronic table node is associated with an electronic table, the electronic table comprising a plurality of cells, the plurality of cells being graphically arranged into a plurality of rows, a plurality of columns, or both a plurality of rows and a plurality of columns; receiving data representing a function to be inserted into a cell of a plurality of cells of the electronic form, the function being configured to perform at least a portion of a machine vision task; and A computer executable version of the graphical program configured to perform the machine vision task is generated, including the spreadsheet-specified functions in the spreadsheet nodes. 2 . The method of claim 1 , further comprising sending the computer executable version of the graphics program to a controller, the controller configured to execute the graphics program. 3 . The method of claim 1 , further comprising receiving execution attributes of the spreadsheet node, which represent computer-readable instructions that specify a computing device to execute the spreadsheet node.

4. The method according to claim 3, further comprising: include: determining, at the controller, a computer-readable instruction specifying execution of the spreadsheet node by a computing device based on the execution attribute of the spreadsheet node; and The computer readable instructions of the spreadsheet node are transmitted to a computing device for execution of the computer readable instructions of the spreadsheet node.

5. The method according to claim 1, further comprising: include: upon receiving a request to view a spreadsheet associated with the spreadsheet node, displaying the spreadsheet in the graphical programming development environment; receiving second data to be associated with a cell in the electronic form; as well as The cells in the spreadsheet are populated with the second data. The method of claim 1 , wherein the computing device is a camera.

7. The method according to claim 1, in, The computer readable instructions of the spreadsheet node are designed to perform machine vision analysis of the image.

8. The method of claim 1, further comprising providing a second spreadsheet node in the graphical programming development environment, wherein the second spreadsheet node is associated with a second spreadsheet designed to be executed at another computing device.

9. The method of claim 1, wherein the spreadsheet node includes an output terminal, wherein the output terminal is associated with another cell of the plurality of cells of the spreadsheet, which represents an output of the computer-readable instructions of the spreadsheet node.

10. The method of claim 9, further comprising connecting an output terminal of a spreadsheet node to a terminal of a node of a graphical program to provide the output of the computer-readable instructions of the spreadsheet node to the terminal of the node of the graphical program.

11. The method of claim 1, further comprising providing a visualization interface having a graphical element, and associating the graphical element with the cell of a plurality of cells of the electronic table to display the content of the cell at the graphical element.

12. The method of claim 11, wherein the contents of the cell include an image, and wherein associating the graphical element with the cell includes assigning a pointer to the graphical element that references the image.

13. The method of claim 11, further comprising displaying a portion of the electronic form in a visualization interface graphical element.

14. The method of claim 13, further comprising using a visualization interface to receive a request to modify the cell of a plurality of cells of a spreadsheet.

15. The method according to claim 1, further comprising: include: Executing computer readable instructions of a spreadsheet node at a computing device, wherein executing the computer readable instructions comprises: receiving a pointer to second data to be processed by the computing device; determining whether the computing device maintains second data locally at the computing device; Upon determining that the second data is maintained locally at the computing device, processing the second data; and Upon determining that the second data is not maintained locally at the computing device, the second data is retrieved from the storage medium referenced by the pointer and processed.

16. The method of claim 1, wherein the computing device comprises a plurality of computing modules, and wherein the method further comprises automatically selecting one or more computing modules to execute the computer-readable instructions of the spreadsheet node.

17. The method of claim 16, wherein automatically selecting one or more computing modules comprises determining a computing load of the one or more computing modules.

18. The method of claim 1, wherein the spreadsheet node comprises an input terminal, wherein the input terminal is associated with a cell in the spreadsheet, which means that an input received by the input terminal is processed by a function in a function list.

19. A system for editing graphics programs using a graphics programming development environment, include: a processor in communication with the memory, wherein the processor is configured to execute a computer program stored in the memory, the computer program being configured to: In the graphical programming development environment, select the Spreadsheet node where: The spreadsheet node graphically represents a portion of a graphical program; and The electronic table node is associated with an electronic table, the electronic table comprising a plurality of cells, the plurality of cells being graphically arranged into a plurality of rows, a plurality of columns, or both a plurality of rows and a plurality of columns; receiving data representing a function to be inserted into a cell of a plurality of cells of the electronic form, the function being configured to perform at least a portion of a machine vision task; and A computer executable version of the graphical program configured to perform the machine vision task is generated, including the spreadsheet-specified functions in the spreadsheet nodes.

20. A non-transitory computer readable medium having executable instructions associated with a system for editing a graphics program using a graphics programming development environment, the instructions being operable to cause the system to: In the graphical programming development environment, select the spreadsheet node. in: The electronic form node graphically represents a portion of the graphical program; as well as The electronic table node is associated with an electronic table, the electronic table comprising a plurality of cells, the plurality of cells being graphically arranged into a plurality of rows, a plurality of columns, or both a plurality of rows and a plurality of columns; receiving data representing a function to be inserted into a cell of a plurality of cells of the electronic form, the function being configured to perform at least a portion of a machine vision task; as well as A computer executable version of the graphical program configured to perform the machine vision task is generated, including the spreadsheet-specified functions in the spreadsheet nodes.

Images