Programming device and program

Abstract

To provide a programming device and a program that allow a user to easily confirm creation of a program such as an image processing algorithm or the appropriateness of a program being edited.SOLUTION: The programming device is a programming device which edits a program by arranging icons of units showing processing targets on an edition screen according to a user operation and connecting the units. The programming device includes: program execution means for executing a program defined by the units arranged on the edition screen and connected to one another and operating outputs of the units; and debug view display control means for displaying a debug view region in which the output of each unit obtained by operations by the program execution means is related to the icons of the units arranged on the edition screen.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a programming device and a program suitable for creating and editing a program used for image processing and the like. [Background technology]

[0002] Nowadays, imaging devices such as cameras that capture images and computers that process images are easily available, and systems incorporating image processing and image analysis are being used in various fields. Image inspection systems using cameras and computers are becoming more and more common in factory production sites for dimensional inspection, appearance inspection, and foreign body inspection of products. As a solution to realize such systems, image inspection systems that have been made into general-purpose devices and software programs sold as packages are provided, which are intended for use in various fields. However, in many cases, it is difficult to meet the detailed specifications required at the field by using general-purpose systems and software programs as they are, so they are customized to meet special specifications or programs that meet the desired specifications are developed independently. Detailed specifications are a combination of a wide range of specifications, such as selecting and changing image processing algorithms to suit the object of image inspection, and adjusting thresholds (parameters) that are essential for image processing.

[0003] Because on-site requirements are so diverse, there is a demand for a programming environment that enables system developers and on-site equipment users to easily set and change image processing algorithms and thresholds, allowing them to develop programs and adjust parameters.

[0004] Graphical programming languages ​​such as function block diagram (FBD) are available as a method that allows users who do not have a high level of programming expertise to perform programming.

[0005] With FBD, it is possible to perform programming graphically using various components, such as function blocks consisting of functions corresponding to various calculation processes, connectors for exchanging input / output parameters with external devices, and connection lines that connect these function blocks and connectors.

[0006] In a programming device for programming FBD, symbols representing such components are placed on an editing screen as objects, the processing procedures are defined by connecting the ports on these objects with connection lines, and the processing of each object is adjusted by changing the properties that indicate the variables that can be set for the object (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2008-204237 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, while the conventional techniques have made programming easier, when checking the validity of a program, it has not been possible to easily check intermediate processing results in the multiple stages that make up the program, and debugging and fine adjustment of variables has required a lot of work. In particular, in a program that performs various image processing steps in multiple stages, a change in a variable in one process (object) can have a large effect on the output in the later stage (including the final output, of course), so variable adjustments can require repeated trial and error, and there has been a problem in that the repeated work of editing the program and checking the validity of the program is a heavy burden.

[0009] The present invention is devised to solve such problems, and aims to provide a programming device and program that can easily check the validity of a program being created or edited, such as an image processing algorithm. [Means for solving the problem]

[0010] In order to solve the above problems, the programming device according to the present invention is a programming device that edits a program by arranging unit icons representing processing units on an editing screen and linking these units in response to a user operation, and is characterized in that it comprises a program execution means that executes a program defined by the units arranged on the editing screen and linked together, and calculates the output of each unit, and a debug view display control means that displays the output of each unit calculated by the program execution means in a debug view area corresponding to the icon of each unit arranged on the editing screen. The units that make up the program may be selectable from a plurality of options including input to the programming device, output from the programming device, conditional branching, and inference based on learned data of deep learning.

[0011] The programming device according to the present invention may further include a unit editing means for accepting changes to the setting values ​​of each unit constituting the program.

[0012] In the present invention, the debug view display control means may update the display of the debug view area in real time in response to changes in the output of each unit while the program execution means is executing the program. Also, the debug view display control means may display a heat map in the debug view area showing the locations that are captured as features in the inference for a unit that performs inference using learned data of deep learning.

[0013] In the present invention, the unit editing means may be capable of accepting changes to the setting values ​​of the units while the program execution means is executing the program. Also, the program execution means may reflect the changes to the setting values ​​accepted by the unit editing means during the execution of the program in the calculations of the program being executed as needed.

[0014] The programming device according to the present invention may be configured to use a flexible image filter unit capable of selecting a plurality of types of image filters as a unit. The unit editing means may accept settings of the flexible image filter by a user through a setting screen including a first preview area for displaying preview images when each of a plurality of selectable types of image filters is applied to input data in an identifiable manner, a second preview area for displaying preview images when a plurality of settable setting values ​​are applied to characteristic parameters related to the selected image filter in an identifiable manner, and a target image display area for displaying a target image, which is a preview image when the selected image filter is applied to input data with the set value of the characteristic parameter, larger than each of the preview images displayed in the first preview area and the second preview area.

[0015] A program according to the present invention is characterized in that it causes a computer to function as any one of the above programming devices. [Brief description of the drawings]

[0016] [Figure 1] FIG. 2 is a block diagram showing the configuration of a programming device. [Diagram 2] FIG. 4 is a diagram illustrating an example of a data structure of a unit. [Diagram 3] FIG. 13 illustrates an example of an editing screen. [Figure 4] FIG. 13 is a diagram showing an example of an icon of a unit having a plurality of output data. [Diagram 5] 1 shows an example of a setting screen of a flexible image filter unit. [Figure 6] FIG. 13 is a diagram showing an example of an edit screen immediately after startup. [Figure 7] FIG. 13 is a diagram showing an example of a unit selection screen. [Figure 8] FIG. 13 is a diagram showing an example of a unit setting screen. [Figure 9] FIG. 13 is a diagram showing an example of an edit screen in which one unit has been added. [Figure 10] FIG. 13 is a diagram showing an example of an edit screen in which three units have been added. [Figure 11] FIG. 13 is a diagram showing an example of an editing screen in a state in which three units are linked. [Figure 12] FIG. 13 is a diagram showing an example of an edit screen in an execution mode. [Figure 13] FIG. 13 is a diagram showing an example of an edit screen in an edit mode. [Figure 14] FIG. 11 is a diagram showing another example of the unit board. [Figure 15] FIG. 13 is a diagram showing yet another example of the unit board. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a programming device according to this embodiment.

[0018] The programming device 10 as a whole comprises an information processing device such as a server device, a personal computer, a mobile information terminal, etc., and has the function of creating and editing programming by arranging various units, which are processing steps that make up a program, on an editing screen in response to user operations and connecting these units with connecting lines.

[0019] Units that can be arranged in the programming device 10 include not only image processing but also various other elements necessary for image inspection (for example, input, output, conditional branching, inference based on learned data of deep learning, etc.), and in programming, all processing can be handled uniformly as units. Input units include cameras, external programs, shared memory, network input by TCP / IP, image files, CSV files, I / O boards, etc. As camera units, not only standard USB cameras (UVC drivers) but also units compatible with major industrial cameras from each company are available. Output units include display, image files, CSV files, shared memory, network communication by TCP / IP, I / O boards, patrol lights, etc. As image processing units, various types of image processing such as filters, calculations for multiple inputs (addition and subtraction of images, etc.), separation of images by RGB, output of HSV (H: hue, S: saturation, V: brightness) information from images, etc. As other units, processing involving judgment such as conditional branching and inference based on learned data of deep learning is also included. For these units, unit template information 14A in which processing contents and the like are defined, as described below, is prepared in advance. A user of the programming device 10 freely arranges and connects these units to create a desired program.

[0020] As shown in FIG. 1, a programming device 10 is provided with a communication I / F section 11, an operation input section 12, a screen display section 13, a storage section 14, and an arithmetic processing section 15 as main functional sections.

[0021] The communication I / F unit 11 is made up of a communication interface circuit, and has a function of exchanging various data by performing data communication with an external device (not shown) via a communication line. Examples of the external device include devices that provide input information to a programming device, such as cameras and various sensors, output devices controlled by a program, such as a patrol lamp and a buzzer, a server device, a personal computer, a mobile information terminal, and an information processing device that receives the execution result of a program and performs further processing.

[0022] The operation input unit 12 is made up of an operation input device such as a keyboard, a mouse, a touch panel, etc., and has a function of detecting an operation indicating a user operation and outputting the operation to the calculation processing unit 15 .

[0023] The screen display unit 13 is made up of a screen display device such as an LCD, and has a function of displaying various data such as an editing screen and a setting screen outputted from the arithmetic processing unit 15 on the screen.

[0024] The storage unit 14 is made up of a storage device such as a hard disk or a semiconductor memory, and has a function of storing various processing information used in the processing in the arithmetic processing unit 15 and a program 14P.

[0025] The program 14P is a program that, when executed by the CPU of the arithmetic processing unit 15, realizes various processing units, and is read out from an external device or a recording medium (neither of which are shown) and stored in the storage unit 14 in advance.

[0026] The main processing information stored in the storage unit 14 includes unit template information 14A and unit board information 14B. The unit template information 14A is information that defines the data structure, variables, algorithms, etc. of each unit, which is a processing step that constitutes a program.

[0027] FIG. 2 shows an example of the data structure of a unit. The unit ID is unique identification information for identifying a unit, and is automatically assigned when the unit is placed during the programming process. It is preferable that the user is prohibited from editing the unit ID. The unit name is the name of the unit displayed on the editing screen, and can be freely given by the user. The unit type is information indicating the type of unit. This unit type is predefined, and editing by the user is prohibited. Examples of unit types include input (camera, input image file, etc.), output (patrol lamp, display, output file, etc.), image processing (filter, image size conversion, etc.), etc. The programming device 10 of this embodiment includes a flexible image filter unit, which will be described later, as one of the units that performs filter processing on an image.

[0028] The unit coordinates are information that indicates the display position of the unit within the editing screen. The initial values ​​of the unit coordinates are determined when the unit is placed during the programming process, and are then updated appropriately as the user moves the unit within the editing screen.

[0029] The previous unit pointer and next unit pointer are information that specifies the connection relationship of units. The previous unit pointer stores a pointer (memory address) to the data of the unit linked to the previous stage of the unit in question. The next unit pointer stores a pointer (memory address) to the data of the unit linked to the subsequent stage of the unit in question. The previous unit pointer and next unit pointer are set by an operation that links multiple units placed during the programming process.

[0030] The unit setting screen is information that specifies the unit-specific setting screen that is displayed when placing the unit during the programming process. The unit setting screen is predefined, and editing by the user is prohibited. For example, in the case of a camera input unit, a screen for setting the camera number (indicating the camera model, etc.) is defined. Note that there is no need to define here the screen for setting the unit display name, etc., which are common setting items regardless of unit type.

[0031] The unit setting data is information on setting data specific to the unit. For example, in the case of a camera input unit, the camera number indicating the selected camera corresponds to the unit setting data. Also, in the case of a resize unit that converts the size of input image data, information that specifies the size of the resized image (number of pixels, magnification rate for the input image, etc.) corresponds to the unit setting data.

[0032] The unit input data indicates data such as images and files input to the unit from a unit in the previous stage. For a unit that is not connected to another unit in the previous stage, such as a camera input unit, input data does not need to be defined.

[0033] The unit logic specifies the algorithm of the processing executed by the unit. The logic of the unit is predefined and is prohibited from being edited by the user. The user is allowed to adjust the processing in the unit to the extent possible by changing the setting data. As the logic of the unit, for example, for a unit that receives input from an external device such as a camera input unit, an algorithm is specified for controlling the external device to obtain necessary data and outputting the data to a subsequent stage. Also, for a unit that controls an external device based on input data such as a patrol lamp unit, an algorithm is specified for controlling the external device to perform a desired operation. Also, for a unit that processes input data and outputs output data such as a filter unit, an algorithm is specified for outputting output data based on the input data.

[0034] The unit output data represents data such as images and files that are output from the unit to the unit in the next stage. For units that are not connected to other units in the subsequent stage, such as a police light or an output file unit, output data does not need to be defined.

[0035] The storage unit 14 stores, as unit template information 14A, the above-mentioned data structures for various units, as well as predefined information such as unit types, setting screens, logic, and the like.

[0036] The unit board information 14B is information that represents a program created by the user by combining various units. Various information about multiple units that are arranged and connected using an editing screen (described later) is stored as the unit board information 14B in a format that conforms to the data structure of the unit described above.

[0037] The arithmetic processing unit 15 has a CPU and its peripheral circuits, and realizes various processes of the programming device by reading and executing the program 14P from the storage unit 14. That is, the arithmetic processing unit 15 functions as a program execution means, a debug view display control means, and a unit editing means in the present invention. One of the main processes realized by the arithmetic processing unit 15 is editing processing of the unit board.

[0038] The editing process includes functions such as arranging various units on the editing screen in response to user operation, connecting the arranged units in response to user operation, and changing the setting values ​​of the arranged units in response to user operation.

[0039] Fig. 3 shows an example of an editing screen used when performing editing processing. As shown in Fig. 3, the editing screen has a control area R1 in which control buttons used for editing and controlling units are arranged, and a unit board editing area R2 in which the user can arrange and connect units and perform graphical programming.

[0040] The control area R1 is provided at the top of the editing screen, and has a number of control objects such as control buttons and check boxes for placing, moving, executing, etc. units. These control objects are broadly divided into four types: those related to the control of the unit board information 14B, those related to programming in the unit board editing area R2, those related to the execution control of the programs programmed in the unit board editing area R2, and those related to the display control of the unit board editing area R2.

[0041] Control objects related to control of the unit board information 14B include a display T1 of the name of the unit board being edited, a save button B1 for saving the unit board (program) programmed in the unit board editing area R2, a load button B2 for loading a saved unit board and displaying it in the unit board editing area R2, an end button B3 for ending editing of the unit board being edited in the unit board editing area R2, and a checkbox CB1 for selecting whether or not to enable the debug view described below.

[0042] Control objects relating to programming in the unit board editing area R2 include an add unit button B4, an edit unit button B5, a link unit button B6, a move unit button B7, a delete unit button B8, and a duplicate unit button B9.

[0043] The Add Unit button B4 is a control button for starting an add mode in which a new unit is placed in the unit board editing area R2. When this Add Unit button B4 is pressed, a unit selection screen is displayed for selecting the unit to be placed. After selecting the desired unit on the unit selection screen and configuring the unit according to the screen display, clicking the desired position in the unit board editing area R2 will place the unit icon at the clicked position. As shown in FIG. 3, the unit icon is a roughly rectangular object, and the unit type and unit name are displayed.

[0044] The unit edit button B5 is a control button for starting an edit mode in which the settings of a unit placed in the unit board edit area R2 are edited. When the unit edit button B5 is pressed and the icon of a unit placed in the unit board edit area R2 is clicked, a setting screen for editing the setting values ​​of the unit is displayed. The user can edit various setting values ​​of the unit on the setting screen.

[0045] The unit connection button B6 is a control button for starting a connection mode in which the connection relationships between units arranged in the unit board editing area R2 are set up. Here, unit connection means defining the transfer of data so that the output of the unit at the connection source becomes the input of the unit at the connection destination.

[0046] FIG. 4 shows, as an example of a unit, a display example of a unit that outputs HSV (H: hue, S: saturation, V: brightness) information from an image. As exemplified in FIG. 4, a joint figure J (e.g., a small rectangle) corresponding to the number of inputs and outputs is displayed for each unit. Furthermore, characters or the like indicating the contents of input and output are displayed in the joint figure J. In a unit icon, the left side represents the input end and the right side represents the output end, so that a joint figure J corresponding to the number of inputs is displayed on the left side of the unit, and a joint figure J corresponding to the number of outputs is displayed on the right side of the unit. To visualize the connection relationship, a connection line (a line connecting the joint figures J) indicating that the units are connected is displayed between the connected units in the unit board editing area R2.

[0047] When the unit connection button B6 is pressed, the connection mode is started and the state becomes one in which the selection of the unit to be connected is accepted. When the icon of the desired unit (for a unit with multiple outputs, the joint graphic J of one of the outputs) is clicked, the output of that unit (if there are multiple outputs, the one corresponding to the clicked joint graphic J) is specified as the connection source. Next, when the icon of the connection destination unit (for a unit with multiple inputs, the joint graphic J of one of the inputs) is clicked, the input of that unit (if there are multiple outputs, the one corresponding to the clicked joint graphic J) is specified as the connection destination, and the two units are connected. By such an operation, a pair of units can be connected. In the unit board editing area R2, a line connecting the joint graphics J of the connected units is displayed to indicate that they are connected. In the unit icon, the left side means the input end and the right side means the output end. Therefore, when the output of the first unit is input to the second unit, a line connecting the right side of the icon of the first unit and the left side of the icon of the second unit is displayed.

[0048] The unit movement button B7 is a control button for starting a movement mode in which connections between units placed in the unit board editing area R2 are set. In the movement mode in which the unit movement button B7 is pressed, the user can freely move units placed in the unit board editing area R2 by dragging and dropping them.

[0049] The unit delete button B8 is a control button for starting a delete mode in which a unit placed in the unit board editing area R2 is deleted. In the delete mode in which the unit delete button B8 is pressed, clicking on a unit placed in the unit board editing area R2 allows the unit to be deleted. Note that a screen requesting confirmation of intent before executing the deletion may be displayed, and the deletion may be executed only if the intent to delete has been confirmed.

[0050] The unit duplication button B9 is a control button for starting a duplication mode in which a unit placed in the unit board editing area R2 is duplicated. In duplication mode with the unit duplication button B9 pressed, clicking a unit placed in the unit board editing area R2 selects that unit as the unit to be duplicated, and further clicking within the unit board editing area R2 adds a duplicate of the selected unit (with a different unit ID and not linked) at the clicked position.

[0051] Control objects relating to control of execution of a program programmed in the unit board editing area R2 include a unit execution button B10, a step execution button B11, and a unit stop button B12.

[0052] The unit execution button B10 is a control button for starting an execution mode in which a program defined in the unit board editing area R2 is executed. In the execution mode in which the unit execution button B10 is pressed, a series of programs defined by a group of units arranged and connected in the unit board editing area R2 are executed from start to finish by the arithmetic processing unit 15.

[0053] The step execution button group B11 is a control button group that controls a step execution mode in which the program defined in the unit board editing area R2 is executed in stages in sequence. The step execution button group B11 is mainly used for debugging such as checking the operation of the program, and includes buttons for setting breakpoints, stepping in, stepping over, stepping out, etc.

[0054] The unit stop button B12 is a control button for terminating the unit execution mode and the step execution mode.

[0055] Control objects related to display control of the unit board editing area R2 include a text box T2 and slider bar SB for setting the display magnification in the unit board editing area R2, and a check box CB2 for setting whether or not to display units in the unit board editing area R2 transparently.

[0056] The unit board editing area R2 is an area where unit icons and connections between units are displayed. The user can freely move units placed in the unit board editing area R2 by dragging and dropping them, or double-click a unit to display a setting screen where the user can adjust the setting values.

[0057] In addition, in the unit board editing area R2, a debug view DV showing the processing results (output data) of each placed unit can be displayed. The debug view DV may be displayed, for example, as a speech bubble from the corresponding unit. Whether or not to display the debug view DV is determined by the input state (ON / OFF) of a check box CB1 provided in the unit board editing area R2. If the check box CB1 is ON, the debug view DV is displayed not only in the unit execution mode and step execution mode, but also when editing a unit in the unit board editing area R2 in other modes such as the add mode and edit mode. Note that, for units whose output is determined even in the execution mode, such as a unit that outputs a specified still image, the output is displayed in the debug view DV simply by placing the unit. On the other hand, for units such as cameras and image processing units that do not output output data unless processing is performed, only the frame of the debug view DV is displayed when the unit is placed, but the output data (image) is not displayed and is blank, and the output data is displayed only when the unit is executed. During execution, the display of the debug view DV is updated in real time according to changes in the output of the unit.

[0058] The output data of the unit displayed in the debug view DV may vary depending on the type of unit. In the case of a unit that outputs an image, such as a camera or image processing, it is preferable to display the image as output data in the debug view DV. In addition, in the case of a unit of inference using trained data of deep learning, it is possible to classify the data by inference based on the input to the unit, and display the rank (a number that ranks the classes in order of the likelihood of being applicable) and the estimated probability of each class (possibility of being applicable) in the debug view DV. Alternatively, in a unit of inference using trained data of deep learning, it is possible to display a heat map showing the parts that the neural network has captured as features in the debug view DV. It is preferable that this heat map is displayed semi-transparently, superimposed on the input image.

[0059] By displaying this kind of debug view DV for each unit placed in the unit board editing area R2, it is possible to check not only the final output but also the intermediate output data of each unit at a glance. In a program that performs various image processing in multiple stages, the transition of the processing results through each stage (unit) can be confirmed in a list form up to the final output, so it is possible to easily perform trial and error of settings to obtain the desired output result while looking at the debug view DV.

[0060] The unit edit button B5 is also enabled during execution mode, and clicking the unit edit button B5 during execution mode and then clicking the icon of the unit to be edited displays the setting screen for that unit. When a setting value is changed on the setting screen, the change in setting value is reflected in the running program in real time. As a result, the effect of the change in setting value is immediately reflected in the display of the debug view DV. This configuration allows the user to adjust the setting value while checking the effect of the change in setting value.

[0061] Next, a flexible image filter unit that can be used in the programming device 10 of this embodiment as one of the units arranged in the unit board editing area R2 using the above-mentioned operation screen will be described. Various types of algorithms such as Hough transform, binarization, inversion, edge extraction, color extraction, and saturation extraction have been used conventionally as filters for images. The flexible image filter unit is a unit in which characteristic parameters that determine the type of filter (algorithm) and the characteristics of the filter are set as setting values. Since the flexible image filter unit is a unit that performs filtering on an image, both the input data and the output data are image data.

[0062] The flexible image filter unit can be used by adding and connecting it to the unit board editing area R2 like other units, but its setting screen has a feature. FIG. 5 shows an example of a setting screen that accepts the setting of the flexible image filter unit. As shown in FIG. 5, the setting screen of the flexible image filter unit includes a target image display area R3, an algorithm preview area R4 (corresponding to the first preview area of ​​the present invention), and a parameter preview area R5 (corresponding to the second preview area of ​​the present invention). The target image display area R3 occupies a main area in the screen including the center of the screen. In the target image display area R3, a target image is displayed, which is a preview image when the filter type and characteristic parameters set at that time are applied to the input data. The target image displayed in the target image display area R3 is preferably displayed larger than the preview images displayed in the algorithm preview area R4 and the parameter preview area R5. The preview images displayed in each area (R3 to R5) may be configured so that the input is previously applied with a filter of each condition and can be viewed and selected, or may be configured so that they are processed and updated sequentially each time they are selected.

[0063] The algorithm preview area R4 is an area provided to the left of the target image display area R3. In the algorithm preview area R4, preview images when selectable algorithm filters (the characteristic parameters may be set to predetermined initial values ​​for each filter) are applied to the input data are displayed vertically. Among the multiple preview images, the one corresponding to the filter type selected at that time is made identifiable by being surrounded by a thick frame, for example. In the example of FIG. 5, a binarization filter is selected.

[0064] In the setting screen of the flexible image filter unit, the operation of selecting the filter type may be an operation corresponding to the display of the algorithm preview area R4. That is, the filter type may be selected by pressing the "up" or "down" key on the keyboard, rotating the scroll wheel with the mouse, clicking the preview image corresponding to the desired filter in the algorithm preview area R4 with the mouse, etc. In addition to these intuitive operations, the selection of the filter type may be accepted by a list box or the like.

[0065] The parameter preview area R5 is an area provided above the target image display area R3. In the parameter preview area R5, multiple preview images with different characteristic parameters in the filter of the algorithm selected at that time are displayed side by side. Among the multiple preview images, the one corresponding to the characteristic parameter set at that time is made identifiable by surrounding it with a thick frame or the like. The preview images are generated with the characteristic parameter values ​​divided into predetermined steps from the minimum value to the maximum value that can be set, and may be displayed in ascending or descending order of the characteristic parameter values. In the example of FIG. 5, preview images with different binarization thresholds in the selected binarization filter are displayed.

[0066] In the setting screen of the flexible image filter unit, the operation of setting the characteristic parameters may be an operation corresponding to the display of the parameter preview area R5. That is, the characteristic parameters may be set by pressing the "right" or "left" key on the keyboard, clicking a preview image corresponding to the desired characteristic parameter in the parameter preview area R5 with the mouse, switching the mouse wheel left and right, clicking the left and right buttons of the mouse while pressing a specific key (e.g., the Shift key or the Ctrl key), etc. In addition to these intuitive operations, an input area for inputting the value of the characteristic parameter may be provided to accept direct input.

[0067] The specific image processing algorithm for the flexible image filter unit has not been decided at the stage of adding the unit. By trial-running the unit board (program) including the flexible image filter unit while selecting the algorithm and setting the threshold value for each algorithm from a two-dimensional overview, it is possible to save time from trial and error and quickly set the appropriate filter conditions.

[0068] In the flexible image filter unit described above, it is possible to switch between algorithms while checking the processing results of each algorithm in the vertical list displayed in the algorithm preview area R4. In addition, it is possible to change parameters while checking the effects of parameter changes at a high level in the horizontal list displayed in the parameter preview area R5, which shows the processing results for each threshold of the selected algorithm. As a result, it is possible to efficiently and intuitively perform trial and error operations on the setting screen while checking the effects of setting changes, whereas in the conventional method, it was time-consuming to set and check each parameter one by one.

[0069] [Operation of this embodiment] Next, a typical flow of programming using the programming device 10 configured as above will be described.

[0070] When the program 14P is started in the programming device 10, an edit screen is displayed as shown in Fig. 6. On this edit screen immediately after start-up, the unit board edit area R2 is blank.

[0071] Next, when the unit addition button B4 is clicked to switch to the addition mode, a unit selection screen is displayed as shown in FIG. 7. In this unit selection screen, one of various units registered in the unit template information 14A is selected. Each unit is classified by unit type (unit type in the data structure). In the unit selection screen, the desired unit type is selected from the list menu L1, and the unit to be added is specified by selecting one of the units belonging to the unit type displayed in a list in the unit display area R6. Here, it is assumed that a USB camera unit belonging to the input type is selected. In addition, in the unit selection screen, the unit name is set by appropriately editing it in the text box T3. Here, it is assumed that the unit name "USB Camera (UVC Universal Video Class)_4" is set.

[0072] After selecting the unit and setting the unit name, click the OK button, and the unit selection screen disappears, and the unit setting screen appears. This unit setting screen is a unit-specific setting screen, as explained in the data structure of the unit. An example of the unit setting screen for the USB camera to be added here is shown in FIG. 8(a). In this setting screen, the camera number is set. FIG. 8(b) is another example of the unit setting screen, and shows the unit setting screen for the resize unit that changes the image size. After making the desired settings on the specific unit setting screen for the unit to be added, click the OK button, and the unit setting screen disappears, and the input waiting state is entered. Then, click the position in the unit board editing area R2 where you want to place the unit icon, and the icon U1 of the unit selected on the unit selection screen is added to the clicked position, as shown in FIG. 9, and the adding mode ends. A debug view DV1 is displayed next to the icon U1 according to the setting of the check box CB2. Repeat the same operation to add the required number and types of units to the unit board editing area R2.

[0073] If you press the Edit Unit button B5 to enter edit mode, you can open the unit setting screen by clicking the unit icon of a unit you have added and change various parameters of the unit, etc. You can also delete units using the Delete Unit button B8, and duplicate units using the Duplicate Unit button B9.

[0074] The unit board (program defined by connecting units) edited in the unit board editing area R2 can be saved by specifying the unit board name and save location on the screen displayed by clicking the Save button B1. Also, on the screen displayed by clicking the Load button B2, a previously created unit board can be selected, and the selected unit board can be loaded and displayed in the unit board editing area R2.

[0075] FIG. 10 shows an example of a screen in which three units (U1 to U3) have been added: a USB camera, a binarization unit, and a BLOB detection unit. In the state shown in FIG. 10, the units are not linked and no input / output data is being exchanged. Click the unit link button B6 to switch to link mode and set up a connection between the units. When link mode starts, the system first accepts the selection of the unit to be linked. Clicking on the joint figure at the output of the desired unit identifies that unit as the link source. Next, clicking on the joint figure at the input of the icon of the unit to be linked connects the two units and ends the link mode. Through the above series of operations, a pair of units can be linked.

[0076] FIG. 11 shows three units (U1 to U3) added to the unit board editing area R2, connected in this order: USB camera, binarization, and BLOB detection. When the unit execution button B10 is clicked in this state, the mode changes to execution mode and the program is executed. That is, imaging with the USB camera is started, and the captured image is converted to grayscale and resized. When the debug view DV is enabled in execution mode, the output of each unit is displayed in real time in the debug views (DV1 to DV3) corresponding to each unit (U1 to U3), as shown in FIG. 12. The execution mode continues until the unit stop button B12 is clicked.

[0077] In the programming device 10, the setting value of each unit is allowed to be adjusted while the execution mode is ongoing. That is, the unit edit button B5 is enabled even during the execution mode, and when the unit whose setting value is to be adjusted is clicked after clicking the unit edit button B5, the setting screen of the unit is displayed. When the setting value is changed on the displayed setting screen, the change is reflected in the debug view in real time. For example, in the example shown in FIG. 13, the setting value (threshold) is adjusted on the setting screen of the binarization unit while checking the images of each stage in real time on the debug view (DV1 to DV3), so that the detected object in the image appears as a white block in the image. With this configuration, the user can efficiently adjust each layer (unit) in the image processing program, which is a compilation of various algorithms, while checking the effect of the adjustment in real time.

[0078] Fig. 14 shows another example of a unit board programmed using the programming device 10. Specifically, Fig. 14 shows a unit board that realizes more complicated processing by using a unit having multiple inputs and a unit having multiple outputs.

[0079] That is, the unit board shown in Fig. 14 has a USB camera unit U4 and a still image unit U5 as input units, and realizes processing to extract an image of a target subject excluding the background by subtracting a still image from an image captured by the USB camera using a subtraction unit U6. The subtraction unit has two inputs (input A and input B), and is a unit that outputs an image obtained by subtracting an image input to input B from an image input to input A.

[0080] The method for adding and connecting units to the unit board editing area R2 is the same as the method described with reference to Fig. 10. When connecting a unit, which input of the subtraction unit to connect to is set by clicking either the joint figure JA or JB.

[0081] As in the example shown in Fig. 14, when there are multiple units that are not connected to other units in the preceding stage, the program is realized assuming that processing starts from these multiple units. The unit that is the starting point of such processing may be configured so that a unit that is not connected to other units in the preceding stage is automatically set, or the user may explicitly set the unit that is the starting point. When the user explicitly sets the unit that is the starting point, an interface for setting the unit that is the starting point may be provided on the editing screen, and the data structure of the unit may include information indicating that it is a starting point in addition to the items shown in Fig. 2.

[0082] Fig. 15 shows yet another example of a unit board programmed using the programming device 10. Fig. 15(a) and (b) each show a display example of a debug view DV in a unit of inference using learned data of deep learning.

[0083] In the example shown in FIG. 15(a), unit U8, which is an inference unit based on learned data of deep learning, classifies the inspection image (in this example, a still image of a dumpling with a mixture of normal and abnormal products) provided by unit U7 as an input image for appearance inspection. In the debug view DV8 of unit U8, the rank (a numerical value that ranks the classes in order of the likelihood of being applicable) and the estimated probability (possibility of being applicable) of each class are listed by classifying the images by inference based on the display of the rank and estimated probability. By displaying such a debug view, the user can adjust each unit that constitutes the algorithm while checking the possibility of the deep learning inference result, including not only the most likely class but also the class with a low possibility of being applicable (low ranked). Furthermore, the inference result can be linked to a subsequent unit, making it possible to use further image processing and judgment results.

[0084] In the example shown in FIG. 15(b), the unit U10 of inference based on learned data of deep learning uses a still image of a dumpling provided by the unit U9 as an input image, and displays a heat map showing the parts that the neural network has captured as features in the debug view DV10. This heat map is superimposed on the input image and displayed semi-transparently. Note that the drawing is in monochrome, but in practice, it is better to express the increase and decrease of the feature amount in color. Specifically, it is better to display the feature area in a gradation from blue to red according to the feature amount. By displaying such a debug view, the user can visually confirm the parts that have been captured as features in the inference of deep learning (in this example, the open dumpling skin and the protruding meat as a defective product), and can quickly check in real time whether the learning of deep learning has been achieved as intended, and can adjust each unit that constitutes the algorithm. In addition, if it is determined from the display result of the heat map that the learning is not going well, re-learning can be quickly performed based on the result of the debug view.

[0085] In this way, with the programming device 10, it is possible to easily create complex programs that include processes with multiple inputs, such as adding and subtracting images, and processes with multiple outputs, such as conditional branching and separation, by using a graphical programming technique of arranging and linking units.

[0086] As described above, the programming device 10 allows the output of each unit placed in the unit board editing area R2 to be checked at a glance and in real time in the debug view. This allows the validity of the program to be checked and variable adjustments to be made efficiently through trial and error.

[0087] [Extended embodiment] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0088] 10: programming device, 11: communication I / F section, 12: operation input section, 13: screen display section, 14: memory section, 15: calculation processing section

Claims

1. A programming device for editing a program by arranging icons of units representing processing units on an editing screen in response to a user operation and linking these units, comprising: a program execution means for executing a program defined by the units arranged and connected on the editing screen and calculating an output of each unit; a debug view display control means for displaying an output of each unit calculated by the program execution means in a debug view area corresponding to an icon of each of the units arranged on the editing screen; Equipped with The debug view display control means updates the display of the debug view area in real time in response to changes in the output of each unit while the program execution means is executing the program, and controls the display so that the changes in the processing results based on the parameter settings of each unit can be confirmed in a list form on a single screen.

2. The programming device according to claim 1, characterized in that the units constituting the program can be selected from a plurality of options including input to the programming device, output from the programming device, conditional branching, and inference based on learned data of deep learning.

3. The programming device according to claim 1 or 2, characterized in that the debug view display control means displays a heat map in the debug view area indicating the points that are regarded as features in the inference for the unit that performs inference using learned data of deep learning.

4. 4. The programming device according to claim 1, further comprising a unit editing means for accepting changes to set values ​​of each unit constituting the program.

5. the unit editing means is capable of accepting a change in a setting value of a unit while the program execution means is executing the program, 5. The programming device according to claim 4, wherein said program execution means reflects, as required, changes in set values ​​accepted by said unit editing means during execution of said program in operations in said program being executed.

6. The programming device can use a flexible image filter unit capable of selecting a plurality of types of image filters as the unit, The unit editing means includes: a first preview area for displaying a list of preview images obtained by applying each of a plurality of selectable types of image filters to input data in such a manner that the preview images to which a selected image filter has been applied can be identified; a second preview area for displaying a list of preview images obtained when a plurality of settable setting values ​​are applied to characteristic parameters related to the selected image filter in a manner that makes it possible to distinguish the preview images to which the set setting values ​​are applied; a target image display area for displaying a target image, which is a preview image obtained when a selected image filter is applied to input data with the set values ​​of the characteristic parameters, in a larger size than the preview images displayed in the first preview area and the second preview area; 6. The programming device according to claim 4, wherein a setting screen including: a setting unit for setting the flexible image filter unit by a user is received through the setting screen.

7. A program for causing a computer to function as the programming device according to any one of claims 1 to 6.

Images