Electronic device, method and storage medium for setting a control process of a device

Abstract

The present invention relates to an electronic device, method and storage medium for setting a control process of a device. The electronic device is used to set a process for controlling the device, and the electronic device includes: at least one processor configured to enable the electronic device to perform the following operations: obtain a captured image; determine the type of an object included in the captured image; notify an item representing an object process corresponding to the object included in the captured image in multiple analysis processes; notify a result obtained by applying the object process to the captured image; and receive a user instruction for adding a process corresponding to the item to the process for controlling the device.

Description

Technical Field

[0001] The present invention relates to an electronic device, a method and a storage medium for setting a process for controlling a device. Background Art

[0002] There is a control apparatus for controlling a robot such as a robot arm, an actuator, or a conveying mechanism based on an image obtained by photographing a workpiece.

[0003] In this type of control device, there is a device that enables a user to set a processing procedure (flow chart) by combining multiple processing units (functional units) based on a purpose. The user sets the processing procedure in advance so that the processing suitable for the purpose desired by the user can be automatically performed by executing the processing procedure through the control device.

[0004] Japanese Patent Laid-Open No. 2001-178686 discusses displaying the result of applying each of a plurality of image processing units to an input image.

[0005] In a case where a processing unit to be used for generating a processing procedure can be selected from a plurality of types of processing units, it may be difficult for a user to select an appropriate processing unit. Summary of the invention

[0006] The present invention relates to an electronic device, a method, and a storage medium for notifying a user of a processing unit to be used by the user to generate a processing procedure.

[0007] In order to solve the above problems, an embodiment of the present invention provides an electronic device for setting a processing process for controlling a device, the electronic device comprising: at least one processor, which is configured to enable the electronic device to perform the following operations: obtain a captured image; determine the type of object included in the captured image; notify an item representing an object processing corresponding to the object included in the captured image in multiple analysis processes; notify a result obtained by applying the object processing to the captured image; and receive a user instruction for adding a processing corresponding to the item to the processing process for controlling the device.

[0008] Another aspect of the present invention provides a method for setting a processing process for controlling a device, the method comprising: obtaining a captured image; determining the type of object included in the captured image; notifying an item representing an object processing corresponding to the object included in the captured image in multiple analysis processes; notifying a result obtained by applying the object processing to the captured image; and receiving a user instruction for adding a processing corresponding to the item to the processing process for controlling the device.

[0009] Another aspect of the present invention provides a computer-readable storage medium storing a program, wherein the program is used to cause a computer to execute a method for setting a processing process for controlling a device, the method comprising: obtaining a captured image; determining the type of object included in the captured image; notifying an item representing an object processing corresponding to the object included in the captured image in multiple analysis processes; notifying a result obtained by applying the object processing to the captured image; and receiving a user instruction for adding a processing corresponding to the item to the processing process for controlling the device.

[0010] According to the present invention, a processing unit suitable for the type of object included in a captured image is notified to the user. Therefore, the processing unit to be used by the user to generate a processing procedure including obtaining a captured image can be notified to the user.

[0011] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram illustrating a configuration example of an entire system including an image processing apparatus according to an exemplary embodiment.

[0013] Figure 2 is a block diagram illustrating an example of a detailed internal structure of an image processing apparatus according to an exemplary embodiment.

[0014] Figure 3 : is a diagram showing an example of a flowchart creation screen for executing an execution program.

[0015] Figure 4 is a diagram showing a flowchart creation process.

[0016] Figure 5 is a diagram showing an example of a flowchart creation support screen.

[0017] Figure 6 is a flowchart illustrating an example of a processing procedure for supporting creation of a flowchart.

[0018] Figure 7 is a diagram of a table showing the relationship between the types of workpieces and recommended processing items.

[0019] Figure 8 is a diagram showing an example of a flowchart creation support screen.

[0020] Fig. 9 is a diagram showing another example of the flowchart creation support screen.

[0021] Fig.10 is a diagram showing an example of a flowchart creation support screen including two types of artifacts. DETAILED DESCRIPTION

[0022] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to be limiting. A plurality of features are described in the embodiments, but the features of these embodiments may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are provided to the same or similar structures, and their repeated descriptions are omitted.

[0023] <First Exemplary Embodiment>

[0024] Embodiments according to the present invention will be described with reference to the accompanying drawings.

[0025] Figure 1 1 is a diagram showing a configuration example of the entire system including the image processing apparatus 101. The system includes the image processing apparatus 101, a camera 102, a robot 103, a robot control device 104, a client terminal 105, and a lighting device 106. The system is used to place a workpiece 108 arranged on a work table 107 on any of the placement tables 109 to 111 by using the robot 103. The use of the system is not limited thereto.

[0026] The image processing device 101 controls the camera 102 and the robot 103. The image processing device 101 controls the robot 103 via the robot control device 104. The image processing device 101 can directly control the robot 103 at a predetermined timing based on an internal program or a control instruction from an external control device. The image processing device 101 will be described as a device in a housing different from the respective housings of the robot 103 and the robot control device 104; however, the image processing device 101 may be in the same housing as any of the housing of the robot 103 and the housing of the robot control device 104.

[0027] The image processing device 101 sets the shooting conditions (shooting parameters) of the camera 102 and issues a shooting instruction. The image processing device 101 obtains a shot image from the camera 102 and applies image processing to the obtained image. The image processing device 101 outputs the processed shot image to the client terminal 105 to display the image. The image processing device 101 sends a control instruction to the robot control device 104.

[0028] The image processing apparatus 101 performs control processing for the camera 102, the robot 103, the robot control apparatus 104, the client terminal 105, and the lighting apparatus 106 and image processing for the obtained image based on a predefined processing procedure. The processing procedure is set as a combination of a plurality of processing units executable by the image processing apparatus 101. The image processing apparatus 101 is an electronic device that can create, edit, and update a program describing the processing procedure.

[0029] The camera 102 is a photographing device that photographs an image including an object (hereinafter, "workpiece"). The camera 102 has a mechanism for changing various photographing parameters including zoom magnification, focal length, aperture, and signal amplification. These photographing parameters can be determined individually based on instructions from the image processing device 101.

[0030] The robot 103 is an industrial device driven based on instructions from the robot control device 104. In the present embodiment, the robot 103 includes a gripping mechanism (e.g., a hand) and an arm mechanism (e.g., an arm) for moving the gripping mechanism. The robot 103 is a mechanical arm that grips and moves a workpiece by controlling the hand and the arm. The camera 102 is attached to the arm of the robot 103. Thus, the position of the camera 102 can be controlled by moving the arm of the robot 103.

[0031] The robot control device 104 receives a control instruction from the image processing device 101 , and controls the movement of each axis and hand of the robot 103 .

[0032] The client terminal 105 includes a display, an input device, and a control device, and the client terminal 105 is connected to the image processing apparatus 101. The display is a display device such as a liquid crystal display. The input device is an input unit such as a keyboard, a mouse, an input operation controller, and a gesture input device. The input device is used for various operation inputs. For the input device, a touch panel configured in the display can be used.

[0033] The client terminal 105 displays an image on a display based on an image input from the image processing apparatus 101. A user can input instructions into the image processing apparatus 101 by operating an input device of the client terminal 105. A control device of the client terminal 105 is connected to the image processing apparatus 101 to send and receive images and instructions. The display, input device, and control device of the client terminal 105 will be collectively referred to as the client terminal 105.

[0034] The lighting device 106 is configured by, for example, a halogen lighting unit or a light emitting diode lighting unit.

[0035] Figure 2 2 is a block diagram showing an example of a detailed internal structure of the image processing apparatus 101. The image processing apparatus 101 includes an interface (I / F) 201, a calculation unit 202, and a storage unit 210.

[0036] The image processing device 101 is connected to each of the camera 102, the robot control device 104, the client terminal 105, and the lighting device 106 via an interface 201 arranged on an internal bus. The interface 201 is configured based on a standard suitable for communicating with the above-mentioned respective devices. The interface standard can be selected from interfaces such as a network interface and a serial communication interface.

[0037] The calculation unit 202 is a control unit for controlling the processing performed by the image processing apparatus 101. The calculation unit 202 is configured by, for example, a central processing unit (CPU) which is a general-purpose microprocessor or an image processing processor.

[0038] The storage unit 210 is, for example, a read-only memory (ROM), a random access memory (RAM), or a nonvolatile memory device such as an erasable programmable ROM (EPROM) or an electrically erasable programmable ROM (EEPROM). The image processing apparatus 101 can be connected to an external storage device to expand a data storage area. The storage unit 210 can be configured by an area such as a file area or a virtual storage area on an external storage device connected to the image processing apparatus 101.

[0039] The storage unit 210 includes a program storage area 211 for storing a program executed by the calculation unit 202 and a storage area 212 for storing processing data. The program storage area 211 stores an execution program 213 and a support program 214. The execution program 213 is used to execute a processing procedure. The support program 214 is used to support the user to set the processing procedure described in the execution program 213. The storage area 212 is configured by a RAM area of ​​the storage unit 210, or an area such as a file area or a virtual storage area of ​​an external storage device. The storage area 212 is used to temporarily store processing data, and stores data such as setting parameters used for image processing.

[0040] As an example of the execution program 213 stored in the program storage area 211, a processing procedure is set; the processing procedure is performed by placing the workpiece 108 arranged on the workbench 107 on any of the placement tables 109 to 111 by the robot 103. Specifically, the image processing device 101 controls the position and posture of the robot 103 so that the camera 102 is located at a position (photographing position Pa) where the area including the workpiece 108 can be photographed from above the workbench 107. After moving the camera 102 to the photographing position Pa, the image processing device 101 controls the camera 102 so that the camera 102 photographs the area including the workpiece 108. The image processing device 101 obtains a photographed image from the camera 102, performs image processing (analysis processing) on ​​the photographed image, and obtains parameters representing the position and posture of the workpiece 108. The image processing device 101 controls the robot 103 to correct the gripping position of the workpiece 108 based on the parameters representing the position and posture obtained by the analysis processing. The image processing apparatus 101 controls the movement of the robot 103 based on the parameters indicating the position and posture of the workpiece 108 so that the arm of the robot 103 holds the workpiece 108. The image processing apparatus 101 also aligns and places the held workpiece 108 on any of the placement tables 109 to 111. Figure 1 An example is shown in which the workpiece 112 is placed, and a predetermined number of workpieces are aligned and placed on each of the placement tables 109 to 111. After or during processing by the image processing apparatus 101, information such as progress, results, and responses of the processing may be appropriately displayed on the display device.

[0041] When the computing unit 202 executes the support program 214, the computing unit 202 displays a screen showing the processing procedure of the execution program 213 in the form of a flowchart on the display of the client terminal 105. The computing unit 202 displays an editing screen for updating the processing procedure of the execution program 213 on the display of the client terminal 105. Figure 3 2 is a schematic diagram showing a flowchart representing a processing procedure of the execution program 213 and an editing screen. In another exemplary embodiment, the execution program 213 may be stored in the storage unit 210 of the image processing apparatus 101 after being updated by another electronic device.

[0042] The screen (editing screen) displayed on the display of the client terminal 105 includes an editing area 301 and a flowchart area 302. The editing area 301 includes a list 320 for selecting a plurality of processing parts for editing a flowchart and a list box 316 for selecting an execution program 213 as an object of creation or editing.

[0043] The editing area 301 includes a button (OK button) 317 and a button (Cancel button) 318. The button 317 is used to complete the creation or editing of the execution program 213 and store the created or edited execution program 213 in the save area. The button 318 is used to cancel the creation or editing of the execution program 213.

[0044] The editing area 301 includes a button (creation support button) 319 for starting a function that supports creation of a flowchart. The flowchart area 302 is an area for displaying a flowchart representing a processing procedure of the execution program 213. The user can set the processing procedure of the execution program 213 by operating the client terminal 105 to edit the flowchart displayed in the flowchart area 302.

[0045] List 320 is a list of processing parts (processing units) for creating a flowchart representing the processing procedure of execution program 213. The user drags a processing part from list 320 using the input device of client terminal 105, drops the dragged part into flowchart area 302, and connects the dropped part with a line, thereby creating a flowchart.

[0046] List box 316 is used to select a flowchart to be displayed on the screen. By operating the input device of client terminal 105 to select in list box 316, the user can switch from one flowchart to be displayed to another flowchart, or input a new flowchart name into list box 316. Figure 3 In the example shown, the processing process is shown when the type of the workpiece is a box. When the type of the workpiece is other types, some of these processing parts are different. Therefore, when creating a flowchart for processing other types of workpieces, it may be desirable to switch the flowchart to be displayed by operating the list box 316.

[0047] The button 317 is a button for providing an instruction to store the created flowchart and update the execution program 213. The cancel button 318 is a button for discarding the settings until the time when the button is pressed. The creation support button 319 is a button for starting a function that supports the creation of a flowchart. The processing in the case where the creation support button 319 is selected will be described in detail below.

[0048] The flowchart 330 shown in the flowchart area 302 describes the processing process of the above series of controls. Figure 3 The various processing parts are shown.

[0049] The processing section 304 is a moving section representing a process (movement process) for moving the arm mechanism of the robot 103. When the computing unit 202 executes the process of the processing section 304, the arm mechanism of the robot 103 moves the camera 102 to a point above the work table 107, so that the camera 102 attached to the arm mechanism of the robot 103 moves to a position where the workpiece 108 on the work table 107 can be photographed.

[0050] The processing section 305 is a photographing section representing a process (photographing process) for performing a photographing operation using the camera 102. When the calculation unit 202 performs the process of the processing section 305, a control command for performing the photographing process using the camera 102 is output.

[0051] The processing section 306 is a position measurement section representing a position measurement process for measuring the position of the workpiece 108. The processing section 307 is a phase measurement section representing a phase measurement process for measuring the phase of the workpiece 108. As will be described in more detail below, each of these processing sections is a section for measuring the workpiece 108 using characteristic information such as brightness gradient, shading information, and color information of image information.

[0052] The processing section 308 is a correction movement section representing a correction movement process for relatively moving the position and posture of the hand of the robot 103. When the computing unit 202 executes the process of the processing section 308, the computing unit 202 performs correction movement such as rotation of the hand (end effector) of the robot 103 by controlling the robot 103 in a correction manner based on the position and phase of the workpiece measured by the processing sections 306 and 307.

[0053] The processing section 309 is a moving section representing a process (movement process) for moving the arm mechanism of the robot 103. When the computing unit 202 executes the process of the processing section 309, the arm of the robot 103 is controlled so that the hand of the robot 103 is located at a position where the workpiece 108 can be grasped (picked up) (for example, a position directly above the workpiece 108).

[0054] The processing section 310 is a hand control section representing a hand control process for controlling the opening / closing of the hand of the robot 103 and the suction amount of the suction device. When the calculation unit 202 executes the process of the processing section 310, the opening / closing of the hand of the robot 103 is controlled to hold the workpiece 108.

[0055] The processing section 311 is a moving section representing a process (movement process) for moving the arm mechanism of the robot 103. When the computing unit 202 executes the process of the processing section 311, the movement of the arm is controlled to move the hand of the robot 103 to a point above the placement table suitable for the type of workpiece.

[0056] The processing section 312 is a moving section representing a process (movement process) for moving the arm mechanism of the robot 103. When the computing unit 202 executes the process of the processing section 312, the movement of the arm is controlled to move the hand of the robot 103 to a point above the position where the workpiece 108 is to be placed.

[0057] The processing section 313 is a hand control section representing a hand control process for controlling the opening / closing of the hand and the suction amount of the suction device of the robot 103. When the calculation unit 202 executes the process of the processing section 313, the opening / closing of the hand and the suction amount are controlled so that the workpiece 108 is placed.

[0058] The processing section 314 is a moving section representing a process (movement process) for moving the arm mechanism of the robot 103. When the calculation unit 202 executes the process of the processing section 314, the arm of the robot 103 is moved to a point above the placement table.

[0059] The processing section 315 is a branch section representing a judgment process for judging a conditional branch. When the calculation unit 202 executes the process of the processing section 315, the calculation unit 202 judges whether the number of times (execution count) of a series of processes of the processing sections 304 to 314 has been executed is greater than or equal to a threshold value. In the case where the execution count is less than the threshold value, the calculation unit 202 returns the process to the processing section 304. In the case where the execution count is greater than or equal to the threshold value, the calculation unit 202 terminates the process (terminates the process).

[0060] In response to a specific operation performed by the user, such as double-clicking any of the processing sections of the flowchart 330 displayed in the flowchart area 302 by operating the client terminal 105, a transition to the setting screen of the processing section occurs. For example, in the case of a moving section, parameters such as a target position (coordinates), a moving speed, and a moving path for moving an arm can be set in the setting screen.

[0061] Next, the following will describe the Figure 3 The flowchart shown is a case of problems and their solutions.

[0062] Figure 4 is a diagram showing a flowchart creation process. Figure 4 The editing screen is shown. The editing area 301, the flowchart area 302, and the processing parts and buttons included in these areas correspond to Figure 3 These areas as well as the processing portions and buttons are not shown, so description will be omitted.

[0063] The user starts working to create a flowchart representing a processing procedure for placing the workpiece 108 arranged on the work table 107 on any of the placement tables 109 to 111 using the robot 103 .

[0064] The user first creates a process for obtaining a captured image to identify the workpiece 108 located on the work table 107 .

[0065] The user first selects (drags) a moving part from the list 320 by operating the input device of the client terminal 105, and moves (drops) the selected moving part to the flowchart area 302 in the direction indicated by the arrow 401. Even for the arm that moves the camera 102 to the shooting position, the coordinate information indicating the target position is pre-stored as the default of the target position of the moving part in the storage area 212 of the storage unit 210. Thus, the user can move the robot 103 to a point above the workbench 107 only by selecting the moving part.

[0066] The user selects a shooting part and moves the selected shooting part to the flowchart area 302 in the direction indicated by the arrow 402 by operating the input device of the client terminal 105. The moving part and the shooting part are connected by a straight line indicating continuous processing. The following information is also pre-stored in the storage area 212 of the storage unit 210: shooting parameters (e.g., exposure and zoom ratio) used in the case of executing the shooting part, and a determination method for the shooting parameters (e.g., fixed value use, automatic exposure, and automatic focus). Thus, the user can shoot using shooting parameters only by selecting the shooting part.

[0067] The calculation unit 202 performs the respective processes of the processing section 304 and the processing section 305 , so that the arm of the robot 103 moves, and the camera 102 attached to the arm moves to a predetermined photographing position. Subsequently, the camera 102 performs photographing.

[0068] Next, the user creates a process for holding the workpiece 108 based on the captured image as a flowchart. In this process, if the user does not have detailed knowledge about image processing technology, there may be a case where the user cannot judge which part of the processing parts included in the list 320 is an appropriate selection.

[0069] In the present embodiment, creation support processing for notifying the user of a processing section (processing unit) to be placed at a subsequent stage based on a captured image is performed in order to assist the user in selecting a processing section related to image processing.

[0070] Figure 5 Is shown in the selection Figure 4 Schematic diagram of the support screen 501 to which the support button 319 is transformed when the support button 319 is created. Figure 5The support screen 501 can be displayed immediately after Figure 3 The flowchart creation screen is displayed.

[0071] The support screen 501 includes a display area 502, a display area 503, and a processing item area 504. The display area 502 shows the captured image and the result of the image processing described below. The display area 503 shows the result of analyzing the type of the workpiece included in the captured image. The processing item area 504 shows one or more candidates of the processing unit corresponding to the type of the workpiece obtained by the analysis. The support screen 501 includes buttons 505 to 508 for controlling the creation of the support processing.

[0072] Button 505 is a button for further displaying processing items that are not currently displayed in processing item area 504. When there is no suitable processing item among the recommended items currently displayed in processing item area 504, or when comparison with other processing items is to be performed, different recommended processing items can be displayed by selecting button 505.

[0073] The button 506 is a button for performing (updating) input of a captured image, object recognition, and presentation of a recommended process again, for example. The button 506 is used in cases such as when switching to a clearer captured image of the same workpiece is performed, and when a change occurs to a captured image of a different workpiece of the same type. A mode such as a live view mode in which a captured image is input in real time and the captured image is periodically updated can be adopted.

[0074] Button 507 is a button for determining a processing item to be adopted as a part. When button 507 is selected, the selection result in processing item area 504 is stored in storage area 212 of image processing apparatus 101 in association with execution program 213. In the case of selecting button 508, Figure 5 The processing results in the screen shown are displayed, and the screen returns to Figure 4 The screen shown.

[0075] Next, a flowchart creation support process that the calculation unit 202 starts executing in response to transition to the support screen 501 will be described.

[0076] Figure 6 2 is a flowchart showing the support processing performed by the computing unit 202. The computing unit 202 reads out the support program 214 stored in the storage unit 210 and executes the read support program 214, so that Figure 6 each step of the processing.

[0077] In step S601, the computing unit 202 obtains a captured image of the workpiece. The captured image obtained in step S601 is obtained by capturing a workpiece placed on the workpiece from an upper position. Figure 1The image shown is a captured image of the workpiece 108 on the work table 107. The captured image obtained in step S601 may be an image captured in response to a user pressing a capture instruction button (not shown), or may be a live view image.

[0078] In step S602, the calculation unit 202 performs processing for determining (distinguishing) the type of the workpiece in the captured image. Specifically, for example, the calculation unit 202 separates the workpiece and the background based on background difference or threshold extraction processing, and determines the type of the workpiece using image characteristics such as shape, color, and shading distribution for the extracted workpiece. The type of the workpiece can be determined by making a prediction from a learning model learned in advance using, for example, a neural network. Information indicating the type of the workpiece distinguished by the calculation unit 202 is displayed in the display area 503 of the support screen 501.

[0079] In step S603 , the calculation unit 202 identifies one or more processing units (processing items) corresponding to the type of the workpiece discriminated in step S602 . Figure 7 This is a diagram showing related information indicating the relationship between the types of workpieces and the corresponding processing units.

[0080] exist Figure 7 In , identification (ID) is an ID number assigned to each type of workpiece. Examples of the type of workpiece include "box", "ring", "one-dimensional / two-dimensional (1D / 2D) code", and "character". Figure 7 In the , "candidate processing" is a processing unit (processing item) corresponding to each type of workpiece.

[0081] like Figure 7 As shown, the content of the corresponding candidate process varies depending on the type of the workpiece. For example, in the case where the type of the workpiece is "box", the position measurement process, phase measurement process, defect detection process, and area measurement process are each pre-set as a candidate process. In the case where the type of the workpiece is "ring", the position measurement process, outer diameter measurement process, inner diameter measurement process, and counting process are each pre-set as a candidate process.

[0082] For a plurality of processing items each set as a "candidate process", a priority is set in advance. For example, in a case where a workpiece is discriminated as a "box" as a result of object recognition, since being able to process with a robot is considered to be the most important, a high priority is set for each of position measurement and phase measurement. In a case where a workpiece is discriminated as a "1D / 2D code" or "character", a high priority is set for data reading processing so that character strings or numerical data can be read. Figure 7 The table shown in is stored in advance in the storage area 212 of the storage unit 210.

[0083] Can learn and update when repeating support process Figure 7 The correspondence between the types of workpieces in the table shown and the processing items in the candidate processing and the priorities of the processing items in the candidate processing are shown to increase the priority of the processing items that are selected more frequently. The table can be shared by multiple image processing devices via a network, and the information can be updated. If the table in which the types of workpieces and candidate processing (and priorities) are associated with each other is updated by learning or sharing via the network, the priority of the frequently used processing items is increased. Therefore, it is possible to more practically recommend processing items with high priorities.

[0084] In step S603, the result of obtaining one or more processing units (processing items) corresponding to the type of workpiece identified by the calculation unit 202 in step S602 is displayed in the processing item area 504 of the support screen 501. In the processing item area 504, processing items with high priority are displayed at high positions.

[0085] A check box is also displayed for each processing item in the processing item area 504 of the support screen 501. The user selects a processing item using the check box to preview the processing result of the processing item. For example, position measurement processing and phase measurement processing are selected. The processing item indicating the processing result may also be each processing unit in all one or more processing units (processing items) corresponding to the type of the workpiece identified in step S602.

[0086] In step S604 , the computing unit 202 executes image processing corresponding to the selected processing item for the workpiece included in the captured image.

[0087] In step S605 , the calculation unit 202 displays the result of the image processing performed in step S604 in the display area 502 .

[0088] As described above, in the case where the position measurement process and the phase measurement process are selected, the result of the phase measurement process and the result of the position measurement process are each displayed in the display area 502 .

[0089] In step S606, the calculation unit 202 determines whether the user has selected the button 505 by operating the input device of the client terminal 105. In other words, the calculation unit 202 determines whether the user has requested to switch the display of the processing item area 504 so that another processing item can be selected.

[0090] If the calculation unit 202 determines that the button 505 is selected ("Yes" in step S606), the calculation unit 202 switches the display of the processing item area 504 so that a plurality of processing items including at least one processing item different from the processing item currently displayed in the processing item area 504 are displayed in the processing item area 504. The process returns to step S603. If the calculation unit 202 determines that the button 505 is not selected ("No" in step S606), the process proceeds to step S607.

[0091] In step S607, the calculation unit 202 determines whether the user has completed the selection of the processing item. Specifically, the calculation unit 202 determines whether the user has selected the button 507 by operating the input device of the client terminal 105. When the user selects the button 507, the selected processing item is stored in the storage unit 210 as the selected processing item. For example, in a state where the processing item corresponding to the position measurement processing and the processing item corresponding to the phase measurement processing are selected, the button 507 is pressed. The calculation unit 202 causes the display unit of the client terminal 105 to change from the support screen 501 to Figure 4 If the calculation unit 202 determines that the user's selection of the processing item is not completed (NO in step S607), the process returns to step S601.

[0092] If the calculation unit 202 determines that the user's selection of the processing item is complete (YES in step S607), the process advances to step S608. In step S608, the calculation unit 202 adds the processing item selected in step S607 as a processing part to the flowchart being created in the editing screen.

[0093] Figure 8 It is shown in Figure 6 An example of a screen displayed after the processing of Figure 8 As shown, a position measurement part and a phase measurement part are added to the flowchart 801. In the flowchart area 302, editing such as changing or deleting the added position measurement part and the added phase measurement part can be performed.

[0094] Subsequently, the user adds the correction movement part to the flowchart 801 by selecting the correction movement part in the list 320, dragging the selected correction movement part, and dropping the dragged correction movement part into the flowchart area 302. Likewise, parts such as the movement part and the hand control part are selected from the list 320, and the selected processing parts are sequentially added to the flowchart 801. In the case of holding the workpiece, the case of lifting the workpiece, the case of moving the workpiece to the placement table, and the case of placing the workpiece, detailed parameters such as the coordinate information of the movement destination and the height of the robot are also stored in advance in the storage area 212 of the storage unit 210. Thus, the user can set control such as holding of the workpiece by the robot or movement to the placement table by only selecting the processing part.

[0095] Since the type of workpiece is discriminated, there may be a case where it is desired to change the detailed settings of the processing section. In the case of measuring objects such as position, phase, and area, generally, a captured image in which the shading ratio between the workpiece and the background is high and the amount of change in the color component is large can be input to reliably separate the workpiece and the background. In the case of detecting defects or dirt, since color or shading change information is generally important, a captured image in which the workpiece is widely distributed in the dynamic range can be input.

[0096] As described above, when the user double-clicks the processing section to change the detailed settings of the processing section by operating the input device of the client terminal 105, the current screen is changed to a setting screen for setting the detailed processing of the processing section. In the setting screen, in the case of measuring an object such as position, phase, and area, the shooting parameters can be used to make adjustments to increase the contrast. In addition, in the case of detecting defects or dirt, settings such as exposure value, signal amplification ratio, and white balance can be changed.

[0097] As described above, according to the present embodiment, the type of the workpiece is discriminated from the captured image of the workpiece, and the processing item associated with the workpiece is notified. In addition, the processing result of the processing item is also notified. This enables the user to select a processing item to be added to the flowchart based on the notified processing item and the notified processing result. Since the recommended processing item is notified, the user can omit the work for combining various processing items or confirming the processing items combined in advance.

[0098] In the present embodiment, in the support screen, recommended processing items are notified in association with the type of workpiece, and a processing section related to image processing is added. The processing items can be added to the flowchart in combination with the control of the robot (such as the movement of the robot, etc.). For example, in the case where it is determined as a result of object recognition that the type of the workpiece is a box, the robot correction movement section, the robot movement section, and the robot hand control section can also be added to the flowchart as processing in the subsequent stage. In this way, the processing section closely related to the robot control is added to the flowchart when the support screen is displayed, so that the user can further omit the work of creating the flowchart.

[0099] In this embodiment, the captured images and the image processing results are displayed in the display area 502 of the support screen. The display area 502 can be divided into a plurality of areas, and a plurality of captured images and the processing results of these captured images can be displayed simultaneously in each area.

[0100] Fig. 9 It is a schematic diagram showing a modified example of the support screen. Fig. 9 The support screen 901 shown includes a plurality of display areas 902. Each display area 902 corresponds to at least one processing item, and displays the result of the image processing corresponding to each processing item. Fig. 9 In the image processing, the result of the image processing is displayed for each of the four processing items. The user can select a processing item to be added as a processing part while confirming the result of each image processing and comparing the results.

[0101] <Second Exemplary Embodiment>

[0102] The first embodiment has been described on the premise that only one workpiece exists in the captured image to be subjected to object recognition processing. In this embodiment, an example will be described in which two or more workpieces are included in the captured image and the workpieces are of different types. In this embodiment, when two or more workpieces are distinguished, a flowchart creation support screen for each workpiece is started. Only points different from the first embodiment are described below.

[0103] Fig.10 1 is a diagram showing an example of a support screen 1001 in a case where a plurality of workpieces are identified. Fig.10 When multiple artifacts are identified, each artifact is displayed in the display area 1002, and an identifier (hereinafter referred to as an "artifact ID") is assigned to each artifact. Fig.10 In the example shown, the box 1011 is assigned a workpiece ID “ 1 ”, and the donut-shaped workpiece 1012 is assigned a workpiece ID “ 2 ”.

[0104] When object recognition and measurement are performed for each workpiece, the result of image processing is displayed for each workpiece in display area 1002. Analysis results of the type of workpiece displayed in display area 503 and processing item area 504 are displayed only for the selected workpiece. The workpiece ID can be changed with a switch button 1010.

[0105] The process of creating a flowchart for robot control is basically similar to Figure 6 However, in the case where a flowchart for robot control is to be created for workpieces having different workpiece IDs, steps S601 to S604 may be omitted. In other words, according to the present embodiment, when a plurality of types of workpieces are identified as objects in one captured image, a part of the processing in the process for creating a flowchart may be omitted.

[0106] Other embodiments

[0107] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0108] Although the present invention includes exemplary embodiments, it is to be understood that the present invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An electronic device for setting a process for controlling a device, the electronic device comprising: At least one processor is configured to cause the electronic device to perform the following operations: Obtaining captured images; determining the type of the object included in the captured image by using image characteristics; Identifying one or more processing items among a plurality of processing items according to the determined type of the object, wherein the identifying comprises: identifying a first processing item corresponding to the first type among the plurality of processing items when the determined type of the object is a first type, and identifying a second processing item corresponding to the second type among the plurality of processing items when the determined type of the object is a second type different from the first type; displaying the identified one or more processing items on a screen; selecting at least one of the one or more displayed processing items in response to a user indication; and The processing procedure is set in response to a user instruction to add the selected processing item to the flowchart.

2. The electronic device according to claim 1, further comprising a storage unit, wherein the storage unit is used to store the relevant information representing the correspondence between the type of the object and the processing item in a storage medium, in, A processing item corresponding to the type of the object included in the captured image is identified based on the relevant information.

3. The electronic device according to claim 2, wherein: The related information also includes information indicating the priority of the processing item corresponding to the type of the object.

4. The electronic device according to claim 1, wherein: Each of the plurality of processing items includes at least one of image processing for measuring a position of an object included in the captured image, image processing for measuring a size of the object, and image processing for analyzing information represented in the object.

5. The electronic device according to claim 1, wherein: The device is an industrial device configured to move an object included in the captured image.

6. The electronic device according to claim 1, wherein: The at least one processor is configured to cause the electronic device to perform the following further operations: In a case where a plurality of objects are included in the captured image, identifying a processing item for each of the plurality of objects; and The processing items and results of applying the processing items to the corresponding objects are displayed.

7. A method for setting a process for controlling a device, the method comprising: Obtaining captured images; determining the type of the object included in the captured image by using image characteristics; Identifying one or more processing items among a plurality of processing items according to the determined type of the object, wherein the identifying comprises: identifying a first processing item corresponding to the first type among the plurality of processing items when the determined type of the object is a first type, and identifying a second processing item corresponding to the second type among the plurality of processing items when the determined type of the object is a second type different from the first type; displaying the identified one or more processing items on a screen; selecting at least one of the one or more displayed processing items in response to a user indication; and The processing procedure is set in response to a user instruction to add the selected processing item to the flowchart.

8. The method according to claim 7, further comprising: storing the relevant information representing the correspondence between the type of the object and the processing item in a storage medium, Wherein, a processing item corresponding to the type of the object included in the captured image is identified based on the relevant information.

9. The method according to claim 8, wherein: The related information also includes information indicating the priority of the processing item corresponding to the type of the object.

10. The method according to claim 7, wherein: Each of the plurality of processing items includes at least one of image processing for measuring a position of an object included in the captured image, image processing for measuring a size of the object, and image processing for analyzing information represented in the object.

11. The method according to claim 7, wherein: The device is an industrial device configured to move an object included in the captured image.

12. The method according to claim 7, further comprising: In a case where the captured image includes a plurality of objects, identifying a processing item for each of the plurality of objects; as well as The processing items and results of applying the processing items to the corresponding objects are displayed.

13. A computer-readable storage medium storing a program for causing a computer to execute a method for setting a process for controlling a device, the method comprising: Obtaining captured images; determining the type of the object included in the captured image by using image characteristics; Identifying one or more processing items among a plurality of processing items according to the determined type of the object, wherein the identifying comprises: identifying a first processing item corresponding to the first type among the plurality of processing items when the determined type of the object is a first type, and identifying a second processing item corresponding to the second type among the plurality of processing items when the determined type of the object is a second type different from the first type; displaying the identified one or more processing items on a screen; selecting at least one of the one or more displayed processing items in response to a user indication; and The processing procedure is set in response to a user instruction to add the selected processing item to the flowchart.

Images