Program, information processing method, and information processing device

Abstract

Disclosed is a program that causes a computer to execute: first output processing for outputting an input screen including an image in a prescribed area; input acquisition processing for acquiring a user input from a user; image processing for image-recognizing a target having a boundary corresponding to the position or an area pertaining to a designated input by performing image recognition processing on the image on the basis of the user input including the designated input of the position or the area with respect to the prescribed area or the image; second output processing for generating a prescribed display representing an image recognition result of the image processing, and outputting the prescribed display in association with the target; and setting processing for generating setting information pertaining to the target pertaining to the prescribed display on the basis of a user input including a setting instruction from the user.

Description

Program, information processing method, and information processing device

[0001] The present disclosure relates to a program, an information processing method, and an information processing device.

[0002] A technology is known in which an image recognition unit performs image recognition to recognize parts in the parts storage unit using images acquired by a camera that captures images of the parts in the parts storage unit, and when the image recognition recognizes that a part is present in the picking section, the robot control unit controls the robot to pick up the recognized part (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2018-202599

[0004] However, when performing various processes based on image recognition of various objects, not just parts in the parts storage unit, settings are required for preparation. Therefore, the present disclosure enables efficient image-based settings via a user interface.

[0005] According to one aspect of the present disclosure, there is provided a program that causes a computer to execute the following steps: a first output process that outputs an input screen including an image in a predetermined area; an input acquisition process that acquires user input from a user; an image process that performs image recognition processing on the image based on the user input including a specified input of a position or area for the predetermined area or the image, thereby image-recognizing an object having a boundary corresponding to the position or area related to the specified input; a second output process that generates a predetermined display representing the image recognition result of the image processing and outputs the predetermined display in association with the object; and a setting process that generates setting information for the object related to the predetermined display based on the user input including a setting instruction from the user.

[0006] According to the present disclosure, image-based settings can be efficiently performed via a user interface.

[0007] 1 is a configuration diagram showing an example of an information processing apparatus according to the present embodiment. FIG. 1 is a diagram showing an example of the hardware configuration of a processing unit of the information processing apparatus shown in FIG. 1. FIG. 2 is a block diagram for explaining functions of the information processing apparatus according to the present embodiment. FIG. 3 is a diagram showing examples (part 3) of an input screen output by the input screen output processing unit when the user selects the teaching mode. FIG. 4 is a diagram showing examples (part 4) of an input screen output by the input screen output processing unit when the user selects the teaching mode. FIG. 5 is a schematic flowchart for explaining the flow of processing executed when an input screen for the teaching mode is selected. FIG. 6 is a schematic flowchart for explaining the flow of processing executed when an input screen for the picking execution mode is selected. FIG. 7 is an explanatory diagram of FIG. 9 showing an example (part 1) of an input screen for the picking execution mode. FIG. 8 is an explanatory diagram of FIG. 9 showing an example (part 2) of an input screen for the picking execution mode. FIG. 9 is an explanatory diagram of FIG. 9 showing an example (part 3) of an input screen for the picking execution mode. 10 is a timing chart showing an example of information exchange between components when picking is performed.FIG. 11 is a diagram showing an input screen output by an input screen output processing unit according to another embodiment.FIG.

[0008] Each embodiment will be described in detail below with reference to the accompanying drawings.

[0009] FIG. 1 is a diagram showing an example of the configuration of a system 1 including an information processing device 20 according to this embodiment.

[0010] The system 1 includes an information processing device 20 , a robot arm controller 40 , a robot arm 42 , and a camera 50 .

[0011] The information processing device 20 is, for example, a PC (Personal Computer), but may also be realized by a mobile phone, a smartphone, a tablet terminal, etc. The information processing device 20 may also be realized by a combination of multiple PCs that are communicatively connected via a network. The information processing device 20 may also be realized by a combination of PCs and server computers that are communicatively connected via a network.

[0012] The information processing device 20 realizes a setting support function according to this embodiment. The setting support function is a function that supports the user in setting various settings based on images, and the details of the setting support function will be described later with reference to FIG. 3 and subsequent figures.

[0013] In the example shown in FIG. 1 , the information processing device 20 includes a communication unit 21 , a storage unit 22 , a display unit 23 , an input unit 24 , and a control unit 25 .

[0014] The communication unit 21 includes an interface for communicating with an external device wirelessly or via a wired connection to send and receive information. The communication unit 21 can communicate with the robot arm controller 40 and the camera 50 via a network.

[0015] The storage unit 22 includes, for example, a primary storage device and a secondary storage device. For example, the storage unit 22 may include a semiconductor memory, a magnetic memory, an optical memory, or the like. The storage unit 22 stores various information and programs related to the setting assistance function. The information and programs related to the setting assistance function may be acquired from an external device via an external memory or the communication unit 21. For example, a setting assistance application program may be acquired from a predetermined application distribution server. Hereinafter, the application program may also be simply referred to as an application. The setting assistance application may be in the form of a web application.

[0016] The display unit 23 includes a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 23 is capable of displaying a variety of images. The display unit 23 may be configured as a touch panel, for example. The display unit 23 may be built into a head-mounted display.

[0017] The input unit 24 may include physical keys, and may further include any input interface including a pointing device such as a mouse.

[0018] The control unit 25 controls the overall operation of the information processing device 20. The control unit 25 executes a setting assistance application to perform processing related to the setting assistance function. For example, the control unit 25 outputs a GUI (Graphical User Interface) (described later with reference to FIG. 4 ) that detects operation inputs on the screen of the display unit 23. The control unit 25 receives various inputs by touch operations, input operations, etc. via the input unit 24, and performs processing related to the setting assistance function in accordance with the various inputs.

[0019] The robot arm controller 40 is communicatively connected to the information processing device 20. The robot arm controller 40 controls the robot arm 42. Note that some or all of the functions of the robot arm controller 40 may be realized by the information processing device 20. The robot arm 42 may be, for example, an articulated arm. The robot arm 42 may be used for any purpose, but in this embodiment, as an example, it is used for picking up objects. In this case, the robot arm 42 may have a gripping portion (hand) at the tip, and may be able to grasp an object by opening and closing the gripping portion, etc.

[0020] The camera 50 is positioned so as to be able to capture images of various objects, including objects to be picked by the robot arm 42 (hereinafter also referred to as "picking objects"). For example, the camera 50 may be positioned in association with a conveyor in a factory line and capture images of various objects moving on the conveyor in real time. Image data captured by the camera 50 may be supplied to the information processing device 20. The camera 50 may include a camera that captures RGB color images (hereinafter also referred to as "RGB images") and / or a depth camera that captures depth images. The camera 50 may also include a camera that uses electromagnetic waves having wavelengths outside the wavelength range of visible light, such as infrared or ultraviolet rays (for example, a camera equipped with an element that forms an image on a plane).

[0021] FIG. 2 is a diagram showing an example of the hardware configuration of the processing unit 10 of the information processing device 20 shown in FIG.

[0022] In the example shown in FIG. 2 , the processing unit 10 includes a CPU (Central Processing Unit) 11, a GPU (Graphics Processing Unit) 11A, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an auxiliary storage device 14, a drive device 15, and a communication interface 17, all connected by a bus 19, as well as a wired transceiver unit 18A and a wireless transceiver unit 18B connected to the communication interface 17.

[0023] The GPU 11A performs image processing such as drawing various images, which will be described later.

[0024] The auxiliary storage device 14 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and is a storage device that stores data related to application software and the like.

[0025] The wired transceiver 18A includes a transceiver capable of communicating over a wired network. Peripheral devices such as the input unit 24 and the display unit 23 shown in Fig. 1 are connected to the wired transceiver 18A. However, some or all of the peripheral devices may be connected to the bus 19 or to the wireless transceiver 18B.

[0026] The wireless transceiver 18B is a transceiver capable of communicating using a wireless network. The wireless network may include a wireless communication network for mobile phones, the Internet, a virtual private network (VPN), a wide area network (WAN), etc. The wireless transceiver 18B may also include a near field communication (NFC) unit, a Bluetooth (registered trademark) communication unit, a Wi-Fi (Wireless-Fidelity (registered trademark)) transceiver unit, an infrared transceiver unit, etc.

[0027] The wired transceiver 18A and the wireless transceiver 18B form, for example, the communication unit 21 shown in FIG.

[0028] The processing unit 10 may be connectable to a recording medium 16. The recording medium 16 stores a predetermined program. The program stored in the recording medium 16 is installed in the auxiliary storage device 14 of the processing unit 10 via the drive device 15. The installed predetermined program can be executed by the CPU 11 of the processing unit 10. For example, the recording medium 16 may be a recording medium that records information optically, electrically, or magnetically, such as a CD (Compact Disc)-ROM, a flexible disk, or a magneto-optical disk, or a semiconductor memory that records information electrically, such as a ROM or a flash memory. The recording medium 16 does not include a carrier wave.

[0029] 3 is a block diagram for explaining the functions of the processing unit 10 of the information processing device 20 of this embodiment. In the following, a user is assumed to be a person who provides various operational inputs to the input unit 24 of the information processing device 20.

[0030] 3, the information processing device 20 includes an image acquisition processing unit 210, an input screen output processing unit 212, an input acquisition processing unit 214, an image recognition processing unit 216, a recognition result output unit 218, and a setting processing unit 220. Each unit from the image acquisition processing unit 210 to the setting processing unit 220 can be realized by the CPU 11 shown in FIG. 2 executing one or more programs stored in a storage device (for example, the ROM 13).

[0031] The image acquisition processing unit 210 acquires an image including the picking target. As described above, the picking target can be any object. In this embodiment, the picking target is a specific type of tool stored in a tray (case), and the image including the picking target is also referred to as a "tray image G10." Note that the picking target is not limited to objects stored in a container such as a tray, but may be objects placed directly on a conveyor or the like, or may be stored in other containers such as bags.

[0032] The user may specify the tray image G10 to be acquired by the image acquisition processing unit 210. The tray image G10 acquired by the image acquisition processing unit 210 may be an image stored in an external database or the like, or may be an image stored in the recording medium 16 described above.

[0033] The input screen output processing unit 212 outputs an input screen for performing various settings and the like.

[0034] 4 to 7 are diagrams showing examples (transition states) of input screens output by the input screen output processing unit 212 when the user selects the teaching mode. Fig. 4 shows one state of the input screen G400, and Fig. 5 shows another state of the input screen G400 formed by user input, etc. Fig. 6 shows yet another state of the input screen G400 formed by user input, etc. Fig. 7 shows yet another state of the input screen G400 formed by user input, etc.

[0035] The input screen G400 forms a GUI (Graphical User Interface). The input screen G400 (FIG. 4, etc.), G1000 (FIG. 10, etc., described later), etc. may be switchable by the user selecting (switching input) tabs Tb1 and Tb2. When tab Tb1 is selected, settings related to the picking execution mode are enabled, and when tab Tb2 is selected, settings related to the teaching mode are enabled. Information that the user can input via the input screen G400, etc., will be described later.

[0036] The input screen G400 has an image display area G310 that displays a tray image G10 acquired by the image acquisition processing unit 210. In this embodiment, as an example, the tray image G10 is an image of a tray in a plan view (top view) in which the items to be picked are stored, as described above. Note that in a modified example, the tray image G10 may be another view (for example, an oblique view), or a combination of two or more views may be used.

[0037] The input screen G400 becomes available for input when the tab Tb2 is selected. The input screen G400 includes a user interface for making various settings in the teaching mode.

[0038] The input screen G400 includes buttons BT1 to BT6 for the user to input various commands. The buttons BT1 to BT6 are buttons for causing the information processing device 20 to perform image recognition and extract (cut out) the object to be picked from the tray image G10.

[0039] The button BT1 is for "add target object point" and is a button that allows the user to associate a target object point P400 (an example of a point with the first attribute) with the image display area G310 or the tray image G10. Three target object points P400 are illustrated in FIG. 4. In this case, the user can associate the target object point P400 in FIG. 4 with the image display area G310 or the tray image G10 by specifying one or more desired positions while the button BT1 is active. In a preferred method of specification, the user specifies the target object point P400 so that the target object point P400 is located within the image portion of the desired picking target.

[0040] The button BT2 is for "add exclusion point" and is a button that allows the user to associate an exclusion point P402 (an example of a point with a second attribute) with the image display area G310 or the tray image G10. Three exclusion points P402 are illustrated in FIG. 4. In this case, the user can associate the exclusion point P402 in FIG. 4 with the image display area G310 or the tray image G10 by specifying one or more desired positions while the button BT2 is active. In a preferred method of specification, the user specifies the exclusion point P402 so that the exclusion point P402 is located around the image portion of the desired picking target.

[0041] The button BT3 is used for issuing a "Cut Out" command, and is a button for extracting (cutting out) an image portion bounded by an outline display G700 (described later). Hereinafter, the button BT3 is also referred to as the cut out button BT3.

[0042] The button BT4 is for "remove all points" and is a button for clearing all points (target object points P400 and excluded points P402) associated with the image display area G310 or the tray image G10. For example, the user can redo all associations by operating the button BT4 only once.

[0043] The button BT5 is used to "go back one step" and clear the most recently associated point (the target object point P400 or the excluded point P402). For example, the user can operate the button BT5 one or more times to redo the previous association.

[0044] The button BT6 is used to "go forward one step" and to cancel the operation of the button BT5. For example, if the user mistakenly operates the button BT5 one or more times, the user can restore the desired association state by operating the button BT6 the same number of times.

[0045] Note that one or more of the operations related to buttons BT1 to BT6 may be operable using shortcut keys.

[0046] The input screen G400 further includes an image loading button BT440A, an image capturing button BT440B, a work type display area G430, a cut-out work output area G450, a similar product extraction (search) button BT460, a picking location estimation button BT470, a settings loading (indicated as "Load settings" in the figure) button BT475, and a settings saving (indicated as "Save settings" in the figure) button BT480.

[0047] The image read button BT440A is a button for reading the tray image G10, and when operated, it may be possible to specify an image file (the user may specify a desired image file). The read tray image G10 is output to the image display area G310.

[0048] The image capture button BT440B is typically used when there is no image to load or when teaching is performed using a new image. For example, the image capture button BT440B may be a button for capturing and loading an image on the spot using the camera 50, and when operated, the image captured by the camera 50 is loaded as a tray image G10. The loaded tray image G10 is output to the image display area G310.

[0049] The work type display area G430 is an area that displays the type of work related to the picking target. The type of work related to the picking target may be specified by the user, or may be selectable from a pull-down list. In the latter case, the pull-down list may be automatically generated based on the image recognition results. In this embodiment, as an example, the items that can be selected from the pull-down are already registered work types and items for adding new work. When an item for adding new work is selected, a new name is entered and registered. The "Delete Item" button can also be selected, and the item can be deleted.

[0050] The cut-out work output area G450 is an area where a cut-out image, which will be described later, is output. By visually checking the image in the cut-out work output area G450, the user can confirm whether the desired image portion has been cut out as the cut-out image.

[0051] The Extract Similar Item button BT460 is a button for extracting from the tray image G10 an object that is the same as or similar to the target object associated with the cropped image. For example, the tray image G10 may contain multiple (e.g., numerous) image portions of the same type of tool. In this case, the user can efficiently crop multiple desired image portions by operating the Extract Similar Item button BT460 without repeatedly operating the buttons BT1 and BT2. Figure 5 shows the state after the Extract Similar Item button BT460 is operated in the state shown in Figure 4. In this case, an image portion (cropped image) of an object similar to the target object associated with the outline display G700 in the state shown in Figure 4 is extracted in the cropped work output area G450. In this case, the image portion is preferably output as is, with its orientation, etc., maintained in the tray image G10 (see area Q50). Maintaining such orientation is advantageous when a specific orientation of an object is desired to be set as a picking target.

[0052] In the example shown in Figure 5, an edit / delete button BT500 (see area Q51) appears, and the user can edit or delete the image portion (cut-out image) in the cut-out work output area G450 by operating the edit / delete button BT500.

[0053] The degree of similarity used to extract similar objects when the similar item extraction button BT460 is operated may be changeable (customizable) by the user. For example, the degree of similarity may be set appropriately to be completely identical, identical in type, etc. Furthermore, factors other than shape, such as color and posture, may also be taken into consideration.

[0054] The picking location estimation button BT470 is a button for performing (trying to perform) a picking location estimation process for the current tray image G10 or a newly loaded tray image G10. The picking location estimation process may include a process of recognizing a picking target and estimating the picking target as the picking location.

[0055] The save setting button BT480 is a button for saving the current setting state. When the save setting button BT480 is operated in the state shown in Figure 4 or 5, the target object related to the image portion extracted in the cut-out work output area G450 at that time is set as the picking target.

[0056] In this embodiment, in addition to setting (registering) picking targets, objects that should not be picked (or objects that should not be touched by the robot arm 42) may also be set (registered). Hereinafter, objects that should not be picked or objects that should not be touched by the robot arm 42 are collectively referred to as "non-touch objects." In this embodiment, as shown in FIG. 6 , non-touch objects can be set by selecting the "Don't Touch" box in the work type display area G430 (see Q61 in FIG. 6 ). Note that the notation is arbitrary. The method for specifying non-touch objects may be the same as the method for specifying picking targets described above. In this case, the user can specify non-touch objects by appropriately operating buttons BT1 to BT6. FIG. 6 illustrates a state in which a target object point P400 is associated with an image portion related to the tray in the image display area G310, and an exclusion point P402 is associated with its surroundings (see Q60 in FIG. 6 ). In this state, the user can specify the tray as a non-contact object by operating the cut-out button BT3 for the "cut-out" command.

[0057] Note that image portions of objects that may be picking targets or non-contact objects may be automatically extracted. For example, in the input screen G400A shown in FIG. 7 , when a tray image G10 is loaded, image portions G720 of each object that may be picking targets or non-contact objects are extracted and output to the unregistered display area G452. The user may be able to edit or delete the image portions, as described above with reference to FIG. 5 . For example, by dragging and dropping a desired image portion of each image portion output in the unregistered display area G452 to the picking target area G454, the object associated with the image portion can be set as the picking target. Furthermore, by dragging and dropping a desired image portion of each image portion output in the unregistered display area G452 to the non-contact object area G456, the object associated with the image portion can be set as the non-contact object. In this case, a new item may be added by operating the new item addition BT458. The new item is a new item among the items that represent the content of the process (in this embodiment, the content of the process related to the robot arm 42), such as the picking target. In addition, it may be possible to delete the item by clicking a cross button BT459 near the item name.

[0058] The input acquisition processing unit 214 acquires various inputs via the above-described input screen G400. Note that the input information to be acquired is not limited to information directly input by the user, but may also include automatically input information. Automatically input information may be definitively acquired when the user approves it (for example, by operating the save settings button BT480).

[0059] The image recognition processing unit 216 detects the contour (segment) of an object by performing image recognition processing on the tray image G10 loaded into the image display area G310. Any algorithm may be used to detect the contour of an object, and for example, the Segment Anything Model (SAM) developed by Meta, Inc. or similar may be used.

[0060] The image recognition processing unit 216 may execute image recognition processing when the tray image G10 is read in. For example, each time a target object point P400 or a non-contact object is added, the image recognition processing unit 216 may update the image recognition processing result in a manner that reflects the addition.

[0061] In addition, in response to a user's operation of the similar object extraction button BT460, the image recognition processing unit 216 recognizes (searches for) an object similar to the object whose contour has been detected (hereinafter also referred to as a "similar object").

[0062] In addition, in response to the user's operation of the cut-out button BT3, the image recognition processing unit 216 cuts out (extracts) the object whose contour is detected at that time (the image portion surrounded by the contour display G700 described below).

[0063] The recognition result output unit 218 outputs the image recognition results from the image recognition processing unit 216. Specifically, the recognition result output unit 218 generates and outputs an outline display G700 (an example of a predetermined display). The outline display G700 is, for example, a display that highlights the outline (boundary) detected (recognized) by the image recognition processing unit 216. The outline display G700 represents the image recognition result of the outline (boundary) of an object recognized by the image recognition processing unit 216 based on the target object point P400 and / or the non-contact object. As described above, the image portion surrounded by the outline display G700 can be set (registered) as a picking target and / or a non-contact object. The outline display G700 is preferably superimposed on the tray image G10, but may also be displayed elsewhere (e.g., in the cut-out work output area G450) along with the image portion within the outline display G700. In this embodiment, the contour display G700 is displayed superimposed on the tray image G10 in accordance with the detection result of the contour of the corresponding target object.

[0064] The recognition result output unit 218 may update the contour display G700 in real time when the image recognition result by the image recognition processing unit 216 is updated. For example, each time a target object point P400 or a non-contact object is added, the recognition result output unit 218 may update the contour display G700 in a manner that reflects the update of the image recognition result by the image recognition processing unit 216 resulting from the addition. This allows the user to stop operating button BT1 or button BT2 and proceed to operating the cut-out button BT3 when the desired contour display G700 is obtained.

[0065] Instead of or in addition to the outline display G700, the recognition result output unit 218 may output a display that highlights (emphasizes) the image portion that should be surrounded by the outline display G700.

[0066] Furthermore, in response to a user's operation of the similar object extraction button BT460, if the image recognition processing unit 216 is able to recognize a similar object, the recognition result output unit 218 generates a contour display G700 of the similar object based on the image recognition result. Note that if multiple similar objects are recognized, the recognition result output unit 218 may output multiple contour displays G700.

[0067] Furthermore, when the image recognition processing unit 216 cuts out an image portion in response to the user's operation of the cut-out button BT3, the recognition result output unit 218 may output the cut-out image portion to the cut-out work output area G450. At this time, the tray image G10 may be displayed with the cut-out image portion missing, or may remain as is.

[0068] The setting processing unit 220 generates setting information for various settings based on the input information obtained via the input screen G400 up to that point, in response to the user's operation of the Save Settings button BT480. At this time, the setting information includes setting information for objects related to the contour display G700. For example, from the input screen G400 shown in FIG. 4, an object related to an image portion that will be output to the cut-out work output area G450 by a subsequent operation of the Cut-out button BT3 is set as a picking target. Also, from the input screen G400 shown in FIG. 6, an object related to an image portion that will be output to the cut-out work output area G450 by a subsequent operation of the Cut-out button BT3 is set as a picking target.

[0069] In this way, according to this embodiment, various settings based on the tray image G10 can be efficiently performed via the user interface of the input screen G400.

[0070] For example, picking and defect inspection require annotation to find the detection location, reading CAD (Computer Aided Design) data (e.g., 3D data), and setting rules and procedures. These various processes and tasks are burdensome for users and difficult to perform efficiently and accurately.

[0071] In this regard, according to this embodiment, the user can efficiently perform these processes and tasks via the teaching mode input screen G400, significantly reducing the burden on the user in picking, defect inspection, etc.

[0072] Next, the operation of the system 1 of this embodiment will be further described with reference to FIG. 8 and subsequent figures.

[0073] 8 is a schematic flowchart for explaining the flow of processing executed when the teaching mode input screen G400 is selected. In the following processing flow diagrams (flowcharts), the processing order of each step may be changed as long as the relationship between the input and output of each step is not impaired.

[0074] When the user operates the setting read button BT475 ("Y" in step S800), the corresponding setting file is loaded (step S801). In this case, the user can edit the setting file. Note that when creating a new setting file, loading the setting file using the setting read button BT475 is not necessary. When the user operates the image capture button BT440B ("Y" in step S802), the image captured by the camera 50 is acquired as the tray image G10 (steps S803 and S804), as described above. Next, the selection status of the work type display area G430 is updated in accordance with the user's selection (step S806), and image recognition processing and output processing of the outline display G700 are executed (step S808). When the user operates the cut-out button BT3, the image portion within the outline display G700 at that time is cut out and displayed in the cut-out work output area G450 (step S810). If there are further cut-outs, the processing from step S806 is repeated. Next, if the user operates the similar item extraction button BT460 ("Y" in step S812), the image portion relating to the similar object is extracted as described above and displayed in the extraction work output area G450 (step S814). If the user operates the picking location estimation button BT470 ("Y" in step S816), the above-described picking location estimation process is executed for the current tray image G10 (step S818). If the user operates the setting save button BT480 ("Y" in step S820), setting information is generated (saved) based on the input information at that time (step S822). The setting information may be saved as a setting file, associated with a desired file name specified by the user. In this case, the user can easily load the desired file in the picking execution mode described below.

[0075] 9 is a schematic flowchart for explaining the flow of processing executed when the picking execution mode input screen G1000 (see FIG. 10, etc.) is selected. FIGS. 10 and 11B are explanatory diagrams of FIG. 9 and show examples of the picking execution mode input screen G1000.

[0076] A selection button B1010 (automatic picking or setting read button) is operated from the input screen G1000 as shown in FIG. 10 (step S900). At this time, if the setting read button is checked, the setting information set in the teaching mode described above is used. When picking is started (step S902), the picking target (and the picking position accordingly) is estimated based on the image from the camera 50 and the setting information during picking (step S904), and it is determined whether or not there is a picking target (a pickable point) (step S906). If there is no pickable point ("N" in step S906), processing for this tray is terminated. If there is a picking target (a pickable point) ("Y" in step S906), it is determined whether or not the machine can determine whether picking is necessary (step S908). If the machine is capable of making the decision ("Y" in step S908), picking is performed by the robot arm 42 (step S912). If the machine is unable to make the decision ("N" in step S908), the system waits for a user instruction regarding whether picking should be performed via an input screen G1000 such as that shown in FIG. 11B. Specifically, if the machine is unable to make the decision ("N" in step S908), an interface (see Q1000 in FIG. 11B) is output and the system waits for a user selection (step S910). If a user input indicating that picking should be performed is received ("Y" in step S910), picking is performed by the robot arm 42 (step S912). If no input indicating that picking should be performed is received ("N" in step S910), the system repeats the process from step S904 (i.e., automatic picking continues).

[0077] In this embodiment, after the picking operation mode is started, an input screen G1000 having a stop button BT1100 as shown in FIG. 11A may be output. In this case, the user can stop automatic picking by operating the stop button BT1100. That is, when the stop button BT1100 is operated ("Y" in step S903), automatic picking is stopped.

[0078] In this embodiment, when the picking target (and the picking position) is estimated, the outline of the estimated picking target may be highlighted as shown by reference symbol G111 in Fig. 11B. This allows the user to easily determine whether to stop automatic picking (i.e., whether to operate the stop button BT1100) and make a decision (selection) in a situation where the machine cannot make a decision.

[0079] FIG. 12 is a timing chart showing an example of information exchange between the components (information processing device 20, camera 50, and robot arm controller 40) during picking.

[0080] The camera 50 acquires an RGB image (camera image) and a depth image of the tray containing the object to be picked (S1200), and transmits these images to the information processing device 20 (step S1202). The image transmission timing may be, for example, the timing when the tray on the conveyor is positioned upstream of the robot arm 42.

[0081] When the information processing device 20 receives the RGB image and the depth image from the camera 50 (step S1204), it identifies the object to be picked based on these images and calculates the picking position (step S1206).The information processing device 20 then transmits the calculated picking position to the robot arm controller 40 (step S1208).

[0082] When the robot arm controller 40 receives the picking position (step S1210), it performs the picking operation, and when the picking is completed, it generates picking completion information indicating that fact (step S1212) and sends it to the information processing device 20 (step S1214).

[0083] Then, the information processing device 20 receives the picking completion information (step S1216).

[0084] Although each embodiment has been described in detail above, it is not limited to the specific embodiment, and various modifications and changes are possible within the scope of the claims. It is also possible to combine all or a plurality of components of the above-described embodiments.

[0085] For example, in the above-described embodiment, the processing method related to the setting support function is picking, but this is not limited to this. For example, the setting support function can be applied to any processing setting such as quality inspection processing based on image recognition, cleaning processing, various cooking processing, etc.

[0086] In the above-described embodiment, the setting support function has only one processing method, but it may include a combination of two or more. For example, in the above-described embodiment, the only processing method that can be set is picking, but it may also be possible to set the placement location after picking (the location to which the picking target is moved), etc.

[0087] In the above-described embodiment, the display includes tabs Tb1 and Tb2 that allow selection of either the teaching mode or the picking execution mode, but this is not limited to this. Dedicated software may be provided for each of the teaching mode and the picking execution mode. In this case, as shown in FIG. 13, the input screen G400A can have a relatively large image display area G310 that does not require the tabs Tb1 and Tb2. Note that while FIG. 13 shows the input screen G400A for the teaching mode, the same applies to the input screen (not shown) for the picking execution mode.

[0088] REFERENCE SIGNS LIST 1 System 10 Processing section 14 Auxiliary storage device 15 Drive device 16 Recording medium 17 Communication interface 18A Wired transmission / reception section 18B Wireless transmission / reception section 19 Bus 20 Information processing device 21 Communication section 22 Storage section 23 Display section 24 Input section 25 Control section 40 Robot arm controller 42 Robot arm 50 Camera 210 Image acquisition processing section 212 Input screen output processing section 214 Input acquisition processing section 216 Image recognition processing section 218 Recognition result output section 220 Setting processing section

Claims

1. A program that causes a computer to execute the following steps: a first output process that outputs an input screen including an image in a specified area; an input acquisition process that acquires user input from a user; an image process that performs image recognition processing on the image based on the user input including a specified input of a position or area for the specified area or the image, thereby recognizing an object having a boundary corresponding to the position or area related to the specified input; a second output process that generates a specified display representing the image recognition result of the image processing and outputs the specified display in correspondence with the object; and a setting process that generates setting information for the object related to the specified display based on the user input including a setting instruction from the user.

2. The program according to claim 1, wherein the specified display includes at least one of a display that highlights the contour of the object and a display that highlights the inside of the contour of the object, and is superimposed on the image in accordance with the position of the object in the image.

3. The program according to claim 2, wherein the image processing includes a process of extracting or cutting out an image portion relating to the specified display from the image.

4. The program according to claim 2, wherein the input screen has an input area for inputting specific information, and the setting process includes a process of associating the specific information with the target based on the input information in the input area.

5. The program according to claim 4, wherein the specific information includes at least one of information representing a processing method for the object, information representing attributes of the object, and information representing a name of the object.

6. The program according to claim 5, wherein the information representing the processing method includes information representing whether or not to execute a specified process on an object that is image-recognized as being the same as or similar to the target.

7. The program according to claim 6, wherein the predetermined process includes a process of picking up a specific type of object from among a plurality of types of objects using a robot arm.

8. The program according to claim 2, wherein the designation input includes an entire image designation input that designates the entire image, and the second output process causes the specified display to be superimposed on each of the targets based on the entire designation input.

9. The program of claim 8, wherein the input screen has an input area for inputting specific information, the setting process includes a process for associating the specific information with the targets based on the input information in the input area, and the program further causes the computer to execute a third output process for cutting out and outputting image portions of the image relating to each of the targets within the input area.

10. A program according to any one of claims 2 to 9, wherein the specified input includes an input of a first attribute that specifies one or more points or areas of a first attribute, and the second output process superimposes the specified display so that the contour includes the points or areas of the first attribute based on the input of the first attribute.

11. The program described in claim 10, wherein the specified input includes an input of a second attribute that specifies one or more points or areas of a second attribute, and the second output process superimposes the specified display based on the input of the second attribute so that the contour does not include any points or areas of the second attribute.

12. The program of claim 1, wherein the first output process updates the image from one image to another image based on the user input including a change instruction from the user, and the setting process generates a setting file related to the setting information based on a result of the image recognition process on one or more of the images.

13. An information processing device comprising: a first output processing unit that outputs an input screen including an image in a predetermined area; an input acquisition processing unit that acquires user input from a user; an image processing unit that performs image recognition processing on the image based on the user input including a specified input of a position or area for the predetermined area or the image, thereby performing image recognition on the image to recognize an object having a boundary corresponding to the position or area related to the specified input; a second output processing unit that generates a predetermined display representing the image recognition result of the image processing unit and outputs the predetermined display in correspondence with the object; and a setting processing unit that generates setting information regarding the object related to the predetermined display based on the user input including a setting instruction from a user.

14. An information processing method executed by a computer, comprising: a first output processing step of outputting an input screen including an image in a predetermined area; an input acquisition processing step of acquiring user input from a user; an image processing step of performing image recognition processing on the image based on the user input including a specified input of a position or area for the predetermined area or the image, thereby image recognizing an object having a boundary corresponding to the position or area related to the specified input; a second output processing step of generating a predetermined display representing the image recognition result of the image processing step and outputting the predetermined display in correspondence with the object; and a setting processing step of generating setting information for the object related to the predetermined display based on the user input including a setting instruction from a user.

Images