Estimate method of reliability degree about image recognition, and information processing device

Abstract

To provide a technology which can grasp a reliability degree of recognition of a hand about an instruction image, before operating a robot.SOLUTION: An estimate method which estimates a reliability degree about image recognition performed on the basis of a plurality of frame images includes: an object recognition step which executes processing which recognizes a recognition object including at least a part of an informer's hand, about a plurality of frame images obtained by capturing the informer's work with respect to a workpiece; and an acquisition step which finds the reliability degree about the image recognition, by using a number of the frame images in which the recognition object has not been recognized, or the number of the frame images in which the recognition object has been recognized.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a method for estimating reliability regarding image recognition, and an information processing device. [Background technology]

[0002] In the technology described in Patent Document 1, a robot is taught using a teaching image obtained by capturing the hands of an instructor performing a task with a camera. In this technology, coordinates representing the positions of the instructor's hands are identified from the teaching image, and teaching data for the robot is created based on the identified coordinates. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-110621 A Summary of the Invention [Problem to be solved by the invention]

[0004] If the teaching image includes subject blur, the teacher's hand is not recognized, and for example, defective teaching data is created. In this case, the robot cannot accurately imitate the teaching operation of the teacher. In the technology described in Patent Document 1, it is not possible to determine whether the teacher's hand has been correctly recognized based on the teaching image unless the robot is operated. For this reason, there has been a demand for a technology that can grasp the reliability of hand recognition in the teaching image before operating the robot. [Means for solving the problem]

[0005] According to one embodiment of the present disclosure, there is provided a method for estimating reliability of image recognition, which is a method for estimating reliability of image recognition performed based on a plurality of frame images, and includes a recognition step of executing a process for recognizing a recognition target including at least a part of the instructor's hand for a plurality of frame images capturing an operator's work on a workpiece, and an acquisition step of calculating the reliability of the image recognition using the number of frame images in which the recognition target could not be recognized or the number of frame images in which the recognition target could be recognized.

[0006] According to another aspect of the present disclosure, there is provided an information processing device including: an object recognition unit that recognizes a recognition object including at least a part of a hand of an instructor in a plurality of frame images capturing an operator's work on a workpiece; and an acquisition unit that calculates a reliability of image recognition using the number of frame images in which the recognition object is not recognized or the number of frame images in which the recognition object is recognized. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram showing an overall configuration of a robot system. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the information processing device. [Diagram 3] 13 is a flowchart of a process related to teaching. [Figure 4] 4 is a flowchart showing the first half of detailed processing of step S20 in FIG. 3. [Diagram 5] 4 is a flowchart showing the latter half of the detailed process of step S20 in FIG. 3. [Figure 6] FIG. 13 is a diagram illustrating an example of a display screen for presenting a reliability. [Figure 7] FIG. 13 is a diagram illustrating an example of a display screen according to another embodiment 7. [Figure 8] 23 is an example of a display screen for presenting the reliability according to another embodiment 10. [Figure 9]23 is an example of a display screen for presenting the reliability according to another embodiment 11. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A. Embodiment: 1 is a schematic diagram showing the overall configuration of a robot system 1 according to this embodiment. The robot system 1 includes a robot 100, a camera 200, and an information processing device 300. In the robot system 1, the robot 100 having a plurality of joints is operated so as to imitate human work. In order to operate the robot 100, it is necessary to teach the robot 100 the work to be performed on a workpiece. Teaching means uniquely determining the angles of all the joints of the robot 100 when performing work on a workpiece.

[0009] The robot 100 is a multi-axis robot having multiple joints. As an end effector of the robot 100, an end effector of any configuration capable of holding a workpiece can be used.

[0010] The camera 200 captures an image of the instructor TP performing a predetermined task in the first working area WA1 in order to teach the robot 100 under the control of the information processing device 300. In this embodiment, the predetermined task is pick-and-place to move a workpiece. The camera 200 is disposed in a position where it can capture an image of the instructor TP's hand. The camera 200 captures an image of a predetermined imaging range at a predetermined frame rate. The frame images captured by the camera 200 are still images that can be displayed in a progressive (non-interlaced) format. The camera 200 outputs the captured frame images to the information processing device 300.

[0011] The first working area WA1 is an area set as an area where the instructor TP performs work. The first working area WA1 is a three-dimensional space on the worktable TB1. For example, the space formed by the upper surface of the worktable TB1 and the height direction is set as the first working area WA1. The upper surface of the worktable TB1 is assumed to be a surface along the horizontal direction, and the height direction is assumed to be a direction perpendicular to the upper surface of the worktable TB1. The distance in the height direction of the first working area WA1 is assumed to be a predetermined distance.

[0012] A first supply area SA1 and a first target area TA1 are provided on the upper surface of the worktable TB1. The first supply area SA1 is an area where the first workpiece WK1 is placed at the start of the teaching work. The first target area TA1 is an area where the first workpiece WK1 is moved from the first supply area SA1 by the instructor TP.

[0013] FIG. 1 illustrates a robot coordinate system Σr set for the robot 100 and a first camera coordinate system Σc1 set for the camera 200. Both the robot coordinate system Σr and the first camera coordinate system Σc1 are orthogonal coordinate systems defined by three axes X, Y, and Z. The correspondence between the robot coordinate system Σr and the first camera coordinate system Σc1 is determined by calibration. In the embodiment, the first camera coordinate system Σc1 is used as a reference coordinate system. In addition, the position and posture of the instructor TP are expressed in the reference coordinate system.

[0014] 2 is a block diagram showing a functional configuration of the information processing device 300. The information processing device 300 is, for example, a personal computer. The information processing device 300 includes a memory 310, an input / output interface 320, an input device 330, a display device 340, and a processor 360.

[0015] The information processing device 300 performs image recognition processing for recognizing recognition targets such as the arms and hands of the worker on frame images capturing the instructor TP performing work. The information processing device 300 also creates a control program for controlling the robot 100 based on the result of the image recognition. The control program is, for example, a program for controlling a pick-and-place operation for moving a workpiece to the robot 100.

[0016] The memory 310 stores programs and data used for various processes executed by the information processing device 300 .

[0017] An input device 330, a display device 340, and the camera 200 are connected to the input / output interface 320. The input device 330 is, for example, a keyboard or a mouse. The display device 340 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

[0018] Processor 360 realizes various functions by executing the programs stored in memory 310. Processor 360 executes the programs stored in memory 310 to function as an image receiving unit 361, an object recognition unit 362, an action recognition unit 363, a reliability estimation unit 365, and a control program creation unit 366.

[0019] The image receiving unit 361 receives a frame image from the camera 200 .

[0020] The object recognition unit 362 executes image recognition using the frame image accepted by the image acceptance unit 361. Image recognition includes object recognition. Object recognition refers to identifying the presence of a recognition target in the frame image. Object recognition includes at least one of object detection and region division. Object detection refers to identifying the position of the recognition target and the type of the recognition target in the frame image. Specifically, the object recognition unit 362 executes a process of detecting a recognition target for the frame image accepted by the image acceptance unit 361. In this embodiment, the recognition target is the upper limbs and the first workpiece WK1 of the instructor TP. The upper limbs include the instructor TP's shoulders to fingertips. The hand includes fingers. For example, the object recognition unit 362 executes a process of detecting the upper limbs and the first workpiece WK1 of the instructor TP using a machine learning model for object detection. The object recognition unit 362 is also called an "object recognition unit."

[0021] The action recognition unit 363 executes a process of recognizing the action of the instructor TP using the frame image received by the image reception unit 361. At this time, the action recognition unit 363 may use the detection result by the object recognition unit 362. The action recognition unit 363 executes a process of recognizing the action of the instructor TP using, for example, a machine learning model prepared in advance. The action of the instructor TP to be recognized is, for example, an action of gripping the first workpiece WK1 and an action of releasing the gripped first workpiece WK1.

[0022] The reliability estimation unit 365 obtains the reliability of image recognition by using the number of frame images in which the object recognition unit 362 failed to recognize the recognition target, or the number of frame images in which the object recognition unit 362 recognized the recognition target. The reliability estimation unit 365 is also called the "acquisition unit."

[0023] The control program creation unit 366 creates a control program for the robot 100. Specifically, the control program creation unit 366 executes a process of detecting a plurality of joints included in the upper limbs of the instructor TP, which is a recognition target, in the frame image accepted by the image acceptance unit 361. At this time, the control program creation unit 366 may use the detection result by the object recognition unit 362. For example, the control program creation unit 366 uses a machine learning model for extracting joint points to obtain the positions and joint angles of a plurality of joints included in the upper limbs of the instructor TP. The control program creation unit 366 creates a control program for the robot 100 using the detection results for a plurality of joints included in the upper limbs of the instructor TP and the recognition results by the action recognition unit 363.

[0024] 3 is a flowchart of a process related to teaching. In step S10, the camera 200 captures an image of the work of the instructor TP to obtain a frame image for teaching. The image receiving unit 361 receives a plurality of frame images from the camera 200. This step is also called an "image receiving step."

[0025] In step S20, the object recognition unit 362 executes image recognition based on a plurality of frame images, and executes a process of estimating the reliability of the image recognition. The detailed process of step S20 will be described later.

[0026] In step S40, the image receiving unit 361 determines whether or not the instructor TP has instructed to perform re-teaching. If the instructor TP has instructed to perform re-teaching (step S40; YES), the processing from step S10 onwards is executed again. For example, the image receiving unit 361 causes the display device 340 to display a notice to start re-teaching, and causes the camera 200 to start capturing an image. When step S10 is executed again, the camera 200 captures an image of the work of the instructor TP again, thereby acquiring a frame image for teaching. The image receiving unit 361 receives the frame image from the camera 200. Thereafter, the processing of steps S20 and S40 is executed.

[0027] In step S40, if the instructor TP has not instructed to re-teach (step S40; NO), the process shown in Fig. 3 is terminated. Note that, in this embodiment, the instructor TP checks the reliability of the image recognition and issues an instruction as to whether or not to re-teach, but a person other than the instructor TP may check the reliability of the image recognition and issue an instruction as to whether or not to re-teach.

[0028] 4 and 5 are flowcharts showing the detailed process of step S20. As shown in Fig. 4, in step S21, the reliability estimation unit 365 resets the undetected counter Cnt1 by setting the value of the undetected counter Cnt1 to "0". The undetected counter Cnt1 will be described later.

[0029] In step S22, the object recognition unit 362 executes image recognition. Specifically, the object recognition unit 362 executes a process of detecting a recognition target for one of the multiple frame images to be processed, using a machine learning model for object detection. The multiple frame images to be processed are acquired by the camera 200 (see step S10 in FIG. 3). The recognition target is the upper limbs of the instructor TP and the first workpiece WK1. This step is also called an "object recognition step."

[0030] In step S23, the reliability estimation unit 365 determines whether or not a recognition target was detected in the target frame image in step S22. In this embodiment, the reliability estimation unit 365 determines that a recognition target was detected in the target frame image when at least a part of a hand was detected in the target frame image. A frame image in which at least a part of a hand was recognized is treated as a frame image in which the recognition target was recognized, thereby appropriately determining a reliability Lc regarding image recognition, which will be described later. If a recognition target was detected (step S23; YES), the reliability estimation unit 365 executes the process of step S25. On the other hand, if a recognition target was not detected (step S23; NO), the reliability estimation unit 365 executes the process of step S24.

[0031] In step S24, the reliability estimation unit 365 adds "1" to the undetected counter Cnt1. The undetected counter Cnt1 is used to count consecutive frame images in which the recognition target is not detected. After that, the reliability estimation unit 365 executes the process of step S28.

[0032] In step S25, the reliability estimation unit 365 determines whether the value of the undetected counter Cnt1 is within a predetermined range. The predetermined range is equal to or greater than a lower limit value Vl1 and less than an upper limit value Vl2. The lower limit value Vl1 is an integer equal to or greater than 1. The upper limit value Vl2 is an integer greater than the lower limit value Vl1. For example, the lower limit value Vl1 is set to "3" and the upper limit value Vl2 is set to "15."

[0033] When the number of consecutive frame images in which the recognition target is not recognized is less than the lower limit Vl1, this corresponds to a case where consecutive failure to recognize the recognition target is acceptable. When consecutive failure to recognize the recognition target is acceptable, the corresponding frame image is not treated as a frame image in which the recognition target was not recognized when calculating the reliability Lc. This makes it possible to obtain a more useful value as the reliability when using the reliability Lc to determine whether or not to use the image recognition results in other processes such as creating a control program.

[0034] When the number of consecutive frame images in which the recognition target could not be recognized is equal to or greater than the upper limit value Vl2, it corresponds to, for example, a case where the worker's hand is outside the imaging range immediately after the start of shooting, or a case where the worker's hand is inevitably outside the imaging range for some reason. In such a case, it is not desirable to use the number of frame images in which the recognition target could not be recognized as the calculation of the reliability Lc of the image recognition. In the embodiment, when there is a valid reason why the recognition target could not be recognized consecutively, the information processing device 300 does not treat the corresponding frame image as a frame image in which the recognition target could not be recognized when calculating the reliability of the image recognition. When the reliability Lc is used to determine whether or not to use the result of image recognition in other processes such as creating a control program, a more useful value can be obtained as the reliability.

[0035] In step S25, if the value of the undetected counter Cnt1 is within the predetermined range (step S25; YES), the reliability estimation unit 365 executes the process of step S26. On the other hand, if the value of the undetected counter Cnt1 is not within the predetermined range (step S25; NO), the reliability estimation unit 365 executes the process of step S27. The value of the undetected counter Cnt1 is not within the predetermined range when the value of the undetected counter Cnt1 is less than the lower limit Vl1 or is equal to or greater than the upper limit Vl2.

[0036] In step S26, the reliability estimation unit 365 adds the value of the undetected counter Cnt1 to the first list L1. The first list L1 is made up of a plurality of records R. The value of the undetected counter Cnt1 added to the first list L1 is stored in the record R. After that, the reliability estimation unit 365 executes the process of step S27.

[0037] In step S27, the reliability estimation unit 365 resets the undetected counter Cnt1 by setting the value of the undetected counter Cnt1 to the value of “0.” After that, the reliability estimation unit 365 executes the process of step S .

[0038] In step S28, the reliability estimation unit 365 determines whether or not the process of detecting the recognition target has been performed for all of the target frame images. If the process of detecting the recognition target has been performed for all of the target frame images (step S28; YES), the reliability estimation unit 365 executes the process of step S29. On the other hand, if the process of detecting the recognition target has not been performed for all of the target frame images (step S28; NO), the reliability estimation unit 365 executes the process of step S22 and subsequent steps again. When executing step S22 again, the object recognition unit 362 executes the process of detecting the recognition target for the frame image next to the frame image that was the target of the process immediately before detecting the recognition target. The next frame image refers to an image in which there is no other frame image between the frame image that was the target of the process immediately before detecting the recognition target and the frame image in question. The process of step S22 and subsequent steps that is executed again is also called a "reexecution step."

[0039] In step S29, the reliability Lc of the image recognition is calculated. Specifically, the reliability estimation unit 365 first calculates the total value of each record R included in the first list L1 as the total number NF of frame images in which the recognition target was not detected. The value of each record R included in the first list L1 represents the number of consecutive frame images in which the recognition target was not detected. The reliability estimation unit 365 calculates the reliability Lc of the image recognition by the following formula (1). Lc = (TF - NF) / TF (1) Lc: confidence level, TF: total number of target frame images, NF: total number of frame images where target could not be recognized

[0040] For example, if the total number of target frame images TF is "100" and the total number of frame images in which the recognition target could not be recognized NF is "40", the reliability Lc is "0.6". Using the above formula (1), the reliability Lc can be easily calculated based on the total number of frame images in which the recognition target could not be recognized. This step is also called the "acquisition step".

[0041] 5, in step S30, the reliability estimation unit 365 determines whether the reliability Lc is less than a predetermined threshold Vth1. If the reliability Lc is less than the threshold Vth1 (step S30; YES), the reliability estimation unit 365 executes the process of step S21. The threshold Vth1 is, for example, "0.65." On the other hand, if the reliability Lc is not less than the predetermined threshold Vth1 (step S30; NO), the reliability estimation unit 365 executes the process of step S34.

[0042] In step S31, the reliability estimation unit 365 presents the reliability Lc to the instructor TP. FIG. 6 is a diagram showing an example of a display screen displayed on the display device 340 to present the reliability Lc. On the display screen, together with the value of the reliability Lc, a button for receiving an instruction from the instructor TP as to whether to perform re-teaching or to start creating a control program without re-teaching is displayed. Since the reliability is presented, the instructor TP can grasp the reliability regarding image recognition before creating a program for controlling the robot 100. This step is also called a "presentation step."

[0043] In step S32 shown in Fig. 5, the reliability estimation unit 365 judges whether or not the instructor TP instructs the instructor TP to perform re-teaching. If the instructor TP instructs the instructor TP to perform re-teaching (step S32; YES), in step S33, the reliability estimation unit 365 decides to perform re-teaching. Then, the process shown in Fig. 5 is terminated.

[0044] In step S32, if the instructor TP does not instruct to perform re-teaching (step S32; NO), the reliability estimation unit 365 executes the process of step S34. If the instructor TP does not instruct to perform re-teaching, this is the case where the instructor TP instructs to start creating a control program without performing re-teaching.

[0045] In step S34, the reliability estimation unit 365 decides to create a control program, and ends the process shown in Fig. 5. After that, the process of step S40 shown in Fig. 3 is executed.

[0046] 3 ends, creation of a control program is started by the control program creation unit 366. When creating the control program, the detection result of the recognition target by the object recognition unit 362 may be used.

[0047] As described above, in this embodiment, the reliability Lc of image recognition is obtained using the number of frame images in which the recognition target could not be recognized. Therefore, the instructor TP can grasp the reliability Lc of whether the recognition target could be recognized in multiple frame images. Furthermore, when using the image recognition result in other processes such as creating a control program, the instructor TP can determine whether or not to use the recognition result based on the reliability Lc.

[0048] If the hand of the instructor TP cannot be correctly recognized due to reasons such as the inclusion of subject blur in the teaching image, it becomes difficult to accurately identify the finger coordinates that indicate the positions of the joints of the instructor TP's fingers. In such a case, the created control program may be insufficient for the robot 100 to reproduce the teaching work of the instructor TP. For example, if a part of a series of movements of the instructor TP is not recognized, the control program will not allow the robot 100 to reproduce the corresponding movement. As a result, the robot 100 cannot accurately imitate the work of the instructor TP.

[0049] If the reliability Lc is not sufficient, the process of recognizing the recognition target is performed again for new frame images of a new task of the instructor TP. In this way, a plurality of frame images with sufficient reliability are prepared for creating a program for controlling the robot 100.

[0050] B. Other embodiments: B1. Alternative embodiment 1: In the embodiment, the predetermined range for the value of the undetected counter Cnt1 (see step S25 in FIG. 4) is equal to or greater than the lower limit Vl1 and less than the upper limit Vl2, but is not limited to this.

[0051] For example, a range equal to or greater than the lower limit Vl1 may be set as the predetermined range. In this case, even if the value of the undetected counter Cnt1 is equal to or greater than the upper limit Vl2, consecutive frame images in which the recognition target could not be recognized are treated as frame images in which the recognition target could not be recognized.

[0052] B2. Alternative embodiment 2: Alternatively, a range less than the upper limit Vl2 may be set as the predetermined range. In this case, when the value of the undetected counter Cnt1 is less than the lower limit Vl1, one or a series of multiple frame images in which the recognition target could not be recognized are treated as frame images in which the recognition target could not be recognized.

[0053] B3. Alternative embodiment 3: Alternatively, if the number of one or multiple consecutive frame images in which the recognition target could not be recognized is equal to or greater than the upper limit value Vl2, the number of one or multiple consecutive frame images in which the recognition target could not be recognized may not be included in the total number TF of frame images subject to equation (1) in the embodiment.

[0054] For example, suppose that the lower limit value Vl1 is set to "2" and the upper limit value Vl2 is set to "10." Suppose that the number of frame images acquired by the camera 200 capturing images of the instructor TP's work is "120." Suppose that the number of consecutive frame images in which the recognition target could not be recognized is "20," and that there is no other one or consecutive frame images in which the recognition target could not be recognized. In this case, the total number of target frame images is "100." The reliability Lc is calculated from equation (1) as Lc = (100-20) / 100 = 0.8.

[0055] In cases where it is acceptable for the recognition target to be continuously unable to be recognized and where there is a valid reason for the recognition target not being continuously able to be recognized, the relevant frame image can be excluded from the total number of frame images considered when calculating the reliability of image recognition, thereby making it possible to obtain a more realistic value for the reliability of image recognition.

[0056] B4. Alternative embodiment 4: In the embodiment, the reliability estimation unit 365 determines that a recognition target has been detected in a target frame image when at least a part of a hand has been detected in the target frame image (see step S23 in FIG. 4). However, the condition for determining that a recognition target has been detected in a target frame image is not limited to this.

[0057] For example, assume that the work of the instructor TP is to paint the first workpiece WK1 with a brush. In this case, the reliability estimation unit 365 may determine that the recognition target is detected in the target frame image when a part of a hand and a part of a brush, which is a tool, are detected in the target frame image. The reliability Lc regarding the image recognition is appropriately obtained by treating the frame image in which a part of a hand and a part of a tool are recognized as the frame image in which the recognition target is recognized.

[0058] Alternatively, the reliability estimation unit 365 may determine that a recognition target has been detected in a target frame image when a part of a brush, which is a tool, is detected in the target frame image.

[0059] B5. Alternative embodiment 5: In the estimation process, the processor 360 of the information processing device 300 may use values ​​set as the lower limit value Vl1 and the upper limit value Vl2 based on the nature of the work scheduled for the robot 100. The nature of the work is a feature that the work has. For example, if the work is a precision part, the instructor TP and the robot 100 must handle the work carefully when handling the work. If the work is a part that is easily damaged, the instructor TP and the robot 100 must not apply too much force when gripping the work. If the scheduled work is a work performed in collaboration with a person, the person works near the robot 100. In order to avoid contact between the robot 100 and the person, the robot 100 needs to accurately imitate the teaching work. In addition, if another robot works near the robot 100 when performing the scheduled work, the robot 100 needs to accurately imitate the teaching work in order to avoid contact between the robots. Furthermore, if there is an obstacle that impedes the operation of the robot 100 in the space in which the robot 100 performs the task, the robot 100 needs to accurately imitate the taught task in order to avoid the robot 100 coming into contact with the obstacle.

[0060] For example, assume that the work scheduled for the robot 100 is a work performed in collaboration with a human. In this case, it is desirable for the robot 100 to accurately imitate the scheduled work. Therefore, it is required that the accuracy with which the robot 100 imitates the teaching work of the instructor TP is high, compared to when the work scheduled for the robot 100 is not a work performed in collaboration with a human. For this reason, a smaller value is set as the lower limit value Vl1, compared to when the work scheduled for the robot 100 is not a work performed in collaboration with a human. Since the lower limit value Vl1 and the upper limit value Vl2 are set according to the nature of the work scheduled for the robot 100, a reliability of image recognition that matches the nature of the work is required.

[0061] B6. Alternative Embodiment 6: The instructor TP or another person (hereinafter, referred to as instructor TP, etc.) may specify the desired lower limit value Vl1 and upper limit value Vl2. In this case, the processor 360 of the information processing device 300 displays an input screen for receiving the lower limit value Vl1 and upper limit value Vl2 on the display device 340 prior to execution of an estimation process (see Figs. 4 and 5) for estimating the reliability of image recognition. The instructor TP, etc. inputs the lower limit value Vl1 and upper limit value Vl2 using the input device 330. The processor 360 stores the specified lower limit value Vl1 and upper limit value Vl2 in the memory 310, and uses the specified lower limit value Vl1 and upper limit value Vl2 when executing the estimation process. The instructor TP, etc. is also referred to as a "user."

[0062] B7. Alternative Embodiment 7: In the embodiment, when the reliability Lc is less than the threshold Vth1, the processor 360 presents the reliability Lc and accepts an instruction from the instructor TP as to whether to perform re-teaching or to start creating a control program (see step S21 in FIG. 5). However, the processor 360 does not have to present the reliability Lc.

[0063] Fig. 7 is a diagram showing an example of a display screen displayed on the display device 340. In the example shown in Fig. 7, a notification that the reliability is low and buttons for accepting an instruction as to whether to perform re-teaching or to start creating a control program without re-teaching are displayed.

[0064] B8. Alternative Embodiment 8: In the embodiment, the processor 360 presents the reliability Lc only when the reliability Lc is less than the threshold Vth1. Alternatively, the processor 360 may present the reliability Lc not only when the reliability Lc is less than the threshold Vth1, but also when the reliability Lc is equal to or greater than the threshold Vth1. In this case, the instructor TP or the like can grasp the reliability Lc regarding the image recognition regardless of the value of the reliability Lc.

[0065] B9. Alternative Embodiment 9: Instead of the confidence level Lc, the processor 360 may present a confidence level for the image recognition based on the confidence level Lc. The confidence level indicates the degree of confidence for the image recognition based on the confidence level Lc. For example, the confidence level is defined as follows: Trust level when trust level Lc is 0 to 50: "Low level" Trust level when trust level Lc is between 51 and 74: "Medium" Trust level when trust level Lc is between 75 and 100: "High level"

[0066] Since the confidence level is presented, the instructor TP or the like can understand the confidence level regarding the image recognition before creating a program for controlling the robot.

[0067] B10. Alternative embodiment 10: Fig. 8 is an example of a display screen for presenting the reliability Lc according to another embodiment 10. As shown in Fig. 8, the processor 360 may display a plurality of frame images used in estimating the reliability Lc on the display device 340. For example, when the instructor TP or the like presses a play button, a moving image made up of the plurality of frame images used in estimating the reliability is played.

[0068] Since the reliability is accompanied by a plurality of frame images used to estimate the reliability, the instructor TP or the like can check the state of the work performed by the instructor TP.

[0069] B11. Other embodiment 11: Fig. 9 is an example of a display screen for presenting the reliability Lc according to another embodiment 11. As shown in Fig. 9, the processor 360 may display the reliability Lc, a plurality of frame images used in estimating the reliability, and a graph showing the positions on the time axis of the frame images in which the recognition target could not be recognized and the frame images in which the recognition target could be recognized, on the display device 340. For example, when the instructor TP or the like presses a play button, a moving image consisting of the plurality of frame images used in estimating the reliability is played. When playing the moving image, an indicator for recognizing which of the plurality of frame images is being played may be displayed below the graph or the like.

[0070] In the eleventh embodiment, since a plurality of frame images used for estimating the reliability are presented, the instructor TP or the like can check the state of the instructor TP's work. In addition, the instructor TP or the like can check the position on the time axis of the frame image in which the recognition target could not be recognized.

[0071] B12. Other embodiment 12: In the embodiment, when presenting the reliability Lc, the processor 360 receives an instruction from the user as to whether to perform re-teaching or to start creating a control program without re-teaching (see FIG. 6). Alternatively, the information processing device 300 may only present the reliability Lc. The information processing device 300 may not receive an instruction from the user as to whether to perform re-teaching or to start creating a control program without re-teaching.

[0072] B13. Other embodiment 13: In the embodiment, the information processing device 300 calculates the reliability Lc using the formula (1). Alternatively, the information processing device 300 may calculate the reliability Lc using the following formula (2). Lc = DF / TF (2) (Lc: confidence level, TF: total number of target frame images, DF: total number of frame images in which the target was recognized)

[0073] In another embodiment 13, the reliability Lc of the image recognition is obtained using the number of frame images in which the recognition target was recognized. Therefore, the instructor TP can grasp the reliability Lc of whether the recognition target was recognized in multiple frame images. Furthermore, when using the image recognition result in other processes such as creating a control program, the instructor TP can determine whether or not to use the recognition result based on the reliability Lc.

[0074] B14. Other embodiment 14: In the embodiment, an example has been described in which the reliability Lc is calculated using formula (1). However, the processor 360 does not have to calculate the reliability Lc. When the number of one or a plurality of consecutive frame images in which the recognition target could not be recognized is equal to or greater than a predetermined second threshold value Vth2, the processor 360 may determine that the reliability is low and notify the user accordingly. Since the reliability Lc is not calculated, the processing load of the information processing device 300 is reduced.

[0075] B15. Other embodiment 15: In the embodiment, an example has been described in which the information processing device 300 uses a machine learning model for object detection to detect a recognition target in a frame image. Alternatively, the information processing device 300 may use a machine learning model for object recognition.

[0076] B16. Other embodiment 16: Different upper and lower limit values ​​may be used for a part of the series of actions performed by the instructor TP and another part. In this case, the action recognition unit 363 recognizes, for example, the action of the instructor TP gripping the first workpiece WK1 and the action of releasing the gripped first workpiece WK1 for the frame image. For the recognition of the frame image while the instructor TP is gripping the first workpiece WK1, the lower limit value Vl1 is set to "3" and the upper limit value Vl2 is set to "none". For the recognition of the frame image while the instructor TP is not gripping the first workpiece WK1, the lower limit value Vl1 is set to "5" and the upper limit value Vl2 is set to "15". In this way, the lower limit value Vl1 is set smaller for the period during which the instructor TP is gripping the first workpiece WK1 than for the period during which the instructor TP is not gripping the first workpiece WK1. During the period when the instructor TP is holding the first workpiece WK1, the accuracy of image recognition can be improved, and the quality of the created control program can be improved.

[0077] B17. Other embodiment 17: In the embodiment, the instructor TP is imaged using one camera 200, but two or more cameras may be used. For example, the upper body of the instructor TP may be imaged by one camera, and the hands of the instructor TP may be imaged by another camera. This allows the motion of the instructor TP to be recognized more accurately.

[0078] In the embodiment, an example in which the robot 100 is a vertical articulated robot has been shown (see FIG. 1), but the robot 100 may be a vertical articulated robot. Alternatively, the robot 100 may be a dual-arm robot. In this case, the instructor TP uses both arms to perform the work for teaching.

[0079] In the above embodiment, pick-and-place is described as an example of the predetermined operation, but the predetermined operation is not limited to this. The predetermined operation may be, for example, a coating operation, a screw driving operation, a nail driving operation with a hammer, a workpiece insertion operation, a fitting operation, an assembly operation, etc.

[0080] Furthermore, the means for realizing the functions of the information processing device 300 is not limited to software, and may be partly or entirely realized by dedicated hardware. For example, the dedicated hardware may be a circuit such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In the embodiment, an example has been described in which the functions of the control program creation unit 366 are realized by the processor 360 of the information processing device 300. However, the functions of the control program creation unit 366 may be realized by a computer different from the information processing device 300.

[0081] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted.

[0082] C. Other forms: (1) According to one embodiment of the present disclosure, there is provided a method for estimating reliability of image recognition, which is a method for estimating reliability of image recognition performed based on a plurality of frame images, and includes an object recognition step of executing a process for recognizing a recognition object including at least a part of the instructor's hand for the plurality of frame images capturing an image of an operator working on a workpiece, and an acquisition step of calculating the reliability of the image recognition using the number of frame images in which the recognition object could not be recognized or the number of frame images in which the recognition object could be recognized. According to this embodiment, the reliability of image recognition is obtained using the number of frame images in which the recognition target could not be recognized or the number of frame images in which the recognition target could be recognized, so that the user can grasp the reliability of whether the recognition target including at least a part of a hand in a plurality of frame images could be recognized. Furthermore, when using the result of image recognition for other processing, the user can determine whether or not to use the result based on the reliability.

[0083] (2) In the estimation method of the above aspect, the reliability of the image recognition may be calculated by the following formula (1), Lc=(TF-NF) / TF (1), where Lc is the reliability, TF is the total number of target frame images, and NF is the total number of frame images in which the recognition target cannot be recognized. According to this aspect, the reliability can be easily calculated based on the total number of frame images in which the recognition target could not be recognized.

[0084] (3) In the estimation method of the above form, in the acquisition step, if the number of one or multiple consecutive frame images in which the recognition target could not be recognized is equal to or less than a lower limit value that is an integer greater than or equal to 1, the one or multiple consecutive frame images do not have to be treated as frame images in which the recognition target could not be recognized. When the number of consecutive frame images in which the recognition target is not recognized is less than the lower limit, this corresponds to a case in which consecutive failures to recognize the recognition target are acceptable. According to this embodiment, when consecutive failures to recognize the recognition target are acceptable, the corresponding frame image is not treated as a frame image in which the recognition target is not recognized when calculating the reliability of the image recognition. Therefore, when using the reliability to determine whether or not to use the image recognition results in other processing, a more useful value can be obtained as the reliability.

[0085] (4) In the estimation method of the above aspect, in the acquisition step, if the number of the one or multiple consecutive frame images in which the recognition target could not be recognized is equal to or greater than an upper limit value that is greater than the lower limit value, the one or multiple consecutive frame images do not have to be treated as frame images in which the recognition target could not be recognized. When the number of consecutive frame images in which the recognition target could not be recognized exceeds the upper limit, it corresponds to, for example, a case where the worker's hand is outside the imaging range immediately after the start of shooting, or a case where the worker's hand is inevitably outside the imaging range for some reason. In such a case, it is not desirable to use the number of frame images in which the recognition target could not be recognized as a calculation of the reliability of image recognition. According to this embodiment, when there is a valid reason for the consecutive failure to recognize the recognition target, the corresponding frame image is not treated as a frame image in which the recognition target could not be recognized when calculating the reliability of image recognition. Therefore, when the reliability is used to determine whether or not to use the result of image recognition in other processing, a more useful value can be obtained as the reliability.

[0086] (5) In the estimation method of the above embodiment, in the acquisition step, if the number of the one or multiple consecutive frame images in which the recognition target could not be recognized exceeds an upper limit value that is greater than the lower limit value, the number of the one or multiple consecutive frame images does not have to be included in the total number of the target frame images. According to this form, in cases where it is acceptable that the recognition target cannot be recognized consecutively and where there is a valid reason for the recognition target not being recognized consecutively, the corresponding frame image is not included in the total number of target frame images when calculating the reliability of image recognition, thereby making it possible to obtain a more realistic value for the reliability of image recognition.

[0087] (6) The estimation method of the above aspect may further include a step executed before the object recognition step, of receiving designation of the lower limit threshold and the upper limit value from a user. This aspect allows the user to specify the desired lower and upper limits.

[0088] (7) In the estimation method of the above aspect, the lower limit value and the upper limit value are set based on the nature of the task performed by the instructor on the workpiece. According to this aspect, the lower and upper limits are set according to the nature of the work, so that a reliability of image recognition suited to the nature of the work can be obtained.

[0089] (8) In the estimation method of the above form, the task of the instructor on the workpiece is a predetermined task that the instructor performs on the workpiece using the hand, and in the object recognition step, the frame image in which at least a part of the instructor's hand can be recognized may be treated as a frame image in which the recognition object can be recognized. Since the worker performs work on the workpiece using his / her hands, it is necessary to recognize at least a part of the hand. According to this embodiment, a frame image in which at least a part of the hand is recognized is treated as a frame image in which the recognition target is recognized, thereby appropriately determining the reliability of the image recognition.

[0090] (9) In the estimation method of the above form, the work by the instructor on the workpiece is performed to create a program for controlling a robot based on a frame image, and the method may further include a presentation step of presenting the reliability. According to this embodiment, the reliability is presented, so that the user can understand the reliability of the image recognition before creating a program for controlling the robot.

[0091] (10) The estimation method of the above form may further include a step executed before the target recognition step, the step being an image acceptance step of accepting new frame images capturing new work by the instructor, and a re-execution step of executing the image acceptance step, the target recognition step, and the acquisition step again when the reliability is equal to or less than a predetermined threshold value. According to this aspect, if the reliability is not sufficient, new frame images capturing a new task of the worker are accepted, and the process of recognizing the recognition target is performed again. When creating a program for controlling a robot, it is possible to prepare a plurality of frame images that provide sufficient reliability.

[0092] (11) In the estimation method according to the above aspect, in the presenting step, the plurality of frame images used in estimating the reliability may be presented together with the reliability. According to this aspect, the reliability is presented along with a plurality of frame images used to estimate the reliability, so that the user can check the state of the worker's work.

[0093] (12) According to another aspect of the present disclosure, there is provided an information processing device used for creating a program for controlling a robot, the information processing device including: an object recognition unit that executes a process for recognizing a recognition object including at least a part of an instructor's hand in a plurality of frame images capturing an instructor's work on a workpiece; and an acquisition unit that calculates a reliability of image recognition using the number of frame images in which the recognition object is not recognized or the number of frame images in which the recognition object is recognized. According to this embodiment, the reliability of image recognition is obtained using the number of frame images in which the recognition target could not be recognized or the number of frame images in which the recognition target could be recognized, so that the user can grasp the reliability of whether the recognition target including at least a part of a hand in a plurality of frame images could be recognized. Furthermore, when using the result of image recognition for other processing, the user can determine whether or not to use the result based on the reliability. [Explanation of symbols]

[0094] 1...robot system, 100...robot, 200...camera, 300...information processing device, 310...memory, 320...input / output interface, 330...input device, 340...display device, 360...processor, 361...image reception unit, 362...object recognition unit, 363...motion recognition unit, 365...reliability estimation unit, 366...control program creation unit, Cnt1...undetected counter, L1...first list, SA1...first supply area, TA1...first target area, TB1...work table, TF...total number, TP...instructor, Vl1...lower limit, Vl2...upper limit, Vth1...threshold, WA1...first work area, WK1...first work

Claims

1. 1. A method for estimating reliability of image recognition performed based on a plurality of frame images, comprising: an object recognition step of executing a process of recognizing a recognition object including at least a part of the instructor's hand from the plurality of frame images obtained by capturing an instructor's work on a workpiece; an acquisition step of calculating a reliability of the image recognition using the number of frame images in which the recognition target was not recognized or the number of frame images in which the recognition target was recognized; A method for estimating confidence in image recognition, comprising:

2. A method for estimating reliability of image recognition according to claim 1, Let Lc be the reliability of the image recognition, TF be the total number of target frame images, and NF be the total number of frame images in which the recognition target could not be recognized. The reliability is calculated by the following formula (1): Lc=(TF-NF) / TF...(1) Required, A method for estimating confidence in image recognition.

3. A method for estimating reliability of image recognition according to claim 2, comprising: In the obtaining step, If the number of one or a plurality of consecutive frame images in which the recognition target could not be recognized is equal to or less than a lower limit value which is an integer equal to or greater than 1, the one or a plurality of consecutive frame images are not treated as frame images in which the recognition target could not be recognized. A method for estimating confidence in image recognition.

4. A method for estimating reliability of image recognition according to claim 3, comprising: In the obtaining step, if the number of the one or a plurality of consecutive frame images in which the recognition target could not be recognized exceeds an upper limit value that is greater than the lower limit value, the one or a plurality of consecutive frame images are not treated as frame images in which the recognition target could not be recognized; A method for estimating confidence in image recognition.

5. A method for estimating reliability of image recognition according to claim 4, comprising: In the obtaining step, If the number of the one or a plurality of consecutive frame images in which the recognition target could not be recognized exceeds an upper limit value that is greater than the lower limit value, the number of the one or a plurality of consecutive frame images is not included in the total number of the target frame images. A method for estimating confidence in image recognition.

6. A method for estimating reliability of image recognition according to claim 5, comprising: A step performed before the object recognition step, comprising: receiving designation of the lower limit value and the upper limit value from a user; Further comprising: A method for estimating confidence in image recognition.

7. A method for estimating reliability of image recognition according to claim 5, comprising: The lower limit value and the upper limit value are set based on the nature of the work performed by the teacher on the workpiece. A method for estimating confidence in image recognition.

8. A method for estimating reliability of image recognition according to claim 6 or 7, comprising: The task of the instructor on the workpiece is a predetermined task that the instructor performs on the workpiece using the hands, In the object recognition step, The frame image in which at least a part of the hand of the instructor is recognized is treated as a frame image in which the recognition target is recognized. A method for estimating confidence in image recognition.

9. A method for estimating reliability of image recognition according to claim 8, comprising: the work by the instructor on the workpiece is performed for creating a program for controlling a robot based on a frame image, a presentation step of presenting the reliability; Further comprising: A method for estimating confidence in image recognition.

10. A method for estimating reliability of image recognition according to claim 9, comprising: an image receiving step, which is executed before the target recognition step, of receiving a plurality of new frame images obtained by capturing a new task of the instructor; a re-execution step of re-executing the image receiving step, the object recognizing step, and the acquiring step when the reliability is equal to or less than a predetermined threshold; Further comprising: A method for estimating confidence in image recognition.

11. A method for estimating reliability of image recognition according to claim 10, comprising: the presenting step presents the plurality of frame images used in estimating the reliability together with the reliability. A method for estimating confidence in image recognition.

12. An information processing device used to create a program for controlling a robot, an object recognition unit that executes a process of recognizing a recognition object including at least a part of the instructor's hand from a plurality of frame images obtained by capturing an instructor's work on a workpiece; an acquisition unit that calculates a reliability of image recognition using the number of frame images in which the recognition target is not recognized or the number of frame images in which the recognition target is recognized; An information processing device comprising:

Images