Image processing device and image processing method

Abstract

The present invention prompts a user to review a parameter that is effective for re-teaching a visual detection function and / or improving a detection result from visual detection before the occurrence of non-detection or erroneous detection of an object by the visual detection function, and prevents the occurrence of detection failure in advance.　This image processing device comprises: an image processing unit that performs image processing on an image acquired by a visual sensor and performs visual detection of an object; a detection threshold value setting unit that sets a plurality of detection threshold values for individual detection parameters used in visual detection of the object in the image processing unit; and a determination unit that determines the detection threshold value with which the object visually detected by the image processing unit has been detected, from among the plurality of detection threshold values.

Description

Image processing device and image processing method

[0001] The present disclosure relates to an image processing device and an image processing method for detecting an object from an image.

[0002] Prior art includes a vision detection function that uses an image processing device to detect a specific object within an image within the field of view of an imaging device and acquire the position of the detected object. For example, a robot installed in a production facility can acquire the position of an object detected by the vision detection function and use the acquired position information to manipulate an unpositioned object. Furthermore, by having the robot grasp the object, the vision detection function can detect scratches on the object. Another proposed method for controlling an imaging device in vision detection involves, for example, obtaining main and secondary images of an object captured by the imaging device using predetermined main and secondary parameters different from the main and secondary parameters, performing image processing such as pattern matching on each of the main and secondary images, and comparing the image processing results of the main and secondary images to determine whether or not to update the secondary parameters as new main and secondary parameters, thereby reviewing and updating imaging parameters while the device is operating without adversely affecting the operating status, such as stopping the device. For example, see Patent Document 1.

[0003] Japanese Patent Application Laid-Open No. 2019-106209

[0004] When detecting objects using a vision detection function, detection may not always be performed correctly due to disturbances caused by environmental changes or small changes in the object. Furthermore, if a false detection of an object occurs, the robot may move to a location different from the actual location of the object, or if the robot stops due to a failure to detect the object, the user of the robot may become aware of a malfunction in the vision detection function. Therefore, to prevent malfunctions caused by vision detection malfunctions (failure to detect or false detection), it is necessary to monitor detection results one by one. However, because a large number of object detections are obtained by robots deployed in production facilities, etc., it is difficult to monitor and understand all detection results. Furthermore, it is clear that the shutdown and malfunction of production facilities due to the above-mentioned malfunctions of the detection function will result in significant losses for the user.

[0005] Therefore, in order to determine the vision detection result, the vision detection result is determined using an index calculated based on detection parameters described below. Examples of detection parameters include, but are not limited to, a score, position information of the object, angle, size, flattening ratio, flattening direction, distortion, and contrast. By monitoring and understanding the vision detection result using the detection parameters, it is desirable to prompt the user to re-teach the vision detection function and / or review parameters effective for improving the vision detection result ("taught parameters") before the vision detection function fails to detect or erroneously detects an object, thereby preventing detection failures from occurring.

[0006] One aspect of the image processing device of the present disclosure includes an image processing unit that performs image processing on an image acquired by a visual sensor and performs vision detection of an object, a detection threshold setting unit that sets multiple detection thresholds for each detection parameter used in the vision detection of the object in the image processing unit, and a determination unit that determines which of the multiple detection thresholds was used to detect the object that was vision detected by the image processing unit.

[0007] One aspect of the image processing method of the present disclosure includes an image processing step of performing image processing on an image acquired by a visual sensor and performing vision detection of an object, a detection threshold setting step of setting multiple detection thresholds for each detection parameter used in the vision detection of the object in the image processing step, and a determination step of determining which of the multiple detection thresholds was used to detect the object that was vision detected by the image processing step.

[0008] 1 is a diagram illustrating an example of the configuration of a robot system according to an embodiment. FIG. 1 is a diagram illustrating an example of a vision program in this embodiment. FIG. 2 is a diagram illustrating an example of a condition determination tool. FIG. 3 is a diagram illustrating an example of the functional block configuration of a robot control device. FIG. 4 is a diagram illustrating an example of a display screen for vision detection when pattern matching is selected as image processing for vision detection. FIG. 5 is a diagram illustrating an example of a display screen for vision detection when three-dimensional model detection is selected as image processing for vision detection. FIG. 6 is a diagram illustrating an example of a display screen for vision detection when three-dimensional box detection is selected as image processing for vision detection. FIG. 7 is a diagram illustrating an example of a case where the position of an object is used as a detection parameter. FIG. 8 is a diagram illustrating an example of a dialog box for instructing readjustment of teaching parameters. FIG. 9 is a diagram illustrating an example of a display screen displaying information necessary for improving detection results. FIG. 10 is a diagram illustrating an example of a screen listing parameters effective for improving detection results. FIG. 11 is a flowchart illustrating notification processing of a robot control device.

[0009] A robot system according to an embodiment will be described in detail below with reference to the drawings. Here, an example in which an image processing device is included in a robot control device is illustrated. FIG. 1 is a diagram illustrating an example of the configuration of a robot system according to an embodiment. As illustrated in FIG. 1, the robot system 1 includes a visual sensor control device 10, a robot control device 20, a teaching pendant 25, a robot 30, a storage device 40, and a workbench 50, all of which are disposed in a production facility such as a factory. The robot control device 20 also includes a display unit 21 such as a liquid crystal display. The display unit 21 may be provided on the teaching pendant 25. Alternatively, the display unit 21 may be a display device separate from the robot control device 20 and connected to the robot control device 20. The visual sensor control device 10, the robot control device 20, the teaching pendant 25, the robot 30, and the storage device 40 are directly connected to each other via a connection interface (not shown). The visual sensor control device 10, robot control device 20, teaching pendant 25, robot 30, and storage device 40 may be interconnected and communicate with each other via a network (not shown) such as a LAN (Local Area Network) or the Internet. In this case, the visual sensor control device 10, robot control device 20, teaching pendant 25, robot 30, and storage device 40 are provided with communication units (not shown) for communicating with each other via such connections.

[0010] <Visual Sensor Control Device 10> The visual sensor control device 10 is an information processing device such as a computer. For example, based on setting information for the visual sensor 32 installed at the hand of the robot 30 (described later), the visual sensor control device 10 controls the imaging operation of the visual sensor 32 and acquires an image of the workpiece 60 placed on the worktable 50. The visual sensor control device 10 outputs the acquired image to the robot control device 20. Note that while the visual sensor control device 10 is illustrated as being connected to the robot control device 20, it may also be configured within the robot control device 20. The visual sensor 32 is a digital camera or the like, and captures a two-dimensional image of the area where the workpiece 60 placed on the worktable 50 is located, projected onto a plane perpendicular to the optical axis of the visual sensor 32. The image captured by the visual sensor 32 may be a visible light image such as an RGB color image, a grayscale image, or a depth image. The visual sensor 32 may also be configured to include an infrared sensor to capture thermal images, or may be configured to include an ultraviolet sensor to capture ultraviolet images for inspecting scratches, spots, etc. on the surface of an object. The visual sensor 32 may be configured to include an X-ray camera sensor to capture X-ray images, or may be configured to include an ultrasonic sensor to capture ultrasonic images. In the case of a three-dimensional measuring device such as a stereo camera, the visual sensor 32 may capture distance images or point cloud images. While the visual sensor 32 is disposed at the tip of the robot 30 as shown in FIG. 1 , it may also be fixed to a wall, ceiling, or the like of the production facility.

[0011] <Storage Device 40> The storage device 40 is a solid-state drive (SSD) or a hard disk drive (HDD), and is connected to the robot control device 20. In addition to storing the execution history, the storage device 40 stores setting information (camera data) for the visual sensor 32, detection method programs (vision program, image processing program), and the like. While the storage device 40 is illustrated as being connected to the robot control device 20, it may be configured within the robot control device 20 or connected to the teaching operation panel 25. The storage device 40 may also be a data server on a network, and may store the execution history, setting information (camera data) for the visual sensor 32, detection method programs (vision program, image processing program), and the like. FIG. 2 is a diagram showing an example of a vision program in this embodiment. In the vision program shown in FIG. 2, the "..." portion on the first and second lines stores a program instructed (specified) for detection by the user. The "Moshi" portion on the third line sets a determination condition for notifying the user. The "..." part of "Vision User Information" on the fourth line sets the content and method of notification to the user. That is, the "..." part of "Vision User Information" sets the notification content as well as the notification method, such as displaying the notification on the screen of the display unit 21, notifying the user with a sound such as an alarm, or notifying the user with light such as an alarm lamp.

[0012] 2, the vision program has a "Vision User Information" command, and the "..." portion of the "Vision User Information" specifies the notification content, notification method, etc., but is not limited to this. For example, as will be described later, the robot control device 20 may display a condition judgment tool shown in FIG. 3 on the display unit 21 and set the process of "notifying the user" based on the user's input operation via the teaching operation panel 25. By doing so, in the case of the setting shown in FIG. 3, the robot control device 20 sets the score in pattern match 1 (pattern matching) as "value 1," and performs the process of notifying the user when the detection result is that "value 1" is less than "60."

[0013] <Robot 30> The robot 30 operates under the control of the robot control device 20, which will be described later. The robot 30 includes a base for rotating around a vertical axis, a moving and rotating arm, and a hand 31 attached to the arm for grasping a workpiece 60 placed on the worktable 50. While FIG. 1 shows the robot 30 equipped with a gripping-type pick-up hand as the hand 31, an air suction-type pick-up hand may also be attached. As described above, a visual sensor 32 is disposed at the end of the robot 30, and captures an image of the workpiece 60 under the control of the visual sensor control device 10. As will be described later, the robot control device 20 executes a robot program and a vision program to detect the workpiece 60 as an object from the image captured by the visual sensor 32 and output a control signal to the robot 30 according to the position of the detected workpiece 60. The robot 30 drives the arm and hand 31 to move the hand 31 to the position of the workpiece 60 on the worktable 50 and grasp the workpiece 60. The destination of the gripped workpiece 60 is not shown in the drawings. The specific configuration of the robot 30 is well known to those skilled in the art, so a detailed description thereof will be omitted.

[0014] <Robot controller 20> The robot controller 20 is a device known to those skilled in the art for controlling the operation of the robot 30, and has a display unit 21. In FIG. 1, a teaching operation panel 25 that teaches the robot 30 an operation is connected to the robot controller 20. FIG. 4 is a diagram showing an example of the functional block configuration of the robot controller 20. As shown in FIG. 4, the robot controller 20 is configured to include an image processing unit 210, a detection threshold setting unit 220, a notification information setting unit 230, a determination unit 240, and a notification unit 250. In order to realize the operation of the functional blocks in FIG. 4, the robot controller 20 is equipped with an arithmetic processing device (not shown), such as a CPU (Central Processing Unit). The robot control device 20 also includes auxiliary storage devices (not shown) such as a ROM (Read Only Memory) or HDD that store various control programs and vision programs (image processing programs), and a main storage device (not shown) such as a RAM (Random Access Memory) for storing data temporarily required for the arithmetic processing unit to execute programs.

[0015] In the robot controller 20, the arithmetic processing unit reads application software such as the OS, control programs, and vision programs from the auxiliary storage device, and executes arithmetic processing based on the OS and application software while loading the read OS and application software into the main storage device. Based on the results of this calculation, the robot controller 20 controls each piece of hardware. This allows the processing represented by the functional blocks in Figure 4 to be realized. In other words, the robot controller 20 can be realized by the cooperation of hardware and software. The display unit 21 may be disposed on the teaching pendant 25.

[0016] The image processing unit 210 performs image processing on the image acquired by the visual sensor 32 to perform vision detection of the object. Specifically, the image processing unit 210 selects image processing (vision detection) corresponding to the image captured by the visual sensor 32 based on, for example, a user's input operation via the teaching operation panel 25, and executes the selected image processing to detect the object (workpiece 60) from the captured image. For example, when a two-dimensional image such as an RGB color image or a grayscale image is captured by the visual sensor 32 and the user selects pattern matching as the image processing for vision detection, the image processing unit 210 reads pre-registered two-dimensional image data of the object from the storage device 40 and detects the object (workpiece 60) having the same shape as the read two-dimensional image data from the image based on pattern matching.

[0017] FIG. 5 illustrates an example of a vision detection display screen when pattern matching is selected as the image processing for vision detection. The image processing unit 210 calculates the similarity (degree of match) between the object detected from the image by pattern matching and the teaching data of a model pattern previously taught, based on a "score," which is one of the detection parameters. For example, if the object perfectly matches the teaching data, the image processing unit 210 calculates the score as "100." On the other hand, if the object does not match the teaching data, the image processing unit 210 calculates the score as "0." That is, the image processing unit 210 calculates the score as a value between "0" and "100" depending on the degree of similarity between the object and the teaching data. Note that FIG. 5 illustrates an example in which the object (workpiece 60) is a nut. The display screen in FIG. 5 also includes an area (left side) for displaying the captured image, an area (upper right side) for displaying the pattern match selected by the user, and an area (lower right side) for displaying the pattern match settings.

[0018] Furthermore, when a three-dimensional image such as a distance image or a point cloud image is captured by the visual sensor 32 and the user selects three-dimensional pattern matching (three-dimensional model detection) as the image processing for vision detection, the image processing unit 210 reads the three-dimensional CAD data of the object from the storage device 40 and detects an object (workpiece 60) with the same shape as the read three-dimensional CAD data from the image based on the three-dimensional model detection. FIG. 6 shows an example of the vision detection display screen when three-dimensional model detection is selected as the image processing for vision detection. Based on the three-dimensional model detection, the image processing unit 210 detects a three-dimensional model of the object (workpiece 60) generated by the three-dimensional CAD from the three-dimensional data of the image and detects its three-dimensional position and orientation. The image processing unit 210 then calculates the degree of match between the three-dimensional model and the three-dimensional data of the image for the detected position and orientation of the object based on a "score," which is one of the detection parameters. The display screen in Figure 6 has an area (left side) for displaying a point cloud image, an area (top right side) for displaying a 3D model detection selected by the user, and an area (bottom right side) for displaying the settings for the 3D model detection.

[0019] Furthermore, when a visual sensor 32, such as a bulk sensor or a three-dimensional visual sensor, captures a three-dimensional image of a box palletized on a pallet as the workbench 50, and the user selects three-dimensional box detection as the image processing for vision detection, the image processing unit 210 reads the box size specified by the user from the storage device 40 and detects the top surface of a box of the same size as the read size, thereby detecting the target object (workpiece 60) from the image. FIG. 7 illustrates an example of a display screen for vision detection when three-dimensional box detection is selected as the image processing for vision detection. In the three-dimensional box detection, the image processing unit 210 calculates the accuracy (degree of match) between the specified box size and the detected target size based on a "score," which is one of the detection parameters. The display screen in FIG. 7 includes an area for displaying the captured image (left side), an area for displaying the three-dimensional box detection selected by the user (upper right), and an area for displaying the settings for the three-dimensional box detection (lower right). In addition, on the display screen of Figure 7 (lower right), the box information for the specified box size specifies the width and depth of the top surface size as "330 mm" and "460 mm", and the height as "290 mm".

[0020] 5 to 7, the image processing unit 210 has exemplified pattern matching, three-dimensional model detection, and three-dimensional box detection, but other detection methods (image processing) may be selected by the user. The image processing unit 210 then detects the object (workpiece 60) from the image captured by the visual sensor 32 using the selected detection method (image processing), and calculates the degree of match of the detected object based on a "score," which is one of the detection parameters.

[0021] Furthermore, while the image processing unit 210 calculated the degree of match of the detected object using the "score" as a detection parameter, the degree of match of the detected object may also be calculated using measurement values ​​such as the size, brightness, area, length, position, and orientation of the detected object as detection parameters. FIG. 8 is a diagram illustrating an example in which the position of the object is used as the detection parameter. As shown in FIG. 8 , the image processing unit 210 may, for example, detect an object (workpiece 60a, 60b) having the same shape as the two-dimensional image data from the image by pattern matching, and calculate the position (e.g., x, y coordinates) of the detected object as the detection parameter. The imaging range (robot graspable area) E1 of the visual sensor 32 and the range E2 of the user notification area in FIG. 8 will be described later.

[0022] The detection threshold setting unit 220 sets multiple detection thresholds for each detection parameter used in the vision detection of the target object (workpiece 60) in the image processing unit 210. Specifically, the detection threshold setting unit 220 sets multiple detection thresholds, such as "90," "60," and "0," for "score," which is one of the detection parameters, based on, for example, a user's input operation via the teaching operation panel 25. Note that the detection threshold setting unit 220 may, for example, display a condition judgment tool shown in FIG. 3 on the display unit 21 and set multiple detection thresholds for each detection parameter based on the user's input operation via the teaching operation panel 25. The detection threshold setting unit 220 associates the multiple set detection thresholds with the detection parameters and stores them in the storage device 40.

[0023] As shown in FIG. 8 , the detection threshold setting unit 220 sets, as multiple detection thresholds for the "position" that is one of the detection parameters, an imaging range (robot graspable area) E1 of the visual sensor 32 and a user notification area E2 (upper and lower x and y limits). The imaging range E1 is the imaging range of the visual sensor 32 and an area in which the robot 30 can grasp an object (workpiece 60), and is set on the top surface of the work table 50 or a conveyor (not shown). The user notification area E2 is set inside the imaging range E1. Typically, an object, such as the workpiece 60a, is placed inside the imaging range E1 and the user notification area E2. The object is then detected by the image processing unit 210, grasped by the robot 30, and moves on to the next task. However, for example, if for some reason the workpiece 60b is placed within the imaging range E1 but straddles the user notification area E2, the image processing unit 210 can detect it and the robot 30 can grasp it. However, if the workpiece 60b is slightly misaligned, it will fall outside the imaging range E1, preventing it from being detected by the image processing unit 210 or grasped by the robot 30. Therefore, the notification information setting unit 230 (described later) adds notification information to the detection thresholds for the imaging range E1, the user notification area E2, etc., so that when a workpiece 60b is detected straddling or outside the user notification area E2 and about to leave the imaging range E1, the notification unit 250 (described later) notifies the user of the notification information. This allows the user to devise countermeasures before the system stops. For example, the user may first suspect a defect in the conveyor on which the workpiece 60b is placed, and then review the detection program (vision program).

[0024] In addition, the detection threshold setting unit 220 may set a detection threshold for the rate of change (rate of decrease, rate of increase) of the score indicating the degree of match between the object (workpiece 60) detected from the image captured by the visual sensor 32 and the teaching data or 3D CAD data.

[0025] The notification information setting unit 230 adds notification information to the multiple detection thresholds set by the detection threshold setting unit 220. Specifically, for example, when the score value calculated by the image processing unit 210 is equal to or greater than the detection threshold "90," the detected object substantially matches the model of the teaching data, and therefore the notification information setting unit 230 adds notification information of "detection successful" to the detection threshold "90" based on the user's input operation via the teaching operation panel 25 and stores the notification information in the storage device 40. Furthermore, for example, when the score value calculated by the image processing unit 210 is equal to or greater than the detection threshold "60" but less than the detection threshold "90," the detected object neither matches nor mismatches the model of the teaching data, and therefore the detection is unstable, and therefore the notification information setting unit 230 adds notification information of "detection unstable" to the detection threshold "60" based on the user's input operation and includes a message instructing readjustment of parameters that are effective for improving the detection result of vision detection, and stores the notification information in the storage device 40. Furthermore, for example, if the score value calculated by the image processing unit 210 is less than the detection threshold value of "60", the detected object does not match the model of the teaching data, and the notification information setting unit 230 adds notification information of "detection failed" to the detection threshold value of "0" based on the user's input operation and stores it in the storage device 40.

[0026] 8 , for example, when the position of the workpiece 60 detected by the image processing unit 210 is inside the user notification area E2, the notification information setting unit 230 may add notification information of “detection successful” to the detection threshold of the user notification area E2 based on a user input operation via the teaching operation panel 25, and store the notification information in the storage device 40. When the position of the workpiece 60 detected by the image processing unit 210 is between the user notification area E2 and the imaging range E1, the notification information setting unit 230 may add notification information including “detection unstable” and a message instructing readjustment of parameters effective for improving the detection results of the vision detection to the detection threshold of the imaging range E1 based on a user input operation via the teaching operation panel 25, and store the notification information in the storage device 40. When the position of the workpiece 60 detected by the image processing unit 210 is outside the imaging range E1, the notification information setting unit 230 may add notification information of “detection failed” to the detection threshold of a region (not shown) outside the imaging range E1 based on a user input operation via the teaching operation panel 25, and store the notification information in the storage device 40.

[0027] The determination unit 240 determines which of the multiple detection thresholds the object vision-detected by the image processing unit 210 was detected at. Specifically, the determination unit 240 reads, for example, multiple detection thresholds linked to detection parameters from the storage device 40. For example, when the detection parameter is "score," the determination unit 240 determines, based on the multiple read detection thresholds, whether the score of the detected object is equal to or greater than the detection threshold "90," equal to or greater than the detection threshold "60" but less than the detection threshold "90," or less than the detection threshold "60." Furthermore, when the detection parameter is "position," the determination unit 240 determines, based on the multiple read detection thresholds, whether the position of the detected object is inside the user notification area E2, between the user notification area E2 and the imaging range E1, or outside the imaging range E1.

[0028] The determination unit 240 may calculate the rate of change of the score in pattern matching or the like by comparing the latest score value with the most recent score value, and determine whether the calculated rate of change is equal to or greater than a predetermined detection threshold for a detection parameter. For example, even if the score value is equal to or greater than the detection threshold of 90, which indicates "detection successful," if the calculated rate of change indicates a gradual deterioration that is equal to or greater than the predetermined detection threshold for a detection parameter, the determination unit 240 may determine the detection threshold for the rate of change of the score, which is one of the detection parameters.

[0029] Alternatively, the judgment unit 240 calculated the rate of change of the score by comparing the latest score value with the most recent score value, but this is not limited to this, and the rate of change of the score may be calculated by calculating the average value of the score values ​​up to that point and comparing the latest score value with the average value.

[0030] The notification unit 250 notifies the user of the notification information set by the notification information setting unit 230 based on the determination result of the determination unit 240. Specifically, for example, if the determination result is equal to or greater than the detection threshold "90" or is inside the user notification area E2, the notification unit 250 reads from the storage device 40 notification information indicating the detection threshold "90" or "detection successful" added to the user notification area E2. The notification unit 250 notifies the user by displaying the read notification information on the display unit 21. Furthermore, for example, if the determination result is equal to or greater than the detection threshold "60" but less than the detection threshold "90," or is between the user notification area E2 and the imaging range E1, the notification unit 250 reads from the storage device 40 notification information indicating the detection threshold "60" or "detection unstable" added to the imaging range E1 and a message instructing the user to readjust parameters (teach parameters) effective for improving the vision detection result. The notification unit 250 notifies the user by displaying the read notification information on the display unit 21. 9 is a diagram showing an example of a dialog box for instructing readjustment of teaching parameters, in which a dialog box for instructing readjustment of parameters effective for improving the detection results of vision detection is displayed on the screen of a detection program (vision program) being executed by the robot control device 20.

[0031] For example, when a user presses the "OK" button in the dialog box via the teaching console 25, the robot control device 20 may display on the display unit 21 a screen for editing changeable parameters (e.g., the exposure time and resolution of the visual sensor 32) that are effective for improving the detection results of the detection parameters "score" and "position." This allows the robot control device 20 to accept and set re-teaching and changes to the changeable parameters based on user input via the teaching console 25. Note that changeable parameters are not limited to the exposure time and resolution of the visual sensor 32, but also include the detection range and imaging range of the visual sensor 32, the mounting position and angle of the visual sensor 32, the detection position, angle, and contrast of the target object, and so on. Furthermore, when a user presses the "Review Parameters" button in the dialog box via the teaching console 25, the robot control device 20 may display, for example, information required to improve the detection results when reviewing the changeable parameters and / or a list of changeable parameters that are effective for improving the detection results. FIG. 10 illustrates an example of a display screen displaying information required for improving the detection results. Fig. 10 shows materials related to solving erroneous detection. Fig. 11 shows an example of a screen listing parameters effective for improving detection results. It is preferable that the robot control device 20 displays the screen of Fig. 10 and / or the screen of Fig. 11 in response to an input operation by the user via the teaching operation panel 25.

[0032] In addition, when there are multiple detection parameters, such as "score" or "position," and the notification information is "unstable detection," the notification unit 250 may notify which detection parameter's detection threshold is causing the detection abnormality.

[0033] Furthermore, even if the determination unit 240 determines that the rate of change in the score in pattern matching or the like is equal to or greater than a preset detection threshold for the detection parameter, i.e., the score value is equal to or greater than the detection threshold of "90," indicating "detection successful," if the calculated rate of change indicates that the rate of change is gradually worsening beyond the preset detection threshold for the detection parameter, the notification unit 250 may notify the user by displaying a dialog box of notification information on the display unit 21, the dialog box including a message instructing the user to readjust parameters that are effective in improving the detection results of vision detection, before the score value becomes "unstable detection," which is equal to or greater than the detection threshold of "60" but less than "90." In this way, although the robot control device 20 requires multiple pieces of data regarding the rate of change in the score value, the user does not need to understand the data, thereby reducing the burden on the user.

[0034] <Notification Processing of Robot Controller 20> Next, the flow of the notification processing of the robot controller 20 will be described with reference to Fig. 12. Fig. 12 is a flowchart illustrating the notification processing of the robot controller 20.

[0035] In step S11 , the image processing unit 210 acquires an image captured by the visual sensor 32 .

[0036] In step S12, the image processing unit 210 detects the target object (workpiece 60) from the image acquired in step S11.

[0037] In step S13, the image processing unit 210 calculates a score indicating the similarity (degree of match) between the object detected from the image in step S12 and the teaching data, or a detection parameter for the position of the object in the image.

[0038] In step S14, the determination unit 240 determines which of the multiple detection thresholds the detection parameter calculated in step S13 was used to detect. Note that the determination unit 240 may calculate a rate of change in the value of the calculated detection parameter by comparing the latest detection parameter value with the most recent detection parameter value, and determine whether the calculated rate of change is equal to or greater than a preset detection threshold for the detection parameter.

[0039] In step S15, if the determination result is "unstable detection," the notification unit 250 notifies the user by displaying a dialog box of notification information on the display unit 21, the dialog box including a message instructing the user to readjust parameters that are effective in improving the detection result of vision detection. Note that even if the determination result in step S14 is "successful detection," if the calculated rate of change is equal to or greater than a preset detection threshold value for the detection parameter, the notification unit 250 may notify the user by displaying a dialog box of notification information on the display unit 21, the dialog box including a message instructing the user to readjust parameters that are effective in improving the detection result of vision detection.

[0040] As described above, the robot control device (image processing device) 20 according to one embodiment can prevent detection failures by prompting the user to re-teach the vision detection function and / or review parameters effective for improving vision detection results before the vision detection function fails to detect or erroneously detects an object. Furthermore, the robot control device (image processing device) 20 can lower the hurdle for the user to re-teach by notifying the user of parameter revisions effective for improving vision detection results that are expected to improve scores (detection rates) and the like, simultaneously when scores and the like decrease. This can create an environment that makes re-teaching easier. Furthermore, the robot control device (image processing device) 20 can monitor the rate of change of indicators of the vision detection function by setting a detection threshold for the rate of change. This allows the robot control device (image processing device) 20 to monitor the rate of change of the score values, even though it requires multiple data sets for the rate of change of the score values. Since the user does not need to understand these data, the burden on the user can be reduced.

[0041] <Modification 1> In the embodiment, the robot controller 20 has the function of an image processing device, but this is not limiting. An information processing device such as a computer different from the robot controller 20 may function as the image processing device.

[0042] <Modification 2> In the above-described embodiment, the notification information includes, for example, information on the detection accuracy level ("detection successful," "detection unstable," or "detection failed"), as well as a message instructing the user to readjust parameters that are effective for improving the vision detection result in the case of "detection unstable." However, the notification information is not limited to this. For example, the notification information may include text displayed on the display unit 21, an image in which the target object is color-coded, or both.

[0043] In one embodiment, each function included in the robot control device 20 can be realized by hardware, software, or a combination of these. Here, "realized by software" means that the function is realized by a computer reading and executing a program.

[0044] The program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs). The program may be provided to the computer by various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transient computer-readable media can provide the program to the computer via a wired communication path such as an electrical wire or optical fiber, or via a wireless communication path.

[0045] The step of executing the program recorded on the recording medium includes not only processes that are performed in chronological order, but also processes that are not necessarily performed in chronological order but are performed in parallel or individually. Also, the step of writing the program may be performed by cloud computing.

[0046] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0047] The following supplementary notes are further disclosed regarding the above-described embodiments and variations. (Supplementary Note 1) The image processing device (20) includes an image processing unit (210) that performs image processing on an image acquired by a visual sensor (32) and performs vision detection of an object; a detection threshold setting unit (220) that sets multiple detection thresholds for each detection parameter used in the vision detection of the object (60) in the image processing unit (210); and a determination unit (240) that determines at which of the multiple detection thresholds the object (60) vision-detected by the image processing unit (210) was detected. (Supplementary Note 2) The image processing device (20) described in Supplementary Note 1 further includes a notification information setting unit (230) that adds notification information to the multiple detection thresholds, and a notification unit (250) that notifies the notification information set by the notification information setting unit (230). (Supplementary Note 3) In the image processing device (20) described in Supplementary Note 1 or Supplementary Note 2, the notification information includes at least information regarding the level of detection accuracy, including detection success, detection instability, and detection failure. (Supplementary Note 4) In the image processing device (20) described in Supplementary Note 1 or Supplementary Note 2, the notification information includes information instructing the operator to readjust the teaching parameters. (Supplementary Note 5) In the image processing device (20) described in Supplementary Note 1 or Supplementary Note 2, a display unit (21) is provided, and the notification information includes text displayed on the display unit (21), an image in which the object (60) is color-coded, or both. (Supplementary Note 6) In the image processing device (20) described in Supplementary Note 1 or Supplementary Note 2, the multiple detection thresholds are fixed values ​​or change rates of preset detection parameters. (Supplementary Note 7) In the image processing device (20) described in Supplementary Note 1 or Supplementary Note 2, when there are multiple detection parameters, the notification unit (250) notifies which of the detection parameter's detection thresholds has caused the detection abnormality. (Appendix 8) The image processing method includes an image processing step of performing image processing on an image acquired by the visual sensor (32) and performing vision detection of an object, a detection threshold setting step of setting a plurality of detection thresholds for each detection parameter used in the vision detection of the object (60) in the image processing step, and a determination step of determining at which of the plurality of detection thresholds the object (60) vision-detected by the image processing step was detected.

[0048] REFERENCE SIGNS LIST 1 Robot system 10 Visual sensor control device 20 Robot control device 210 Image processing unit 220 Detection threshold setting unit 230 Notification information setting unit 240 Determination unit 250 Notification unit 30 Robot 31 Hand 32 Visual sensor 40 Storage device 50 Work table 60, 60a, 60b Work

Claims

1. An image processing device comprising: an image processing unit that performs image processing on an image acquired by a visual sensor and performs vision detection of an object; a detection threshold setting unit that sets multiple detection thresholds for each detection parameter used in the vision detection of the object in the image processing unit; and a determination unit that determines which of the multiple detection thresholds was used to detect the object that was vision detected by the image processing unit.

2. The image processing device according to claim 1, further comprising: a notification information setting unit that adds notification information to the plurality of detection thresholds; and a notification unit that notifies the notification information set by the notification information setting unit.

3. The image processing device according to claim 2, wherein the notification information includes information on the level of detection accuracy, including at least successful detection, unstable detection, and failed detection.

4. The image processing device according to claim 2, wherein the notification information includes information instructing the operator to readjust teaching parameters.

5. The image processing device according to claim 2, further comprising a display unit, wherein the notification information includes text displayed on the display unit, an image in which the object is color-coded, or both.

6. An image processing device according to claim 1 or claim 2, wherein the plurality of detection thresholds are fixed values ​​or change rates of preset detection parameters.

7. The image processing device according to claim 2, wherein, when there are a plurality of detection parameters, the notification unit notifies which detection parameter's detection threshold has caused the detection abnormality.

8. An image processing method comprising: an image processing step of performing image processing on an image acquired by a visual sensor and performing vision detection of an object; a detection threshold setting step of setting a plurality of detection thresholds for each detection parameter used in the vision detection of the object in the image processing step; and a determination step of determining which of the plurality of detection thresholds was used to detect the object that was vision detected by the image processing step.

Images