Image processing device

Abstract

Provided is an image processing device comprising: an image acquisition unit that acquires, from a visual sensor, image information obtained as a result of the visual sensor capturing an image within a field of view; a detection unit that performs a detection process to detect an object from the image information on the basis of information representing object features; a blob extraction unit that uses the image information as a basis for extracting a region identified as a blob; an area calculation unit that calculates the area of the object detected by the detection unit as a first area and calculates the area of the region identified as the blob as a second area; and a determination unit that determines whether or not an undetected object that was not detected in the detection process is present in the image information, on the basis of a comparison of the first and second areas.

Description

Image Processing Device

[0001] The present disclosure relates to an image processing device.

[0002] 2. Description of the Related Art A robot system is known that can detect an object using a visual sensor and perform an operation such as picking up the object.

[0003] For example, Patent Document 1 describes a robot system that includes a robot and an imaging device mounted on the robot and performs a task of transporting piled objects from one container to another. Patent Document 2 describes a handling system for transporting workpieces when grinding plate-shaped metal workpieces obtained by fusion cutting or the like.

[0004] JP 2020-21212 A JP 2007-021635 A

[0005] In a system in which a visual sensor detects objects and a robot picks them up, etc., some objects may not be detected in the detection process by the visual sensor due to various factors. When such a non-detection occurs, an image of the non-detection is saved, and the cause of the non-detection is investigated to improve the detection process. However, whether or not a non-detection has occurred is generally determined by a human eye. There is a demand for technology that can automatically and reliably determine whether or not a non-detection has occurred in the detection process.

[0006] One aspect of the present disclosure is an image processing device that includes an image acquisition unit that acquires image information obtained by the visual sensor when the visual sensor captures an image within the field of view, a detection unit that performs a detection process to detect an object from the image information based on information representing the characteristics of the object, a blob extraction unit that extracts an area identified as a blob based on the image information, an area calculation unit that calculates the area of ​​the object detected by the detection unit as a first area and calculates the area of ​​the area identified as the blob as a second area, and a determination unit that determines whether there is an undetected object in the image information that was not detected by the detection process based on a comparison between the first area and the second area.

[0007] These and other objects, features and advantages of the present invention will become more apparent from the detailed description of exemplary embodiments of the invention illustrated in the accompanying drawings.

[0008] 4A is a diagram showing the configuration of a robot system including an image processing device according to one embodiment; FIG. 4B is a diagram showing a functional block diagram relating to an image processing device and a robot teaching device according to the first embodiment; FIG. 4C is a flowchart showing undetection determination processing according to the first embodiment; FIG. 4D is an example of a captured image showing four objects; FIG. 4E is a diagram showing a state in which information indicating a detection result is added to a captured image showing four objects; FIG. 4F is a diagram showing a binarized image obtained by performing binarization processing on the captured image of FIG. 4A; FIG. 4G is a diagram explaining the comparison between the area of ​​an extracted blob and the area of ​​a detected object region; FIG. 4H is a diagram showing an example of a UI screen according to the first embodiment; FIG. 4I is a diagram showing a functional block diagram relating to an image processing device and a robot teaching device according to a second embodiment; FIG. 4J is a flowchart showing undetection determination processing according to the second embodiment; FIG. 4J is a diagram showing an example of a UI screen according to the second embodiment;

[0009] Next, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, like components or functional parts are designated by like reference numerals. The scales of these drawings have been changed appropriately to facilitate understanding. Furthermore, the embodiment shown in the drawings is one example for implementing the present invention, and the present invention is not limited to the illustrated embodiment.

[0010] As used herein, a visual sensor refers to a two-dimensional camera that captures two-dimensional images, a three-dimensional sensor that captures three-dimensional position information of an object, or a sensor that combines the functions of a two-dimensional camera and a three-dimensional sensor. A visual sensor provides image information (two-dimensional images, three-dimensional point clouds, etc.) of an object captured within its field of view. Furthermore, as used herein, a detection process refers to a process of detecting an object within an image using a technique such as pattern matching based on known feature information of the object. Furthermore, as used herein, the term "blob" refers to a mass of an undefined shape that can be extracted by applying predetermined image processing to image information.

[0011] First Embodiment Fig. 1 is a diagram showing the configuration of a robot system including an image processing device according to one embodiment. The image processing device 20 controls a visual sensor 70 and processes images captured by the visual sensor 70. As shown in Fig. 1, the robot system 100 includes a robot 10, a robot control device 50 that controls the robot 10, a teaching device 40 connected to the robot control device 50, the visual sensor 70, and the image processing device 20. The robot system 100 can, for example, detect an object 90 placed in a working area using the visual sensor 70 and handle the object 90 with a hand (not shown) mounted on the robot 10.

[0012] In this embodiment, the robot 10 is a vertical articulated robot, but other types of robots such as a parallel link robot or a dual-arm robot may be used depending on the purpose of the work. The robot 10 can perform a desired work using an end effector attached to the wrist.

[0013] The visual sensor 70 functions as a two-dimensional camera that captures grayscale images and color images. Although Fig. 1 shows an example in which the visual sensor 70 is a fixed camera fixed within the workspace, the visual sensor 70 may also be mounted on the hand of the robot 10.

[0014] The image processing device 20 stores a model pattern of the object 90 (91-94) and can execute a detection process to detect the object by pattern matching the image of the object in the captured image with the model pattern. It is assumed that the visual sensor 70 has been calibrated, and the image processing device 20 stores calibration data that defines the relative positional relationship between the visual sensor 70 and the robot 10. This allows the position on the two-dimensional image captured by the visual sensor 70 to be converted to a position on a coordinate system (e.g., a robot coordinate system) fixed to the workspace. Various factors, such as the placement of the object and the lighting of the workspace, can cause the detection unit 123 to be unable to correctly detect the object in the image containing the object. As described below, the image processing device 20 can provide a function to automatically determine whether any object has been undetected in the detection process.

[0015] In FIG. 1, the image processing device 20 is configured as a device separate from the robot control device 50 , but the function of the image processing device 20 may be incorporated into the robot control device 50 .

[0016] The image processing device 20 may have a hardware configuration as a general computer including a processor 21, memory (ROM, RAM, non-volatile memory, etc.), a storage device 22, an operation unit 23, a display unit 24, an input / output interface, a network interface, etc. (See FIG. 2 ). The image processing device 20 can be configured as a PC (personal computer) or any other type of information processing device. The display unit 24 is, for example, a liquid crystal display. The operation unit 23 can include, for example, various pointing devices such as a keyboard and a mouse.

[0017] The robot control device 50 controls the operation of the robot 10 in accordance with an operation program or commands from the teaching device 40. The robot control device 50 may have a hardware configuration as a general computer having a processor, memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, an input / output interface, a network interface, etc.

[0018] The teaching device 40 is used as an operation terminal for teaching (creating a program) the robot 10 and for performing various settings. The teaching device 40 may be a teaching operation panel, or may be configured as a tablet terminal or the like. The teaching device 40 may have a hardware configuration as a general computer having a processor, memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, a display unit 41, an input / output interface, a network interface, etc. The display unit 41 is configured as, for example, a liquid crystal display.

[0019] Fig. 2 shows a functional block diagram of the image processing device 20 and the robot control device 50. As shown in Fig. 2, the robot control device 50 includes an operation control unit 151. The operation control unit 151 controls the operation of the robot 10 in accordance with an operation program or in accordance with commands from the teaching device 40. The robot control device 50 includes a servo control unit (not shown) that executes servo control for the servo motors of each axis in accordance with commands for each axis generated by the operation control unit 151.

[0020] 2, the image processing device 20 includes a visual sensor control unit 121, an image acquisition unit 122, a detection unit 123, a binarization processing unit 124, an area calculation unit 125, and a determination unit 126. The image processing device 20 may further include a detection result storage unit 127, a setting unit 128, and a learning unit 129. As shown in FIG. 2, these functional blocks may be realized by the processor 21 executing software.

[0021] The visual sensor control unit 121 controls the operation of the visual sensor 70. For example, the visual sensor control unit 121 can receive an operation command for the visual sensor 70 from the robot control device 50 and control the visual sensor 70. The image acquisition unit 122 acquires image information obtained by the visual sensor 70 capturing an image within its field of view. In this embodiment, the image acquisition unit 122 acquires a two-dimensional image from the visual sensor 70.

[0022] The detection unit 123 can execute a detection process to detect an object in image information (here, a two-dimensional image) acquired from the visual sensor 70 based on known feature information of the object. The image processing device 20 stores, for example, model data of the object in the storage device 22. The detection unit 123 has the model data of the object and can detect the object by matching the model data in the image. For example, the detection unit 123 detects the object in the image by comparing the edge features of the model data with the edge features of the object in the captured image. This allows the detection unit 123 to identify the area in the image where the object exists.

[0023] The binarization processing unit 124 can perform binarization processing on an image. The binarization processing unit 124 can extract, as a blob, a region having a specific pixel value (for example, 1) in the binarized image. In other words, the binarization processing unit 124 functions as a blob extraction unit that can extract a region identified as a blob.

[0024] The area calculation unit 125 calculates the area of ​​the object detected by the detection unit 123 (this area will also be referred to as the first area), and can also calculate the area of ​​the region identified as a blob (this area will also be referred to as the second area). For example, the area calculation unit 125 can calculate the sum of the areas of the object regions detected by the detection unit 123 on the image as the first area. Furthermore, the area calculation unit 125 can calculate the sum of the areas of the blob regions extracted by the binarization processing unit as the second area. The area calculation unit 125 may calculate these areas based on a coordinate system set in the image.

[0025] The determination unit 126 can determine whether there is an object in the image that was not detected by the detection unit 123 (e.g., an undetected object) based on a comparison between the first area and the second area calculated by the area calculation unit 125.

[0026] The detection result saving unit 127 provides a function of saving an image and information related to the detection result in, for example, the storage device 22 when an undetected object is found. The setting unit 128 provides a function for making various settings related to the operation of the image processing device 20. The setting unit 128 can provide, for example, a UI (user interface) that accepts settings of parameters used by the binarization processing unit 124 and the determination unit 126 to execute processing. The setting unit 128 can display the UI on the display unit 24 and accept user input to the UI via the operation unit 23.

[0027] The following describes the process performed by the image processing device 20 to determine whether or not there is an object in the captured image that was not detected in the detection process performed by the detection unit 123 (hereinafter also referred to as undetection determination process).

[0028] 3 is a flowchart showing the undetection determination process. This undetection determination process is executed under the control of the processor 21 of the image processing device 20. Here, assume a situation in which four objects 91-94 are placed within the working area as shown in FIG. 1. The visual sensor 70 is placed so that the working area including the four objects 91-94 is included in its imaging range, and the undetection determination process is executed on an image captured under this situation.

[0029] First, the image processing device 20 (setting unit 128) accepts user settings of various parameters (step S1). The parameters to be set here include parameters (such as detection score) used in the detection process by the detection unit 123, as well as one or more of the following parameters related to the determination of non-detection: (1) threshold value for binarization processing, (2) target area for binarization processing on the captured image, and (3) threshold value for determining that non-detection has occurred.

[0030] The "threshold value when performing binarization processing" refers to, for example, a threshold value for determining the luminance value above which the pixel value after binarization must be set to 1 when each pixel in a captured image has a luminance value as its pixel value. The user can set a desired threshold value taking into consideration the lighting conditions in the work space, the properties of the object, and the like. A default value may be set in advance as this threshold value. Note that the setting unit 128 may automatically set the "threshold value when performing binarization processing" based on the number of bits of the pixel value (luminance value) of one pixel of the image, for example, to a value approximately half the maximum luminance value represented by this number of bits.

[0031] The "area on the captured image to be binarized" refers to an area on the captured image to be binarized by the binarization processing unit 124. By being able to specify the area to be binarized, the user can avoid, for example, a situation where a peripheral device appears as a blob in the binarized image. Furthermore, by being able to specify the area to be binarized, the load on image processing can be reduced and processing can be accelerated.

[0032] The "threshold for determining whether an undetected object exists" is a threshold used by the determination unit 126 when determining whether an undetected object exists. Here, the determination unit 126 determines that an undetected object exists when the difference between the second area and the first area is equal to or greater than this threshold. The threshold for determining that an undetected object exists may be, for example, half the area of ​​one object. Note that, if the area of ​​one object is known, the setting unit 128 may automatically set the threshold for determining that an undetected object exists to half the area of ​​one known object. From the perspective of determining the presence of an undetected object, if an object smaller than the object appears in the binarized image, the threshold for determining that an undetected object exists may be set to a relatively large value (for example, a value greater than half the area of ​​one object) so that such an object is not determined to be an undetected object.

[0033] Next, a plurality of objects are supplied to the work area of ​​the robot 10, which is also within the field of view of the visual sensor (step S2). The objects may be arranged in an aligned state in the work area as shown in FIG. 1, or may be supplied in a state in which multiple objects are arranged on a pallet (not shown). In step S2 of FIG. 2, a plurality of objects are supplied on a pallet as an example. At this stage, objects 91-94 are placed in the work area of ​​the robot 10, as shown in FIG. 1 as an example. The series of processes from step S2 to step S12 are repeatedly executed a predetermined number of times as a loop process. In other words, the process of placing objects in the work area and performing steps S3-S11 is repeatedly executed a predetermined number of times.

[0034] Next, the visual sensor 70 captures an image in response to a command from the visual sensor control unit 121 (step S3). The detection unit 123 then executes a detection process on the captured image (step S4). Here, it is assumed that the captured image is image M1, which contains four objects 91-94 as shown in FIG. 4A. It is assumed that the objects detected by the detection process by the detection unit 123 are three objects 92-94. FIG. 4B shows image M1B in which the objects 92-94 detected by the detection unit 123 are superimposed with a "+" symbol indicating that they have been detected. By looking at image M1B of this detection result, the user can understand that the objects that could be detected with the current detection parameters are objects 92-94, and that object 91 was not detected.

[0035] Next, the area calculation unit 125 calculates the sum of the areas of the regions of the objects detected by the detection unit 123 on the captured image (step S5). In this case, the area calculation unit 125 calculates the sum of the areas of the regions of the three objects 92-94 on the image M1 as the first area.

[0036] Next, the binarization processing unit 124 performs binarization processing on the captured image using the threshold value set in step S1 (step S6). Furthermore, if a "region on the captured image to be binarized" is specified, the binarization processing unit 124 performs binarization processing on the specified region on the captured image. Here, it is assumed that the "region on the captured image to be binarized" is specified for the entire captured image. In image M1, the objects 91-94 appear relatively bright, while the floor or the bottom of the pallet appears dark. By performing binarization processing on image M1, a binarized image M2, as shown in FIG. 5, can be obtained. In the example of image M2, the region containing the four objects 91-94 is extracted as the region of blob B1, which has a pixel value of 1. In this case, the region with a pixel value of 0 corresponds to the floor or the bottom of the pallet.

[0037] Next, the area calculation unit 125 calculates the sum of the areas of the regions extracted as blobs in the binarized image as the second area (step S7). In the case of image M2, the area of ​​the region of blob B1 is calculated as the second area. The determination unit 126 then determines whether or not there is an undetected object based on the difference between the second area and the first area (step S8). For example, the determination unit 126 determines that there is an undetected object if the difference between the second area and the first area is equal to or greater than the "threshold for determining that there is an undetected object" set in step S1. In this case, as schematically shown in FIG. 6 , the area of ​​the extracted blob B1 (second area) is substantially larger than the area of ​​the regions of the detected objects 92-94 (first area) by the area of ​​one object. Therefore, for example, by setting a value approximately half the area of ​​one object as the "threshold for determining that there is an undetected object," it is possible to appropriately determine whether or not there is an undetected object.

[0038] If there is an undetected object (S9: YES), the detection result storage unit 127 stores information about the image and the results of the detection process as history information, for example, in the storage device 22 (step S10). In this case, the detection results may include a history image such as image M1B, parameters used in the detection process, and the results of the detection process (score value, etc.) for the undetected object. The stored history information can be used to analyze the cause of the undetection and to improve the detection process. If there is no undetected object (S9: NO), the process proceeds to step S11.

[0039] Next, the robot 10 performs a workpiece transfer operation in accordance with the operation program, picking up the detected object and placing it in another location (step S11). Next, the pallet on which the object was placed is ejected (step S12). The series of processes (loop processes) from steps S2 to S12 are repeated for the next object to be supplied. When the loop process has been executed a predetermined number of times, this process ends.

[0040] The above-described non-detection determination process makes it possible to automatically and reliably determine whether or not a non-detection has occurred in the detection process by the detection unit 123. Furthermore, if a non-detection has occurred, history information including a history image and the like is automatically saved. Therefore, the user does not need to visually determine whether or not a non-detection has occurred. Furthermore, it becomes possible to efficiently save only the history information when a non-detection has occurred. The history information accumulated by the non-detection determination process can be used to analyze the cause of the non-detection and improve the detection process.

[0041] The non-detection determination process can be applied not only to the case where the objects are arranged in a line as shown in Figure 1, but also to situations where the objects are randomly placed on a pallet, for example.The non-detection determination process can also be applied to situations where the number of objects supplied is not the same each time.

[0042] 7 shows an example of a UI (user interface) screen 300 provided by the setting unit 128. The UI screen 300 is configured as a user interface for accepting settings of various parameters in step S1 of the non-detection determination process and for presenting an image representing the detection result. The UI screen 300 includes an image display area 310 and a parameter setting area 320. The UI screen 300 may also include a program display area 330 for displaying program commands related to image capture and image processing.

[0043] The captured image or the binarized image is displayed in the image display area 310. Information indicating the detection result may be added to the image displayed in the image display area 310.

[0044] The parameter setting area 320 may include at least one of the following: (1) an input field 321 for inputting a threshold value for binarization processing; (2) an input field 322 for specifying an area on the captured image to be binarized; and (3) an input field 323 for inputting a threshold value for determining whether an undetected portion exists. The user can execute the undetection determination process by inputting these parameters and performing a predetermined operation. When the undetection determination process is executed using the input parameters, a binarized image may be displayed in the image display area 310 as part of the processing results.

[0045] In the input field 321, a numerical value can be input as a threshold value for use in binarization processing. In the input field 322, for example, the coordinates (pixel values ​​in X and Y coordinates) of the upper left corner and the coordinates (pixel values ​​in X and Y coordinates) of the lower right corner of a rectangular area can be input as the area to be subjected to binarization processing. Note that, although an example of a setting method for specifying the area to be subjected to binarization processing using coordinate values ​​has been shown here, a graphical user interface can also be provided that allows the user to graphically specify the target area on the image using a pointing device. In the input field 323, a numerical value can be input as a threshold value for determining that there is an undetected area.

[0046] Via the UI screen 300, the user can make the following settings, for example. For example, suppose the range of brightness values ​​of the captured image is 0-255. In this case, the user sets, for example, 231 as the "threshold value for performing binarization processing" (brightness threshold) so that only the object has a pixel value of "1" after binarization. For example, suppose the size of one object is 10 pixels by 30 pixels, and the area is 300 pixels. In this case, the user may set 150 pixels, half of the 300-pixel area of ​​one object, as the "threshold value for determining that there is an undetected object."

[0047] 7 shows an example in which image M3 after binarization processing is displayed in the image display area 310. In image M3, six objects are extracted as blobs in the binarized image when binarization processing is performed using the above setting example. Image M3 also shows an example in which, as information indicating the detection results, a "+" symbol is added to four of the six objects, indicating that they were detected. In other words, image M3 shows that six objects are extracted as blobs (bright areas), four of which were detected by the detection processing, and two of which were not detected.

[0048] The UI screen 300 may further display information regarding the results of the undetection determination process. The information regarding the results of the undetection determination process may include, for example, whether or not any objects were undetected and the number of undetected objects. For example, the determination unit 126 can determine the number of undetected objects by using known information regarding the area of ​​each object to calculate how many times the difference between the second area and the first area corresponds to the area of ​​each object. Note that the area of ​​each object may be specified as one of the parameters in the parameter setting area 320.

[0049] In this way, the user can set parameters via the UI screen 300 and efficiently check the image after binarization processing and the results of the non-detection determination processing.

[0050] The above describes an example in which the setting unit 128 accepts user settings for the threshold value used in the binarization process and the threshold value used to determine whether an undetected object exists, and these user-set parameters are used in the undetection determination process. The image processing device 20 may further include a function for automatically setting at least one of these parameters. Here, a function for acquiring appropriate parameter values ​​through learning will be described as the function for automatically setting at least one parameter. As shown in FIG. 2 , the image processing device 20 may include a learning unit 129 that acquires appropriate parameter values, including the threshold value used in the binarization process and the threshold value used to determine whether an undetected object exists, through learning. The function of the learning unit 129 will be described below.

[0051] The learning unit 129 performs learning to obtain optimal values ​​of parameters by machine learning, for example. Here, an example is shown in which optimal values ​​of parameters are learned by supervised learning. A deep learning technique may also be adopted in the learning.

[0052] It is believed that the brightness and contrast of an image of an object may change depending on various factors, such as the brightness of the work space, even if the image is the same object. Therefore, it is believed that there is a correlation between the captured image and the parameters (thresholds used in binarization processing and thresholds used to determine whether an undetected object exists) used when the image is successfully determined to contain an undetected object. The learning unit 129 uses the captured image as input data and accumulates training data (achievement data) in which the parameter values ​​used when the image is successfully determined to contain an undetected object are used as correct answer data. The training data may be stored in, for example, the storage device 22.

[0053] The learning unit 129 performs learning using the accumulated training data. For example, the learning unit 129 uses a captured image as input data and causes an estimator to learn training data in which parameters obtained when the image is successfully determined to be undetected are used as correct answer data. Here, the estimator is configured, for example, by an NN (neural network) or a CNN (convolutional neural network). This allows the learning unit 129 to construct a learning model.

[0054] By inputting any captured image into an estimator (learning model), it is possible to obtain parameters that are estimated to be appropriate for that image.

[0055] As described above, according to the first embodiment, it is possible to automatically and reliably determine whether or not there is an undetected object in the captured image that has not been detected by the detection process.

[0056] Second Embodiment An image processing device 20A (see FIG. 8) according to the second embodiment will now be described. The image processing device 20A according to the second embodiment is configured to determine whether or not there is an undetected object using image information representing a three-dimensional point cloud of the object. The equipment configuration of a robot system 100A including the image processing device 20A according to the second embodiment is the same as that shown in FIG. 1.

[0057] FIG. 8 is a functional block diagram of an image processing device 20A according to a second embodiment. In FIG. 8, functional blocks equivalent to those of the image processing device 20 according to the first embodiment are designated by the same reference numerals, and their descriptions are omitted or simplified. As shown in FIG. 8, the image processing device 20A includes a visual sensor control unit 121, an image acquisition unit 122, a detection unit 123, a plane calculation unit 131, an area calculation unit 125A, and a determination unit 126. The image processing device 20A may further include a detection result storage unit 127, a setting unit 128A, and a learning unit 129A. As shown in FIG. 8, these functional blocks may be implemented by the processor 21 executing software.

[0058] In this embodiment, the visual sensor 70A functions as a three-dimensional sensor that can acquire a three-dimensional point cloud representing the three-dimensional position information of an object to be imaged, in addition to capturing two-dimensional images. As the three-dimensional sensor, for example, a time-of-flight (TOF) camera that captures distance images using a time-of-flight method or a stereo camera including two cameras can be used.

[0059] The image acquisition unit 122 acquires image information including a two-dimensional image and a three-dimensional point cloud of an imaged object from the visual sensor 70A.

[0060] The plane calculation unit 131 can provide a function for extracting a group of points within a specific height range from a three-dimensional point cloud as a plane. Because a three-dimensional point cloud includes three-dimensional position information for each point, a group of points within a specific height range can be extracted as a plane. In other words, the plane calculation unit 131 functions as a blob extraction unit that can extract an area identified as a blob within image information. The "specific height range" is set, for example, by the user.

[0061] Area calculation unit 125A calculates, as a first area, the sum of the areas of the object regions detected on the image by detection unit 123. Area calculation unit 125A also calculates, as a second area, the sum of the areas of the planes calculated by plane calculation unit 131. Then, determination unit 126 can determine whether or not there was any undetected object in the detection by the detection unit by comparing the first area with the second area.

[0062] The undetection determination process executed by the image processing device 20A according to the second embodiment will be described below. Fig. 9 is a flowchart of the undetection determination process according to the second embodiment. The undetection determination process is executed under the control of the processor 21. In Fig. 9, steps that are the same as steps in the undetection determination process according to the first embodiment shown in Fig. 3 are assigned the same step numbers, and their descriptions will be omitted or simplified.

[0063] First, the setting unit 128A accepts user settings for parameters to be used in the undetected object determination process (step S1). The parameters to be set here include parameters (such as the detection score) used in the detection process by the detection unit 123, as well as one or more of the following parameters related to the undetected object determination: (1) the height range of the three-dimensional point cloud for which the area is to be calculated; (2) the "area in the captured image for which a plane is to be calculated"; and (3) a threshold value for determining that an undetected object exists.

[0064] The "height range of the 3D point cloud for which the area is calculated" is a parameter that specifies the height range for extracting a point cloud representing a plane from the 3D point cloud. For example, a user can specify an appropriate height range for extracting the area of ​​the object as a blob based on the size (height) of the object. For example, a certain range that includes the height of the object may be set as the "height range of the 3D point cloud for which the area is calculated." The setting unit 128A may automatically set the "height range of the 3D point cloud for which the area is calculated" based on the model data of the object.

[0065] The "area on the captured image for which a plane is to be found" is the area on the image for which a plane is to be found from the 3D point cloud. By being able to specify the "area on the captured image for which a plane is to be found," the user can avoid, for example, a situation in which a peripheral device is extracted as a blob during the plane finding process. Furthermore, by being able to specify the "area on the captured image for which a plane is to be found," the load on image processing can be reduced and processing can be speeded up.

[0066] Next, a plurality of objects are supplied to the work area of ​​the robot 10 and within the field of view of the visual sensor (step S2). The objects may be arranged in an aligned state in the work area as shown in FIG. 1, or may be supplied in a state where multiple objects are arranged on a pallet (not shown). In step S2 of FIG. 9, it is described as an example that multiple objects are supplied on a pallet. At this stage, objects 91-94 are placed in the work area of ​​the robot 10 as shown in FIG. 1, for example.

[0067] Next, the visual sensor 70A captures an image of the field of view in response to a command from the visual sensor control unit 121, and acquires the image and a three-dimensional point cloud (step S3a). The detection unit 123 then performs a detection process on the captured image based on the object model data (step S4). The area calculation unit 125A calculates the sum of the areas of the object regions detected by the detection unit 123 in the captured image as a first area (step S5).

[0068] Next, the plane calculation unit 131 calculates a plane within the set height range based on the "height range of the 3D point cloud for which the area is to be calculated" specified by the user in step S1 (step S6a). For example, the plane calculation unit 131 extracts a point cloud within the specified height range from the 3D point cloud as a plane. If a "target area in the captured image for which a plane is to be calculated" is specified, the plane calculation unit 131 extracts a plane within the specified area from the image represented by the 3D point cloud.

[0069] Next, the area calculation unit 125A calculates the sum of the areas of the regions identified as planes in step S6a as the second area (step S7a). The determination unit 126 then determines whether there is an undetected object based on the difference between the second area and the first area (step S8). If there is an undetected object (S9: YES), the detection result storage unit 127 stores the history image and the detection results as history information in, for example, the storage device 22 (step S10).

[0070] Next, the robot 10 performs a workpiece transfer operation in accordance with the operation program, picking up the detected object and placing it in another location (step S11). Next, the pallet on which the object was placed is ejected (step S12). The series of processes (loop processes) from steps S2 to S11 are repeated for the next object to be supplied. When the loop processes have been executed a predetermined number of times, this process ends.

[0071] Even when the undetection determination process according to this embodiment is performed on objects 91-94 as shown in FIG. 1, a captured image M1 as shown in FIG. 4A is obtained, and an image M1B of the detection result as shown in FIG. 4B is also obtained. Furthermore, by performing a process to obtain a plane on the three-dimensional point cloud, the same area as the area shown as blob B1 on image M2 in FIG. 5 can be extracted as a plane. Therefore, even in the undetection determination process according to this embodiment, it is possible to automatically and reliably determine whether or not there was an undetected object in the detection process by the detection unit 123. Furthermore, if there was an undetected object, history information including a history image and the like is automatically saved. In other words, the undetection determination process according to this embodiment can also achieve the same effect as the undetection determination process according to the first embodiment.

[0072] In the second embodiment, the setting unit 128A can also provide a UI screen with the same functions as the UI screen 300 in the first embodiment. Fig. 10 illustrates a UI screen 300A provided by the setting unit 128A. In Fig. 10A, parts with the same functions as those in the UI screen shown in Fig. 7 are denoted by the same reference numerals, and their descriptions will be omitted or simplified.

[0073] As shown in FIG. 10 , the UI screen 300A includes an image display area 310, a parameter setting area 320A, and a program display area 330. In this embodiment, the parameter setting area 320A may include at least one of the following: (1) an input field 324 for specifying the “height range of the 3D point cloud for which the area is to be calculated”; (2) an input field 325 for specifying the “target area in the captured image for which a plane is to be calculated”; and (3) an input field 323 for inputting a threshold value for determining whether an undetected object exists. The user can execute the undetection determination process by inputting these parameters and performing a predetermined operation. When the determination process is executed using the input parameters, an image (image M3A in which planes are extracted as blobs) serving as the processing result may be displayed in the image display area 310.

[0074] In input field 324, a numerical value for the "height range of the 3D point cloud for which the area is to be calculated" can be input. In input field 325, the coordinates (pixel values ​​in X and Y coordinates) of the upper left corner and the coordinates (pixel values ​​in X and Y coordinates) of the lower right corner of a rectangular area can be input as the area from which a plane is to be extracted. Note that, although an example of a setting method for specifying the area from which a plane is to be extracted using coordinate values ​​has been shown here, a graphical user interface that allows the target area to be graphically specified on an image using a pointing device may also be provided. In input field 323, a numerical value can be input as a threshold for determining whether there is an undetected object.

[0075] In this embodiment, an image M3A representing a blob extracted by finding a plane from the 3D point cloud can be displayed in the image display area 310. The white parts in image M3A are parts extracted from the 3D point cloud as areas (blobs) representing planes within a set height range.

[0076] The UI screen 300A may also display information about the results of the undetection determination process. The information about the results of the undetection determination process may include, for example, whether or not any objects were undetected and the number of undetected objects. For example, the determination unit 126 can determine the number of undetected objects by using known information about the area of ​​each object to calculate how many times the difference between the second area and the first area corresponds to the area of ​​each object. Note that the area of ​​each object may be specified as one of the parameters in the parameter setting area 320A.

[0077] In this way, the user can set parameters via the UI screen 300A and efficiently check the image after plane extraction and the results of the undetection determination process.

[0078] In this embodiment, the learning unit 129A performs learning using a method similar to that of the learning unit 129 in the first embodiment. The learning unit 129A uses a captured image as input data and performs learning using teacher data in which parameter values ​​("height range of the three-dimensional point cloud for calculating the area" and "threshold value for determining that an undetected object exists") obtained when the unit 129A has successfully determined that an undetected object exists in the image are used as correct answer data. By inputting any captured image into an estimator (learning model), it is possible to obtain parameters ("height range of the three-dimensional point cloud for calculating the area" and "threshold value for determining that an undetected object exists") that are estimated to be appropriate for that image.

[0079] In this way, in the second embodiment as well, it is possible to automatically and reliably determine whether or not there is an undetected object in the captured image that has not been detected in the detection process.

[0080] In the above-described embodiment, an example is shown in which a "height range of the three-dimensional point cloud for which the area is calculated" is specified as a parameter for extracting a plane from the three-dimensional point cloud. However, a "height of the three-dimensional point cloud for which the area is calculated" may also be specified as a parameter for extracting a plane from the three-dimensional point cloud. In this case, the plane calculation unit 131 may extract, as a plane, a point cloud having a height substantially equal to the specified "height of the three-dimensional point cloud for which the area is calculated." Alternatively, the plane calculation unit 131 may extract, as a plane, a point cloud that is within a predetermined threshold range from the value of the "height of the three-dimensional point cloud for which the area is calculated."

[0081] The undetected object determination process according to each of the above-described embodiments is a process that can determine whether or not an object that is not subject to the detection process exists within the work area (the field of view of the visual sensor) to which the object is supplied. For example, by setting the "threshold value for determining that an undetected object exists" to a relatively small value (for example, a value sufficiently smaller than the area of ​​a single object), it can also determine whether or not a relatively small unidentified object exists within the captured image (i.e., within the work area).

[0082] The above-described embodiments have been configuration examples in which the image processing device is disposed as a device separate from the robot control device in the robot system. There may also be a configuration example in which the functions of the image processing device are integrated into the robot control device. In this case, the UI screen 300 or the UI screen 300A may be provided on the display unit 41 of the teaching device 40. Alternatively, the functions of the image processing device may be integrated into the teaching device 40.

[0083] The functional blocks of the image processing device shown in Figures 2 and 8 may be realized by the processor of the image processing device executing various software stored in a storage device, or may be realized by a configuration mainly based on hardware such as an ASIC (Application Specific Integrated Circuit).

[0084] The programs that execute various processes such as the non-detection determination process in the above-described embodiments can be recorded on various computer-readable recording media (e.g., semiconductor memories such as ROM, EEPROM, and flash memory, magnetic recording media, and optical disks such as CD-ROM and DVD-ROM).

[0085] As described above, according to each embodiment, it is possible to automatically and reliably determine whether or not there is an undetected object in a captured image that has not been detected by the detection process.

[0086] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0087] The following supplementary notes are further provided regarding the above-described embodiment and modified examples. (Supplementary Note 1) An image processing device (20, 20A) including: an image acquisition unit (122) that acquires image information obtained by a visual sensor (70, 70A) capturing an image within a field of view from the visual sensor; a detection unit (123) that performs a detection process to detect an object from the image information based on information representing the object's characteristics; a blob extraction unit (124, 131) that extracts an area identified as a blob based on the image information; an area calculation unit (125, 125A) that calculates an area of ​​the object detected by the detection unit (123) as a first area and calculates an area of ​​the area identified as the blob as a second area; and a determination unit (126) that determines whether or not there is an undetected object in the image information that was not detected by the detection process based on a comparison between the first area and the second area. (Supplementary Note 2) The image processing device (20, 20A) according to Supplementary Note 1, wherein the area calculation unit (125, 125A) calculates a sum of areas of the objects detected by the detection unit 123 as the first area, and calculates a sum of areas of regions identified as the blobs as the second area. (Supplementary Note 3) The image processing device (20) according to Supplementary Note 1 or 2, wherein the image information includes a two-dimensional image, the blob extraction unit (124) includes a binarization processing unit (124) that performs binarization processing on the two-dimensional image, the binarization processing unit (124) extracts regions having a specific pixel value in the image after the binarization processing as the regions identified as the blobs, and the area calculation unit (125) calculates a sum of areas of the regions having the specific pixel value in the image after the binarization processing as the second area. (Supplementary Note 4) The image processing device (20) according to Supplementary Note 3, further comprising a setting unit (128) that accepts an operation to specify at least one of a threshold value used when performing the binarization process or an area in the two-dimensional image to be subjected to the binarization process. (Supplementary Note 5) The image processing device (20A) according to Supplementary Note 1 or 2, wherein the image information includes a three-dimensional point cloud, the blob extraction unit (131) includes a plane calculation unit (131) that extracts, from the three-dimensional point cloud, a plane that exists at a specific height or in a specific height range as an area identified as the blob, and the area calculation unit (125A) calculates a sum of areas of the obtained planes as the second area.(Supplementary Note 6) The image processing device (20A) according to Supplementary Note 5, further comprising a setting unit (128A) that accepts an operation to specify at least one of the specific height, the specific height range, or a region in the three-dimensional point cloud for which the plane is to be obtained. (Supplementary Note 7) The image processing device (20, 20A) according to any one of Supplements 1 to 6, wherein the determination unit (126) determines that there is an undetected object when a difference between the second area and the first area is equal to or greater than a predetermined threshold. (Supplementary Note 8) The image processing device (20, 20A) according to any one of Supplements 1 to 7, wherein the determination unit (126) determines how many undetected objects are present in the image information that were not detected by the detection process, based on information representing an area of ​​one of the objects. (Supplementary Note 9) The image processing device (20, 20A) according to any one of Supplementary Notes 1 to 8, further comprising a detection result storage unit (127) that stores the image information and information related to a result of the detection process when the determination unit (126) determines that there is an undetected object. (Supplementary Note 10) The image processing device (20, 20A) according to Supplementary Note 1, 2, 3, or 5, further comprising a setting unit (128, 128A) that accepts an operation to specify a determination threshold value to be used by the determination unit (126) for determination, and the determination unit (126) determines that there is an undetected object when the difference between the second area and the first area is equal to or greater than the specified determination threshold value. (Supplementary Note 11) The image processing device (20) described in Supplementary Note 3 further comprises a learning unit (129) that learns, as training data, actual data relating to the image information and at least one of the threshold values ​​used by the binarization processing unit to perform the binarization processing or the judgment threshold values ​​used by the judgment unit to perform judgment, and provides an estimated value of at least one of the threshold values ​​used to perform the binarization processing or the judgment threshold values ​​to be applied to any input image information.

[0088] 10 Robot 20, 20A Image processing device 21 Processor 22 Storage device 23 Operation unit 24 Display unit 40 Teaching device 41 Display unit 50 Robot control device 70, 70A Visual sensor 100, 100A Robot system 90, 91-94 Object 121 Visual sensor control unit 122 Image acquisition unit 123 Detection unit 124 Binarization processing unit 125, 125A Area calculation unit 126 Determination unit 127 Detection result storage unit 128, 128A Setting unit 129, 129A Learning unit 151 Operation control unit 300, 300A UI screen 310 Image display area 320, 320A Parameter setting area 330 Program display area 321-325 Input field

Claims

1. an image acquisition unit that acquires image information from the visual sensor, the image information being acquired by the visual sensor capturing an image within the field of view; a detection unit that performs a detection process to detect the object from the image information based on information representing the characteristics of the object; a blob extraction unit that extracts an area identified as a blob based on the image information; an area calculation unit that calculates an area of ​​the object detected by the detection unit as a first area and calculates an area of ​​a region identified as the blob as a second area; a determination unit that determines whether or not there is an undetected object in the image information that was not detected in the detection process based on a comparison between the first area and the second area; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the area calculation unit calculates a sum of areas of the objects detected by the detection unit as the first area, and calculates a sum of areas of the regions identified as the blobs as the second area.

3. the image information includes a two-dimensional image; the blob extraction unit includes a binarization processing unit that performs binarization processing on the two-dimensional image, the binarization processing unit extracts an area having a specific pixel value in the image after the binarization processing as an area identified as the blob; The image processing device according to claim 1 , wherein the area calculation unit calculates, as the second area, a sum of areas of regions having the specific pixel value in the image after the binarization process.

4. The image processing device according to claim 3 , further comprising a setting unit that accepts an operation to specify at least one of a threshold value used when performing the binarization process and an area in the two-dimensional image that is to be subjected to the binarization process.

5. the image information includes a three-dimensional point cloud; the blob extraction unit includes a plane calculation unit that extracts, from the three-dimensional point cloud, a plane that exists at a specific height or within a specific height range as a region identified as the blob; The image processing device according to claim 1 , wherein the area calculation unit calculates a sum of the areas of the planes thus obtained as the second area.

6. The image processing device according to claim 5 , further comprising a setting unit that accepts an operation to specify at least one of the specific height, the specific height range, or a target area in the three-dimensional point cloud for which the plane is to be obtained.

7. The image processing device according to claim 1 , wherein the determination unit determines that the undetected object exists when a difference between the second area and the first area is equal to or greater than a predetermined threshold value.

8. The image processing device according to claim 1 , wherein the determination unit determines how many undetected objects are present in the image information, based on information representing an area of ​​each of the objects.

9. The image processing device according to claim 1 , further comprising a detection result storage unit that stores the image information and information related to a result of the detection process when the determination unit determines that there is an undetected object.

10. a setting unit that accepts an operation to specify a judgment threshold value used by the judgment unit for judgment; The image processing device according to claim 1 , wherein the determination unit determines that the undetected object exists when the difference between the second area and the first area is equal to or greater than a specified threshold value for the determination.

11. 4. The image processing device according to claim 3, further comprising a learning unit that learns, as training data, actual data relating to the image information and at least one of the threshold values ​​used by the binarization processing unit to perform the binarization processing or the judgment threshold values ​​used by the judgment unit to perform judgment, and provides an estimated value of at least one of the threshold values ​​used to perform the binarization processing or the judgment threshold values ​​to be applied to any input image information.