Image processing apparatus

Abstract

This image processing device is provided with: an image acquisition unit that acquires, from a vision sensor, image information obtained by capturing an image within the field of view of the vision sensor; a detection unit that performs a detection process for detecting the object from the image information on the basis of information indicating a feature of the object; a mass extraction unit that extracts a region identified as a mass on the basis of the image information; an area calculation unit that calculates, as a first area, the area of the object detected by the detection unit, and that calculates, as a second area, the area of the region identified as a mass; and a determination unit that determines, on the basis of a comparison between the first area and the second area, whether or not there is an undetected object that is not detected in the detection process in the image information.

Description

Technical Field

[0001] The present disclosure relates to an image processing device. Background Art

[0002] There is known a robot system that can detect an object using a visual sensor and perform actions such as transporting the object.

[0003] For example, Patent Document 1 describes a robot system comprising a robot and an imaging device mounted on the robot, which is used to transfer stacked objects from one container to another. Patent Document 2 describes a handling system for transferring workpieces during grinding of sheet metal workpieces obtained by fusing or the like.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2020-21212

[0007] Patent Document 2: Japanese Patent Application Laid-Open No. 2007-021635 Summary of the Invention

[0008] Problems to be solved by the invention

[0009] In systems that detect objects using visual sensors and remove them using robots, for example, the visual sensors sometimes fail to detect some objects during the detection process due to various reasons. When such failures occur, images of the objects are saved, and the cause of the failure is investigated to improve the detection process. However, determining whether a failure has occurred is typically done by humans through visual observation. A technology that can automatically and reliably determine whether a failure has occurred during the detection process is desired.

[0010] Means for solving problems

[0011] One embodiment of the present disclosure is an image processing device comprising: an image acquisition unit that acquires image information obtained from a visual sensor within a field of view captured by the visual sensor; a detection unit that performs detection processing for detecting an object from the image information based on information representing characteristics of the object; a blob extraction unit that extracts an area determined to be a blob based on the image information; an area calculation unit that calculates the area of ​​the object detected by the detection unit as a first area and calculates the area of ​​the area determined to be the blob as a second area; and a determination unit that determines whether there is an undetected object in the image information that was not detected in the detection processing based on a comparison between the first area and the second area.

[0012] These and other objects, features and advantages of the present invention will become more apparent from the detailed description of typical embodiments of the present invention as shown in the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a diagram showing the configuration of a robot system including an image processing device according to one embodiment.

[0014] Figure 2 This is a diagram showing a functional block diagram related to the image processing device and the robot teaching device according to the first embodiment.

[0015] Figure 3 This is a flowchart showing the non-detection determination process according to the first embodiment.

[0016] Figure 4A This is an example of an image in which four objects are captured.

[0017] Figure 4B This figure shows a state where information indicating detection results is added to captured images of four objects.

[0018] Figure 5 Yes Figure 4A FIG. 1 is a diagram showing a binarized image after the captured image is binarized.

[0019] Figure 6 This is a diagram illustrating the comparison between the area of ​​the extracted cluster and the area of ​​the region of the detected object.

[0020] Figure 7 It is a diagram showing an example of a UI screen according to the first embodiment.

[0021] Figure 8 This is a diagram showing a functional block diagram related to the image processing device and the robot teaching device according to the second embodiment.

[0022] Figure 9 This is a flowchart showing the non-detection determination process according to the second embodiment.

[0023] Figure 10 It is a diagram showing an example of a UI screen according to the second embodiment. DETAILED DESCRIPTION

[0024] Next, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the accompanying drawings, identical structural components or functional components are denoted by the same reference symbols. For ease of understanding, the scales of these drawings have been appropriately altered. The embodiments shown in the accompanying drawings are merely examples for implementing the present invention, and the present invention is not limited to the illustrated embodiments.

[0025] In this specification, a visual sensor can be a two-dimensional camera that acquires a two-dimensional image, a three-dimensional sensor that acquires three-dimensional position information of an object, or both. A visual sensor provides image information (such as a two-dimensional image or a three-dimensional point group) of an object captured within its field of view. Furthermore, in this specification, detection processing refers to the process of detecting an object within an image using methods such as pattern matching based on known feature information of the object. Furthermore, in this specification, the term "block" refers to an area of ​​irregular shape that can be extracted by applying predetermined image processing to image information.

[0026] First embodiment

[0027] Figure 1 2 is a diagram showing the structure of a robot system including an image processing device according to one embodiment. The image processing device 20 controls the visual sensor 70 and has the function of processing the image captured by the visual sensor 70. Figure 1 As shown, the robot system 100 includes a robot 10, a robot control device 50 for controlling the robot 10, a teaching device 40 connected to the robot control device 50, a vision sensor 70, and an image processing device 20. The robot system 100 can detect an object 90 placed in a work area using the vision sensor 70, and transport the object 90 using a manipulator (not shown) mounted on the robot 10.

[0028] The robot 10 is a vertical articulated robot in this embodiment, but other types of robots such as a parallel link robot and a dual-arm robot may be used depending on the purpose of the work. The robot 10 can perform desired work using an end effector attached to its wrist.

[0029] The visual sensor 70 has the function of a two-dimensional camera for taking shaded images and color images. Figure 1 , an example is shown in which the vision sensor 70 is a fixed camera fixed in the work space, but the vision sensor 70 may be mounted on the front end of the hand of the robot 10 .

[0030] The image processing device 20 maintains a model pattern of the object 90 (91-94) and can perform detection processing to detect the object by matching the image of the object in the captured image with the pattern of the model pattern. In addition, assuming that the visual sensor 70 has been calibrated, the image processing device 20 maintains calibration data that defines the relative positional relationship between the visual sensor 70 and the robot 10. Thus, the position on the two-dimensional image captured by the visual sensor 70 can be transformed into a position on a coordinate system (robot coordinate system, etc.) fixed to the workspace. Due to various factors such as the configuration state of the object, the lighting of the workspace, etc., there may be a situation where the detection unit 123 cannot correctly detect the object in the image captured of the object. As described later, the image processing device 20 can provide a function of automatically determining whether there is an undetected object in the detection of the object based on the detection processing.

[0031] exist Figure 1 In the embodiment, the image processing device 20 is configured as a separate device from the robot control device 50 , but the function of the image processing device 20 may be incorporated into the robot control device 50 .

[0032] The image processing device 20 may have a hardware structure as a general computer, which includes a processor 21, a memory (ROM, RAM, non-volatile memory, etc.), a storage device 22, an operation unit 23, a display unit 24, an input / output interface, a network interface, etc. (see Figure 2 The image processing device 20 can be formed by various information processing devices such as a PC (personal computer). The display unit 24 is, for example, a liquid crystal display. The operation unit 23 can include various pointing devices such as a keyboard and a mouse.

[0033] The robot control device 50 controls the operation of the robot 10 according to the operation program or the instructions from the teaching device 40. The robot control device 50 may have a hardware structure similar to that of a general computer including a processor, memory (ROM, RAM, nonvolatile memory, etc.), a storage device, an operation unit, an input / output interface, a network interface, and the like.

[0034] The teaching pendant 40 is used as an operating terminal for teaching (program generation) and performing various settings for the robot 10. The teaching pendant 40 may be a teaching operation panel or a tablet terminal. The teaching pendant 40 may have the hardware structure of a general computer, including a processor, memory (ROM, RAM, nonvolatile memory, etc.), a storage device, an operating unit, a display unit 41, input / output interfaces, a network interface, and the like. The display unit 41 may be, for example, a liquid crystal display.

[0035] Figure 2: is a functional block diagram related to the image processing device 20 and the robot control device 50. Figure 2 As shown, the robot control device 50 includes a motion control unit 151. The motion control unit 151 controls the motion of the robot 10 according to the motion program or the instructions from the teaching device 40. The robot control device 50 includes a servo control unit (not shown) that performs servo control of the servo motor for each axis according to the instructions for each axis generated by the motion control unit 151.

[0036] like Figure 2 As shown in FIG. 1 , the image processing device 20 includes a visual sensor control unit 121, an image acquisition unit 122, a detection unit 123, a binarization unit 124, an area calculation unit 125, and a determination unit 126. The image processing device 20 may further include a detection result storage unit 127, a setting unit 128, and a learning unit 129. Figure 2 As shown, these functional blocks can also be implemented by the processor 21 executing software.

[0037] The visual sensor control unit 121 controls the operation of the visual sensor 70. For example, the visual sensor control unit 121 can receive operation commands for the visual sensor 70 from the robot control device 50 and control the visual sensor 70. The image acquisition unit 122 acquires image information obtained by the visual sensor 70 within its field of view. In this embodiment, the image acquisition unit 122 acquires a two-dimensional image from the visual sensor 70.

[0038] The detection unit 123 can perform detection processing to detect an object within the image information (here, a two-dimensional image) obtained from the visual sensor 70 based on known feature information of the object. The image processing device 20 stores model data of the object in the storage device 22, for example. The detection unit 123 has the model data of the object and can use this model data to detect the object within the image by matching. For example, the detection unit 123 compares the edge features of the model data with the edge features of the object in the captured image to detect the object in the image. In this way, the detection unit 123 can determine the area in the image where the object exists.

[0039] The binarization unit 124 can perform binarization on an image. The binarization unit 124 can extract regions with specific pixel values ​​(e.g., 1) from the binarized image as clusters. In other words, the binarization unit 124 functions as a cluster extraction unit capable of extracting regions identified as clusters.

[0040] The area calculation unit 125 can calculate the area of ​​the object detected by the detection unit 123 (this area is also referred to as the first area), and can also calculate the area of ​​the region identified as a mass (this area is also referred to as the second area). As an example, the area calculation unit 125 can calculate the sum of the areas of the regions of the object detected by the detection unit 123 on the image as the first area. Furthermore, the area calculation unit 125 can calculate the sum of the areas of the mass regions extracted by the binarization processing unit as the second area. The area calculation unit 125 can also calculate these areas based on a coordinate system set in the image.

[0041] The determination unit 126 can determine whether an object not detected by the detection unit 123 (for example, an undetected target object) exists in the image based on a comparison between the first area and the second area calculated by the area calculation unit 125 .

[0042] The detection result storage unit 127 provides a function for storing images and information related to the detection results in, for example, the storage device 22 when there are undetected objects. The setting unit 128 provides a function for making various settings related to the operation of the image processing device 20. For example, the setting unit 128 can provide a UI (user interface) that accepts settings for parameters used by the binarization processing unit 124 and the determination unit 126 for processing. The setting unit 128 can display the UI on the display unit 24 and accept user input to the UI via the operation unit 23.

[0043] Hereinafter, a process for determining whether an object that has not been detected by the detection unit 123 exists in a captured image (hereinafter also referred to as non-detection determination process) performed by the image processing device 20 will be described.

[0044] Figure 3 1 is a flowchart showing the non-detection determination process. The non-detection determination process is executed under the control of the processor 21 of the image processing device 20. Figure 1 The visual sensor 70 is arranged so that the work area including the four objects 91 to 94 is included in the imaging range, and performs non-detection determination processing on the image captured in this situation.

[0045] First, the image processing device 20 (setting unit 128) accepts user settings for various parameters (step S1). The parameters set here include parameters used in the detection process of the detection unit 123 (such as detection scores) and one or more of the following parameters related to non-detection determination.

[0046] (1) Threshold value for binarization

[0047] (2) The area to be binarized on the captured image

[0048] (3) Threshold for determining the presence of undetected

[0049] The "threshold value when performing binarization processing" refers to, for example, a threshold value for determining the brightness value above which the binarized pixel value is set to 1 when each pixel of the captured image has a brightness value as a pixel value. The user can set the desired threshold value in consideration of the lighting conditions of the workspace, the nature of the object, etc. A default value may also be set in advance as the threshold value. In addition, the setting unit 128 may also automatically set the "threshold value when performing binarization processing" to, for example, a value of about half of the maximum brightness value represented by the number of bits of the pixel value (brightness value) of one pixel of the image based on the number of bits.

[0050] The "area targeted for binarization in the captured image" refers to the area targeted for binarization by the binarization processing unit 124. By being able to specify the area targeted for binarization, the user can, for example, avoid the situation where peripheral devices appear as a blob in the binarized image. Furthermore, by being able to specify the area targeted for binarization, the image processing load can be reduced, resulting in faster processing speed.

[0051] The "threshold value for determining that there is an undetected object" is a threshold value used when determining whether there is an undetected object in the determination unit 126. Here, the determination unit 126 determines that there is an undetected object when the difference between the second area and the first area is greater than the threshold value. The threshold value for determining that there is an undetected object can be set to, for example, half the area of ​​an object. In addition, when the area of ​​an object is known, the setting unit 128 can also automatically set the threshold value for determining that there is an undetected object to half the area of ​​a known object. From the perspective of determining the existence of an undetected object, when an object smaller than the object is captured in the binarized image, in order not to determine such an object as an undetected object, the threshold value for determining that there is an undetected object can also be set to a relatively large value (for example, a value larger than half the area of ​​an object).

[0052] Next, multiple objects are supplied to the working area of ​​the robot 10 and within the field of view of the visual sensor (step S2). Figure 1 As shown in the figure, they are arranged in the working area, or they can be supplied in a state where a plurality of them are arranged in a tray (not shown). Figure 2 In step S2, as an example, a plurality of objects are supplied via a tray. In this stage, as an example, Figure 1As shown, objects 91-94 are placed in the working area of ​​robot 10. The series of processes from step S2 to step S12 is repeated a predetermined number of times as a loop process. That is, the processes of steps S3-S11 are repeated a predetermined number of times while the objects are placed in the working area.

[0053] Next, the visual sensor 70 takes a picture according to the instruction from the visual sensor control unit 121 (step S3). Then, the detection unit 123 performs a detection process on the taken picture (step S4). Here, it is assumed that the taken picture is Figure 4A The image M1 shown is an image of four objects 91 to 94. It is assumed that the objects detected in the detection process of the detection unit 123 are three objects 92 to 94. Figure 4B The image M1B showing a symbol "+" indicating detection is superimposed on objects 92-94 detected by the detection unit 123. By observing the detection result image M1B, the user can understand that objects 92-94 are detected using the current detection parameters and that object 91 is not detected.

[0054] Next, the area calculator 125 calculates the total area of ​​the regions of the objects detected by the detector 123 on the captured image (step S5). In this case, the area calculator 125 calculates the total area of ​​the regions of the three objects 92-94 on the image M1 as the first area.

[0055] Next, the binarization processing unit 124 performs binarization processing on the captured image using the threshold value for binarization processing set in step S1 (step S6). In addition, when "the area to be binarized on the captured image" is specified, the binarization processing unit 124 performs binarization processing on the area specified on the captured image. Here, it is assumed that "the area to be binarized on the captured image" is specified for the entire area of ​​the captured image. In the image M1, the objects 91-94 are photographed relatively brightly, and the area of ​​the ground or the bottom surface of the tray is photographed darkly. By binarizing the image M1, the following can be obtained. Figure 5 The binarized image M2 is shown. In the example of image M2, the area where four objects 91-94 exist is extracted as the area of ​​a mass B1 having a pixel value of 1. In this case, the area having a pixel value of 0 is the ground or the bottom surface of the tray.

[0056] Next, the area calculation unit 125 calculates the sum of the areas of the regions extracted as clumps in the binarized image as the second area (step S7). In the case of image M2, the area of ​​the region of the clump B1 is calculated as the second area. Then, the determination unit 126 determines whether there is an undetected object based on the difference between the second area and the first area (step S8). The determination unit 126 determines that there is an undetected object when, for example, the difference between the second area and the first area is greater than the "threshold for determining that there is an undetected object" set in step S1. In this case, as Figure 6 As schematically shown, the area of ​​the extracted mass B1 (second area) is substantially larger than the area of ​​the region of detected objects 92-94 (first area) by only the area of ​​one object. Therefore, for example, by setting a value approximately half the area of ​​one object as the "threshold for determining the presence of an undetected object," it is possible to appropriately determine whether an undetected object exists.

[0057] If an undetected object exists (S9: Yes), the detection result storage unit 127 stores the image and information related to the detection process results as historical information in, for example, the storage device 22 (step S10). The detection results in this case may include historical images such as image M1B, parameters used in the detection process, and detection process results related to undetected objects (scores, etc.). This stored historical information can be used to analyze the causes of undetected objects and improve the detection process. If no undetected object exists (S9: No), the process proceeds to step S11.

[0058] Next, the robot 10, in accordance with the program, performs a workpiece transfer operation to remove the detected object and place it elsewhere (step S11). The pallet containing the object is then ejected (step S12). The series of steps S2 through S12 is repeated for the next object supplied (a loop process). The process ends when the loop process is executed a predetermined number of times.

[0059] The above-described non-detection determination process automatically and reliably determines whether a non-detection occurred during the detection process of the detection unit 123. Furthermore, if a non-detection occurred, historical information, including historical images, is automatically saved. Therefore, the user no longer needs to visually determine whether a non-detection occurred. Furthermore, historical information can be efficiently saved only for cases where a non-detection occurred. The historical information accumulated through the non-detection determination process can be used to analyze the causes of non-detections and improve the detection process.

[0060] In addition, the above-mentioned non-detection determination process can be applied not only to Figure 1The case where the objects are arranged in a row as exemplified above can also be applied to a case where the objects are scattered in a tray. In addition, the non-detection determination process can also be applied to a case where the number of objects supplied varies each time.

[0061] Figure 7 An example of a UI (user interface) screen 300 provided by the setting unit 128 is shown. The UI screen 300 is configured as a user interface for accepting settings for various parameters in step S1 of the non-detection determination process and presenting an image representing the detection result. The UI screen 300 includes an image display area 310 and a parameter setting area 320. The UI screen 300 may also include a program display area 330 that displays program commands related to imaging and image processing.

[0062] The captured image or the binarized image is displayed in the image display area 310. Information indicating the detection result may be added to the image displayed in the image display area 310.

[0063] The parameter setting area 320 may also include at least one of (1) an input field 321 for inputting a threshold value for binarization processing, (2) an input field 322 for specifying an area on the captured image to be binarized, and (3) an input field 323 for inputting a threshold value for determining whether a non-detection exists. The user can execute the non-detection determination process by inputting these parameters and performing predetermined operations. When the non-detection determination process is executed using the input parameters, the binarized image as part of the processing result can be displayed in the image display area 310.

[0064] In input field 321, a numerical value serving as a threshold value for binarization can be entered. In input field 322, for example, the coordinates of the upper left corner (pixel values ​​of the X and Y coordinates) and the coordinates of the lower right corner (pixel values ​​of the X and Y coordinates) of a rectangular area can be entered as the area to be binarized. While an example of a setting method for specifying the area to be binarized using coordinate values ​​is shown here, a graphical user interface can also be provided that allows the target area to be graphically specified on the image using a pointing device. In input field 323, a numerical value serving as a threshold value for determining the presence of undetected areas can be entered.

[0065] Through the UI screen 300, the user can make the following settings, for example. For example, the brightness value of the captured image ranges from 0 to 255. In this case, the user sets 231 as the "threshold value when performing binarization processing" (brightness threshold) so that after binarization, only the object has a pixel value of "1". For example, the size of an object is 10 pixels × 30 pixels, and the area is 300 pixels. In this case, the user can also set 150 pixels, which is half of the area of ​​300 pixels of an object, as the "threshold value for determining that there is no detection."

[0066] Figure 7 An example of a binarized image M3 displayed in image display area 310 is shown. In image M3, six objects are extracted as clumps from the binarized image, as in the aforementioned example. Furthermore, image M3 shows an example where a "+" symbol indicating detection is added to four of the six objects as information indicating the detection results. In other words, image M3 shows that six objects were extracted as clumps (bright areas), four of which were detected by the detection process, while two were not.

[0067] Information related to the results of the non-detection determination process can also be displayed on the UI screen 300. Examples of information related to the results of the non-detection determination process include the presence or absence of non-detections and the number of non-detected objects. For example, the determination unit 126 can calculate the number of non-detected objects by using known information about the area of ​​an object and calculating how many times the difference between the second area and the first area corresponds to the area of ​​the object. Furthermore, the area of ​​an object can be specified as one of the parameters in the parameter setting area 320.

[0068] In this way, the user can set parameters via the UI screen 300 and efficiently check the image after the binarization process and the result of the non-detection determination process.

[0069] In the above description, an example of a case is described in which the threshold value for performing binarization processing and the threshold value for determining whether there is non-detection are accepted by the user through the function of the setting unit 128, and these parameters set by the user are used in the non-detection determination process. The image processing device 20 may also have a function of automatically setting at least one of these parameters. Here, as a function of automatically setting at least one parameter, a function of obtaining an appropriate value of the parameter through learning is described. Figure 2 As shown, the image processing device 20 may also include a learning unit 129 that acquires appropriate values ​​of parameters through learning, including a threshold value for binarization and a threshold value for determining whether an image is not detected. The function of the learning unit 129 will be described below.

[0070] As an example, the learning unit 129 performs learning to obtain appropriate values ​​of parameters through machine learning. Here, as an example, an example of learning appropriate values ​​of parameters through supervised learning is shown. Deep learning methods can also be used in learning.

[0071] Even if the image of the object is the same object, it is believed that the brightness and contrast of the image may change due to various reasons such as the brightness of the workspace. Therefore, it can be considered that there is a correlation between the captured image and the parameters (thresholds when binarizing and thresholds used to determine the presence of undetected objects) when the image is successfully determined to have an undetected object. The learning unit 129 uses the captured image as input data and accumulates training data (performance data) using the parameter values ​​when the determination of undetected objects in the image is successful as correct answer data. The training data can also be stored in the storage device 22, for example.

[0072] The learning unit 129 performs learning using the accumulated training data. For example, the learning unit 129 causes the estimator to learn training data. This training data uses a captured image as input data and uses parameters obtained when the judgment of non-detection in the image is successful as correct answer data. Here, the estimator is composed of, for example, a neural network (NN) or a convolutional neural network (CNN). This allows the learning unit 129 to construct a learning model.

[0073] By inputting any captured image into the estimator (learning model), it is possible to obtain parameters estimated to be appropriate for the image.

[0074] As described above, according to the first embodiment, it is possible to automatically and reliably determine whether or not an undetected object that has not been detected by the detection process exists in a captured image.

[0075] Second embodiment

[0076] Next, the image processing apparatus 20A according to the second embodiment (see Figure 8 ) is described. The image processing device 20A of the second embodiment is configured to use image information representing a three-dimensional point group of an object to determine whether there is an undetected object. The device structure of the robot system 100A including the image processing device 20A of the second embodiment is the same as Figure 1 Same as shown.

[0077] Figure 8 FIG. 2 is a functional block diagram showing an image processing device 20A according to the second embodiment. Figure 8 In the embodiment, the same functional blocks as those of the image processing device 20 in the first embodiment are marked with the same reference numerals, and their descriptions are omitted or simplified. Figure 8As shown, the image processing device 20A includes a visual sensor control unit 121, an image acquisition unit 122, a detection unit 123, a plane calculation unit 131, an area calculation unit 125A, and a determination unit 126. The image processing device 20A may further include a detection result storage unit 127, a setting unit 128A, and a learning unit 129A. Figure 8 As shown, these functional blocks can also be implemented by the processor 21 executing software.

[0078] In this embodiment, the visual sensor 70A, in addition to its function of capturing two-dimensional images, also functions as a three-dimensional sensor capable of acquiring a three-dimensional point group representing the three-dimensional positional information of the captured object. For example, a TOF (Time of Flight) camera that captures distance images using the time-of-flight method, or a stereo camera consisting of two cameras, can be used as the three-dimensional sensor.

[0079] The image acquisition unit 122 acquires image information including a two-dimensional image and a three-dimensional point group of the captured object from the visual sensor 70A.

[0080] The plane calculation unit 131 can extract a point group within a specific height range from a 3D point group as a plane. Because a 3D point group includes the 3D positional information of each point, it can extract a point group within a specific height range as a plane. In other words, the plane calculation unit 131 functions as a blob extraction unit capable of extracting areas identified as clumps within image information. The "specific height range" is, for example, set by the user.

[0081] The area calculation unit 125A calculates the sum of the areas of the object regions detected on the image by the detection unit 123 as a first area. Furthermore, the area calculation unit 125A calculates the sum of the areas of the planes calculated by the plane calculation unit 131 as a second area. The determination unit 126 can then determine whether any object was not detected by the detection unit by comparing the first and second areas.

[0082] Hereinafter, the non-detection determination process executed by the image processing device 20A according to the second embodiment will be described. Figure 9 : is a flowchart of the non-detection determination process of the second embodiment. The non-detection determination process is executed under the control of the processor 21. Figure 9 In, with Figure 3 The same steps as those in the non-detection determination process of the first embodiment are denoted by the same step numbers, and their descriptions are omitted or simplified.

[0083] First, the setting unit 128A accepts user settings for parameters used in the non-detection determination process (step S1). The parameters set here include parameters used in the detection process of the detection unit 123 (such as detection score) and one or more of the following parameters related to non-detection determination.

[0084] (1) Find the range of the height of the three-dimensional point group;

[0085] (2) “A region to be determined as a plane on a captured image”;

[0086] (3) A threshold value for determining whether an undetected value exists.

[0087] The "height range of the three-dimensional point group for determining the area" is a parameter that specifies the height range used to extract the point group representing the plane from the three-dimensional point group. For example, based on the size (height) of the object, the user can specify a height range suitable for extracting the object's area as a cluster. For example, a certain range that includes the object's height can be set as the "height range of the three-dimensional point group for determining the area." Setting unit 128A can also automatically set the "height range of the three-dimensional point group for determining the area" based on the object's model data.

[0088] The "area for plane determination in the captured image" is the area on the image where the plane is to be determined from the three-dimensional point group. By being able to specify the "area for plane determination in the captured image," users can, for example, avoid peripheral devices being extracted as clumps during the plane determination process. Furthermore, by being able to specify the "area for plane determination in the captured image," the image processing load can be reduced, speeding up processing.

[0089] Next, multiple objects are supplied to the working area of ​​the robot 10 and within the field of view of the visual sensor (step S2). Figure 1 As shown in the figure, they are arranged in the working area, or they can be supplied in a state where a plurality of them are arranged in a tray (not shown). Figure 9 In step S2, as an example, a plurality of objects are supplied via a tray. In this stage, as an example, Figure 1 As shown, objects 91 - 94 are arranged in the working area of ​​the robot 10 .

[0090] Next, the vision sensor 70A captures the field of view in response to a command from the vision sensor control unit 121, acquiring an image and a three-dimensional point cloud (step S3a). The detection unit 123 then performs detection processing on the captured image based on the object's model data (step S4). The area calculation unit 125A calculates the sum of the areas of the object detected by the detection unit 123 in the captured image as the first area (step S5).

[0091] Next, based on the "height range of the three-dimensional point group whose area is to be calculated" specified by the user in step S1, the plane calculation unit 131 calculates a plane whose height falls within the specified range (step S6a). For example, the plane calculation unit 131 extracts as a plane the points within the specified height range within the three-dimensional point group. If the "area to calculate the plane in the captured image" is specified, the plane calculation unit 131 extracts the plane within the specified area in the image represented by the three-dimensional point group.

[0092] Next, the area calculator 125A calculates the sum of the areas of the regions identified as planes in step S6a as the second area (step S7a). The determination unit 126 then determines whether an undetected object exists based on the difference between the second area and the first area (step S8). If an undetected object exists (S9: Yes), the detection result storage unit 127 stores the historical image and detection results as historical information, for example, in the storage device 22 (step S10).

[0093] Next, the robot 10, in accordance with the program, performs a workpiece transfer operation to remove the detected object and place it elsewhere (step S11). The tray containing the object is then ejected (step S12). The series of steps S2 through S11 is repeated for the next object supplied (a loop process). The process ends when the loop process is executed a predetermined number of times.

[0094] In the case of Figure 1 When the non-detection determination process of this embodiment is executed for the objects 91-94 shown in FIG. Figure 4A The captured image M1 shown in FIG. Figure 4B The detection result image M1B is shown. In addition, by performing a plane-finding process on the three-dimensional point group, the points corresponding to the three-dimensional point group can be extracted. Figure 5 The same area as the area shown as blob B1 on image M2 is used as a plane. Therefore, even in the non-detection determination process of this embodiment, it is possible to automatically and reliably determine whether there are any non-detections during the detection process of the detection unit 123. Furthermore, if there are any non-detections, historical information including historical images is automatically saved. In other words, the non-detection determination process of this embodiment can also achieve the same effects as the non-detection determination process of the first embodiment.

[0095] In the second embodiment, the setting unit 128A can also provide a UI screen having the same function as the UI screen 300 in the first embodiment. Figure 10 The UI screen 300A provided by the setting unit 128A is shown as an example. Figure 10 In A, Figure 7 In the UI screens shown, parts having the same functions are denoted by the same reference numerals, and descriptions thereof are omitted or simplified.

[0096] like Figure 10 As shown, the UI screen 300A includes an image display area 310, a parameter setting area 320A, and a program display area 330. In this embodiment, the parameter setting area 320A may also include at least one of (1) an input field 324 for specifying the "height range of the three-dimensional point group for calculating the area", (2) an input field 325 for specifying the "area as the object for calculating the plane on the captured image", and (3) an input field 323 for inputting a threshold value for determining whether there is an undetected area. The user can execute the undetected area determination process by inputting these parameters and performing predetermined operations. When the determination process is executed using the input parameters, the image as the result of the process (image M3A in which the plane is extracted as a mass) can be displayed in the image display area 310.

[0097] In input field 324, a numerical value can be input for the "range of the height of the three-dimensional point group whose area is to be determined." In input field 325, the coordinates of the upper left corner (pixel values ​​of the X and Y coordinates) and the coordinates of the lower right corner (pixel values ​​of the X and Y coordinates) of the rectangular area can be input as the area to be extracted. In addition, here, an example of a setting method for specifying the area to be extracted as a plane by coordinate values ​​is shown, but a graphical user interface can also be provided that allows the target area to be graphically specified on the image using a pointing device. In input field 323, a numerical value can be input as a threshold value for determining the presence of undetected areas.

[0098] In this embodiment, image M3A representing a blob extracted by finding a plane from a three-dimensional point cloud can be displayed in image display area 310. The portion displayed in white in image M3A is a portion extracted from the three-dimensional point cloud as a region (blob) representing a plane within a set height range.

[0099] Information related to the results of the non-detection determination process may also be displayed on UI screen 300A. Examples of information related to the results of the non-detection determination process include the presence or absence of non-detections and the number of non-detected objects. For example, the determination unit 126 can calculate the number of non-detected objects by using known information about the area of ​​an object and calculating how many times the difference between the second area and the first area corresponds to the area of ​​the object. Furthermore, the area of ​​an object may be specified as one of the parameters in parameter setting area 320A.

[0100] In this way, the user can set parameters via the UI screen 300A and efficiently check the image after plane extraction and the result of the non-detection determination process.

[0101] In this embodiment, learning unit 129A performs learning using the same method as learning unit 129 in the first embodiment. Learning unit 129A receives a captured image as input data and performs learning using parameter values ​​("the range of heights of the three-dimensional point cluster for determining the area" and "the threshold for determining the presence of an undetected object") that are successfully determined for that image as training data for correct answer data. By inputting any captured image into the estimator (learning model), it is possible to obtain parameters ("the range of heights of the three-dimensional point cluster for determining the area" and "the threshold for determining the presence of an undetected object") that are estimated to be appropriate for that image.

[0102] In this manner, also in the second embodiment, it is possible to automatically and reliably determine whether or not an undetected object that has not been detected by the detection process exists in the captured image.

[0103] While the above embodiment illustrates an example in which the "range of heights of the three-dimensional point group for determining the area" is specified as a parameter for extracting a plane from a three-dimensional point group, the "height of the three-dimensional point group for determining the area" can also be specified as a parameter for extracting a plane from a three-dimensional point group. In this case, the plane calculation unit 131 can extract as a plane a point group having a height substantially equal to the specified "height of the three-dimensional point group for determining the area." Alternatively, the plane calculation unit 131 can extract as a plane a point group within a predetermined threshold range from the value of the "height of the three-dimensional point group for determining the area."

[0104] The non-detection determination process described in each of the above-mentioned embodiments is also capable of determining whether an object not targeted for detection exists within the work area (the visual sensor's field of view) where the object is being fed. For example, by setting the "threshold for determining the presence of an undetected object" to a relatively low value (e.g., a value sufficiently smaller than the area of ​​a single object), it is possible to determine the presence of a relatively small unconfirmed object within the captured image (i.e., within the work area).

[0105] The above-described embodiments are configuration examples in which the image processing device is configured as a separate device from the robot control device in the robot system. Alternatively, the image processing device functionality may be integrated into the robot control device. In this case, the UI screen 300 and UI screen 300A may be provided on the display unit 41 of the teaching pendant 40. Alternatively, the image processing device functionality may be integrated into the teaching pendant 40.

[0106] Figure 2 、 Figure 8 The functional blocks of the image processing device shown above may be implemented by a processor of the image processing device executing various software stored in a storage device, or may be implemented by a configuration mainly based on hardware such as an ASIC (Application Specific Integrated Circuit).

[0107] Programs for executing various processes such as the non-detection determination process in the above-mentioned embodiments can be recorded on various computer-readable recording media (e.g., semiconductor memories such as ROM, EEPROM, and flash memory, magnetic recording media, and optical disks such as CD-ROM and DVD-ROM).

[0108] As described above, according to each embodiment, it is possible to automatically and reliably determine whether or not an undetected object that has not been detected by the detection process exists in a captured image.

[0109] The present disclosure has been described in detail, but the present disclosure is not limited to the above-mentioned embodiments. These embodiments can be variously added, replaced, changed, partially deleted, etc. without departing from the scope of the present disclosure, or without departing from the scope of the present disclosure derived from the contents recorded in the scope of the patent protection requested and its equivalents. In addition, these embodiments can also be implemented in combination. For example, in the above-mentioned embodiment, the order of each action and the order of each processing are shown as an example and are not limited to this. In addition, the same applies to the case where numerical values ​​or mathematical formulas are used in the description of the above-mentioned embodiments.

[0110] The following supplementary notes are further described with respect to the above-mentioned embodiment and modifications.

[0111] (Note 1)

[0112] An image processing device (20, 20A) is characterized by comprising: an image acquisition unit (122) that acquires image information obtained from a visual sensor (70, 70A) within a field of view captured by the visual sensor; a detection unit (123) that performs detection processing for detecting an object from the image information based on information representing characteristics of the object; a cluster extraction unit (124, 131) that extracts an area determined as a cluster based on the image information; an area calculation unit (125, 125A) that calculates the area of ​​the object detected by the detection unit (123) as a first area and calculates the area of ​​the area determined as the cluster as a second area; and a determination unit (126) that determines whether there is an undetected object in the image information that was not detected in the detection processing based on a comparison between the first area and the second area.

[0113] (Note 2)

[0114] According to the image processing device (20, 20A) described in Appendix 1, wherein the area calculation unit (125, 125A) calculates the sum of the areas of the objects detected by the detection unit as the first area, and calculates the sum of the areas of the regions determined as the clumps as the second area.

[0115] (Note 3)

[0116] An image processing device (20) according to Note 1 or 2, wherein the image information includes a two-dimensional image, the blob extraction unit (124) includes a binarization processing unit (124) for binarizing the two-dimensional image, the binarization processing unit (124) extracts an area having a specific pixel value in the binarized image as an area determined as the blob, and the area calculation unit (125) calculates the sum of the areas of the areas having the specific pixel value in the binarized image as the second area.

[0117] (Note 4)

[0118] According to the image processing device (20) described in Appendix 3, the image processing device further includes: a setting unit (128) which accepts an operation of specifying at least one of a threshold value when performing the binarization processing and a region in the two-dimensional image that is the object of the binarization processing.

[0119] (Note 5)

[0120] The image processing device (20A) according to Supplementary Note 1 or 2, wherein

[0121] The image information includes a three-dimensional point group,

[0122] The blob extraction unit (131) includes a plane calculation unit (131) that extracts a plane existing at a specific height or a specific height range from the three-dimensional point group as a region determined as the blob,

[0123] The area calculation unit (125A) calculates the sum of the obtained areas of the planes as the second area.

[0124] (Note 6)

[0125] An image processing device (20A) according to Note 5, wherein the image processing device further comprises: a setting unit (128A) which accepts an operation of specifying at least one of the specific height, the range of the specific height, and the area in the three-dimensional point group as an object for obtaining the plane.

[0126] (Note 7)

[0127] The image processing device (20, 20A) according to any one of Supplementary Notes 1 to 6, wherein the determination unit (126) determines that the undetected object exists when the difference between the second area and the first area is greater than a predetermined threshold value.

[0128] (Note 8)

[0129] An image processing device (20, 20A) according to any one of Notes 1 to 7, wherein the determination unit (126) determines how many undetected objects that are not detected in the detection process exist in the image information based on information representing the area of ​​one of the objects.

[0130] (Note 9)

[0131] An image processing device (20, 20A) according to any one of Notes 1 to 8, wherein the image processing device further comprises: a detection result storage unit (127) which stores the image information and information related to the result of the detection processing when the determination unit (127) determines that there is an undetected object.

[0132] (Note 10)

[0133] An image processing device (20, 20A) according to Note 1, 2, 3 or 5, wherein the image processing device further comprises: a setting unit (128, 128A) which receives an operation for specifying a judgment threshold value used by the judgment unit (126) in the judgment, and the judgment unit (126) determines that the undetected object exists when the difference between the second area and the first area is greater than the specified judgment threshold value.

[0134] (Note 11)

[0135] An image processing device (20) according to Note 3, wherein the image processing device further comprises: a learning unit (129) which learns the image information and performance data related to the threshold value when the binarization processing unit performs the binarization processing and at least one of the determination threshold values ​​used by the determination unit in the determination as training data, and provides an estimated value of at least one of the threshold value when the binarization processing is performed and the determination threshold value that should be applied to any input image information.

[0136] Explanation of symbols

[0137] 10 robots

[0138] 20.20A Image processing device

[0139] 21 processors

[0140] 22 Storage devices

[0141] 23 Operation Department

[0142] 24 display unit

[0143] 40 teaching device

[0144] 41 Display unit

[0145] 50 robot control device

[0146] 70, 70A vision sensor

[0147] 100, 100A robot system

[0148] Objects 90, 91-94

[0149] 121 Vision Sensor Control Unit

[0150] 122 Image acquisition unit

[0151] 123 Testing Department

[0152] 124 Binarization Processing Unit

[0153] 125, 125A Area Calculation Unit

[0154] 126 Judgment Department

[0155] 127 test result storage unit

[0156] 128, 128A setting unit

[0157] 129, 129A Learning Department

[0158] 151 Motion Control Unit

[0159] 300, 300AUI screen

[0160] 310 image display area

[0161] 320, 320A parameter setting area

[0162] 330 program display area

[0163] 321-325 input fields.

Claims

1. An image processing device, characterized in that: have: an image acquisition unit that acquires image information obtained by capturing an area within a visual field of the visual sensor from the visual sensor; a detection unit that performs a detection process for detecting the object from the image information based on information indicating a feature of the object; a cluster extraction unit that extracts a region determined to be a cluster based on the image information; an area calculation unit that calculates the area of ​​the object detected by the detection unit as a first area and calculates the area of ​​the region determined as the mass as a second area; as well as A determination unit determines whether or not an undetected object that has not been detected in the detection process exists in the image information based on a comparison between the first area and the second area.

2. The image processing device according to claim 1, wherein The area calculation unit calculates a total of areas of the objects detected by the detection unit as the first area, and calculates a total of areas of regions specified as the clump as the second area.

3. The image processing device according to claim 1 or 2, characterized in that The image information includes a two-dimensional image, The cluster extraction unit includes a binarization unit that performs binarization processing on the two-dimensional image. The binarization unit extracts a region having a specific pixel value from the binarized image as a region determined to be the blob. The area calculation unit calculates the sum of the areas of regions having the specific pixel values ​​in the binarized image as the second area.

4. The image processing device according to claim 3, wherein The image processing apparatus further includes a setting unit that receives an operation for designating at least one of a threshold value for performing the binarization process and a region in the two-dimensional image that is a target of the binarization process.

5. The image processing device according to claim 1 or 2, characterized in that The image information includes a three-dimensional point group, The blob extraction unit includes a plane calculation unit that extracts a plane existing at a specific height or a specific height range from the three-dimensional point group as a region to be determined as the blob. The area calculation unit calculates the sum of the obtained areas of the planes as the second area.

6. The image processing device according to claim 5, wherein: The image processing device further includes a setting unit that receives an operation for designating at least one of the specific height, a range of the specific height, and a region in the three-dimensional point group from which the plane is to be found.

7. The image processing device according to any one of claims 1 to 6, characterized in that: The determination unit determines that the undetected object exists when a difference between the second area and the first area is equal to or greater than a predetermined threshold value.

8. The image processing device according to any one of claims 1 to 7, characterized in that The determination unit determines how many undetected objects that were not detected in the detection process are present in the image information based on information indicating the area of ​​one object.

9. The image processing device according to any one of claims 1 to 8, characterized in that The image processing device further includes a detection result storage unit that stores the image information and information on a result of the detection process when the determination unit determines that an undetected object exists.

10. The image processing device according to claim 1, 2, 3 or 5, characterized in that: The image processing apparatus further includes a setting unit that receives an operation for designating a threshold value for determination used by the determination unit. The determination unit determines that the undetected object exists when the difference between the second area and the first area is equal to or greater than the specified determination threshold.

11. The image processing device according to claim 3, wherein The image processing device also includes: a learning unit, which uses the image information and performance data related to the threshold value when the binarization processing unit performs the binarization processing and at least one of the determination threshold values ​​used by the determination unit in the determination as training data to learn, and provides an estimated value of at least one of the threshold value when the binarization processing is performed and the determination threshold value that should be applied to any input image information.

Images