Image processing device

Abstract

The device provided is an image processing device comprising: an image acquisition unit that captures image information from a visual sensor as a result of the visual sensor taking an image within a field of view; a recognition unit that performs a recognition process to identify an object from the image information based on information representing object features; a blob extraction unit that uses the image information as the basis for extracting an area specified as a blob; and an area calculation unit that calculates the area of ​​the object detected by the recognition unit as the first area and the area of ​​the region identified as a blob as the second area.and an investigation unit that, based on a comparison of the first and second surfaces, determines whether an undetected object, which was not detected in the recognition process, is present in the image information.

Description

Area

[0001] The present disclosure relates to an image processing device. State of the art

[0002] A robot system is known that can detect a target object with a visual sensor and perform a process such as removing the target object.

[0003] PTL 1, for example, describes a robot system that includes a robot and an image acquisition device with which the robot is equipped, and which performs the task of transporting stacked objects from one container to another. PTL 2 describes a handling system for workpiece transfer that is used when applying a grinding process to a plate-shaped metallic workpiece obtained by fusion cutting or the like. List of quotations Patent literature [PTL 1] Unexamined Japanese patent publication (Kokai) No. 2020-21212 A [PTL 2] Unexamined Japanese patent publication (Kokai) No. 2007-021635 A Overview: Technical Task

[0004] In a system that detects a target object with a visual sensor and uses a robot to pick up the target object, some target objects may remain undetected during the detection processing by the visual sensor due to various reasons. When such a non-detection occurs, an image is saved, the cause of the non-detection is investigated, and the detection processing is improved. However, whether a non-detection has occurred is generally determined by human visual inspection. A technology is desired that can automatically and reliably detect whether a non-detection occurs during the detection processing. Technical solution

[0005] One embodiment of the present disclosure is an image processing device comprising: an image acquisition unit configured to acquire image information from a visual sensor, which is acquired by the visual sensor by capturing an image of the interior of a field of view; a recognition unit configured to perform recognition processing to identify a target object from the image information based on information representing a feature of the target object; a blob extraction unit configured to extract an area specified as a blob from the image information; an area calculation unit configured to calculate the area of ​​the target object detected by the recognition unit as the first area and the area of ​​an area specified as a blob as the second area;and an investigation unit trained to determine, based on a comparison between the first surface and the second surface, whether an undetected object, which was not detected during recognition processing, is present in the image information.

[0006] The tasks, features and advantages of the present invention and other tasks, features and advantages will become more apparent from the detailed description of typical embodiments of the present invention, which are illustrated in the accompanying drawings. Brief description of the drawings Fig. Figure 1 is a graphical representation depicting a configuration of a robot system that includes an image processing device according to one embodiment. Fig. Figure 2 is a graphical representation that depicts a graphical function block representation relating to the image processing device and a robot teaching device according to a first embodiment. Fig. Figure 3 is a flowchart representing a non-detection detection process according to the first embodiment. Fig. 4A is an example of a captured image in which four target objects are included. Fig. 4B is a graphical representation that depicts a state of information indicating a recognition result that is added to the captured image in which the four target objects are included. Fig. 5 is a graphical representation that depicts a binarized image, created by performing binarization processing on the captured image. Fig. 4A has been recorded. Fig. Figure 6 is a graphical representation that shows a comparison between the area of ​​an extracted blob and the area of ​​regions of detected target objects. Fig. Figure 7 is a graphical representation that shows an example of a UI screen according to the first embodiment. Fig. Figure 8 is a graphical representation that depicts a graphical function block representation relating to an image processing device and a robot teaching device according to a second embodiment. Fig. Figure 9 is a flowchart that represents a non-detection detection processing according to the second embodiment. Fig. Figure 10 is a graphical representation that provides an example of a UI screen according to the second embodiment. Description of the embodiments

[0007] Next, embodiments of the present disclosure are described with reference to the drawings. In the referenced drawings, similar components or functional parts are identified by similar reference numerals. For ease of understanding, the drawings may use different scales. Furthermore, the configurations shown in the drawings are examples of an implementation of the present invention, and the present invention is not limited to the configurations shown.

[0008] A visual sensor, as used here, is a two-dimensional camera that captures a two-dimensional image, a three-dimensional sensor that captures three-dimensional positional information of a target, or a device that combines the functions of both a two-dimensional camera and a three-dimensional sensor. The visual sensor provides image information (e.g., a two-dimensional image or a three-dimensional point cloud) of a target from which an image is captured within the field of view. Recognition processing, as used here, involves processing to recognize a target object in an image using a technique such as pattern matching based on known feature information of the target object. The term "blob," as used here, represents a solid area without a particularly defined shape that can be extracted by applying a predefined image processing technique to image information. First embodiment

[0009] Fig. Figure 1 is a graphical representation depicting a configuration of a robot system that includes an image processing device according to one embodiment. An image processing device 20 has a function for controlling a visual sensor 70 and for processing an image captured by the visual sensor 70. As shown in Fig. As shown in Figure 1, a robot system 100 comprises a robot 10, a robot control unit 50 that controls the robot 10, a teaching device 40 that is connected to the robot control unit 50, the visual sensor 70, and the image processing device 20. For example, the robot system 100 can detect a target object 90 arranged in a work area by means of the visual sensor 70 and handle the target object 90 with a hand (not shown) attached to the robot 10.

[0010] Although the robot 10 in this embodiment is a vertical articulated robot, depending on the intended task, a different robot type, such as a robot with parallel linkage or a dual-arm robot, can also be used. The robot 10 can perform the desired task with an end effector attached to its wrist.

[0011] The visual sensor 70 functions as a two-dimensional camera, capturing a grayscale image and / or a color image. It should be noted that, although Fig. Figure 1 provides an example of how the visual sensor 70 is a fixed camera that is mounted in a work area; the visual sensor 70 can be attached to the wrist of the robot 10.

[0012] The image processing device 20 stores model patterns of target objects 90 (91 to 94) and can perform recognition processing to detect a target object by pattern matching between an image of the target object in the captured image and the model pattern. The visual sensor 70 is assumed to be calibrated, and the image processing device 20 is assumed to store calibration data that defines a relative positional relationship between the visual sensor 70 and the robot 10. In this way, a position in a two-dimensional image captured by the visual sensor 70 can be converted into a position in a coordinate system defined for a workspace (e.g., a robot coordinate system).A situation in which a recognition unit 123 cannot correctly recognize a target object in an image in which the target object is captured can occur due to various causes, such as the arrangement of the target objects and the lighting in the work area. As described below, the image processing device 20 can provide a function to automatically determine, upon detection of a target object by the recognition processing, whether an object that was identified as unrecognized is present.

[0013] Although the image processing device 20 in Fig. 1 is designed as a device separate from the robot control unit 50, the function as an image processing device 20 can be integrated into the robot control unit 50.

[0014] The image processing device 20 can have a hardware configuration as a general computer, including a processor 21, memory (e.g., ROM, RAM, or non-volatile memory), a storage device 22, an operator unit 23, a display unit 24, an input / output interface, a network interface, and the like (see Fig. 2) The image processing device 20 can be configured with a personal computer (PC) or various other information processing devices. For example, the display unit 24 is a liquid crystal display. The operating unit 23 can, for example, include various pointing devices such as a keyboard and a mouse.

[0015] The robot control unit 50 controls the operation of the robot 10 according to an operating program or a command from the teaching device 40. The robot control unit 50 can have a hardware configuration as a general computer, which includes a processor, memory (e.g., ROM, RAM, or non-volatile memory), a storage device, an operator panel, an input / output interface, a network interface, and the like.

[0016] The teaching device 40 is used as an operator interface for programming the robot 10 and configuring various types of settings. The teaching device 40 can be a handheld programming device or a tablet computer or similar device. The teaching device 40 can have a hardware configuration similar to a general-purpose computer, including a processor, memory (e.g., ROM, RAM, or non-volatile memory), a storage device, an operator interface, a display unit 41, an input / output interface, a network interface, and the like. For example, the display unit 41 is configured with a liquid crystal display.

[0017] Fig. Figure 2 shows a graphical functional block representation relating to the image processing device 20 and the robot control unit 50. As in Fig. As shown in Figure 2, the robot control unit 50 includes an operating control unit 151. The operating control unit 151 controls the operation of the robot 10 according to the operating program or a command from the teaching device 40. The robot control unit 50 includes a servo control unit (not shown) which executes servo control on a servo motor on each axis according to a command to the axis generated by the operating control unit 151.

[0018] As in Fig. As shown in Figure 2, the image processing device 20 includes a control unit 121 of the visual sensor, an image acquisition unit 122, a recognition unit 123, a binarization processing unit 124, an area calculation unit 125, and a determination unit 126. The image processing device 20 may further include a storage unit 127 for recognition results, a setting unit 128, and a learning unit 129. As shown in Fig. As shown in Figure 2, the functional blocks can be provided by the execution of software by the processor 21.

[0019] The control unit 121 of the visual sensor controls the operation of the visual sensor 70. For example, the control unit 121 of the visual sensor can receive an operating command for the visual sensor 70 from the robot control unit 50 and control the visual sensor 70. The image acquisition unit 122 acquires image information, which is captured by taking an image of the interior of the field of view by the visual sensor 70. According to the present embodiment, the image acquisition unit 122 acquires a two-dimensional image from the visual sensor 70.

[0020] The detection unit 123 can perform detection processing to recognize a target object in image information (in this case, a two-dimensional image) acquired by the visual sensor 70, based on known feature information of the target object. For example, the image processing device 20 stores model data of a target object in the storage device 22. The detection unit 123 possesses model data of a target object and can detect the target object in an image by using the model data for comparison. For example, the detection unit 123 detects a target object in a captured image by comparing an edge feature of the model data with an edge feature of the target object in the image. In this way, the detection unit 123 can specify an area in which the target object is present in the image.

[0021] The Binarization Processing Unit 124 can perform binarization processing on an image. Specifically, it can extract a region with a specific pixel value (e.g., 1) from a binarized image as a blob. In other words, the Binarization Processing Unit 124 functions as a blob extraction unit, capable of extracting a region specified as a blob.

[0022] The Area Calculation Unit 125 can calculate the area of ​​a target object detected by the Recognition Unit 123 (this area is also referred to as the first area) and can calculate the area of ​​a region specified as a blob (this area is also referred to as the second area). For example, the Area Calculation Unit 125 can calculate the sum of the areas of regions containing target objects detected by the Recognition Unit 123 in an image as the first area. Furthermore, the Area Calculation Unit 125 can calculate the sum of the areas of regions containing blobs extracted by the Binarization Processing Unit as the second area. The Area Calculation Unit 125 can calculate the areas based on a coordinate system defined in the image.

[0023] The detection unit 126 can determine, based on the comparison between the first area and the second area calculated by the area calculation unit 125, whether an object not detected by the recognition unit 123 (e.g. an unrecognized target object) is present in an image.

[0024] The storage unit 127 for recognition results provides a function for storing an image and information about the recognition result, for example, in the storage device 22, when an unrecognized target object is present. The setting unit 128 provides a function for performing various types of settings related to the operation of the image processing device 20. For example, the setting unit 128 can provide a user interface (UI) that accepts settings of parameters used to perform processing by the binarization processing unit 124 and the detection unit 126. The setting unit 128 can control the display unit 24 to display the UI and accept user input into the UI via the control unit 23.

[0025] A processing operation by the image processing device 20 to determine whether an object that has been identified as unrecognized during the recognition processing by the recognition unit 123 is present in a recorded image (hereinafter also referred to as non-recognition determination processing) is described below.

[0026] Fig. Figure 3 is a flowchart illustrating the non-recognition detection processing. The non-recognition detection processing is executed under the control of the processor 21 of the image processing device 20. A situation is assumed in which four target objects 91 to 94 are arranged in a work area, as shown in Figure 3. Fig. Figure 1 shows the visual sensor 70 arranged such that the working area, which includes the four target objects 91 to 94, is included in the image acquisition area, and the non-recognition detection processing is performed on an image captured in the situation.

[0027] First, the image processing device 20 (setting unit 128) accepts a user setting of various parameters (step S1). The parameters to be set include one or more of the following parameters relating to the determination of non-recognition, in addition to parameters (e.g., a recognition score) that are used in the recognition processing by the recognition unit 123. (1) a threshold value when performing the binarization processing (2) an area that is a target of binarization processing in a captured image (3) a threshold for determining whether a failure to detect has occurred

[0028] For example, if each pixel in a captured image has a brightness value as its pixel value, a "threshold for performing binarization processing" represents a threshold for determining a brightness value, with a value greater than this causing the corresponding binarized pixel value to be set to 1. A user can set a desired threshold, taking into account the lighting conditions in the workspace, the properties of a target object, and so on. A default value can be preset as the threshold. It should be noted that, based on the bit count of a pixel value (brightness value) in an image, setting unit 128 can, for example, automatically set the "threshold for performing binarization processing" to a value approximately half the maximum brightness value represented by the bit count.

[0029] A "region that is a target of binarization processing in a captured image" is an area that is the target of binarization processing performed by the Binarization Processing Unit 124 in the captured image. By making a target of binarization processing specifiable, a user can, for example, avoid a situation where peripheral devices appear as blobs in a binarized image. Furthermore, by making a target of binarization processing specifiable, a reduction in image processing load and an acceleration of processing can be achieved.

[0030] A "non-detection threshold" is a threshold used when the detection unit 126 determines whether a non-detection has occurred. The detection unit 126 determines that an undetected target is present if the difference between the second area and the first area is equal to or greater than the threshold. The non-detection threshold can, for example, be half the area of ​​a target. Note that if the area of ​​a target is known, the setting unit 128 can automatically set the non-detection threshold to half the known area of ​​the target.If an object smaller than a target object appears in a binarized image, from the perspective of determining the presence of an unrecognized target object, the threshold for determining non-recognition can be set to a relatively large value (e.g., a value greater than half the area of ​​a target object) so that such an object is not identified as an unrecognized target object.

[0031] Next, a plurality of target objects are delivered into the workspace of robot 10 and simultaneously into the field of view of the visual sensor (step S2). The target objects can be in a lined-up state, as shown in Fig. 1 shown, in which the workspace is placed, or can be delivered in a state where multiple target objects are placed in a palette (not shown). Step S2 in Fig. Section 2 describes how a plurality of target objects are delivered, for example, in a pallet. In this phase, target objects 91 to 94 are placed in the work area of ​​robot 10, as shown in... Fig. Figure 1 is shown as an example. A series of processing operations from step S2 to step S12 is executed as a loop a predetermined number of times. In other words, the processing in steps S3 to S11 with the target objects placed in the workspace is executed a predetermined number of times.

[0032] Next, the visual sensor 70 performs an image acquisition according to a command from the visual sensor's control unit 121 (step S3). Subsequently, the detection unit 123 performs the detection processing on the acquired image (step S4). The acquired image is assumed to be an image M1 containing four target objects 91 to 94, as shown in Fig. Figure 4A is shown. It is assumed that the target objects recognized by the recognition unit 123 during recognition processing are the three target objects 92 to 94. Fig. Image 4B represents image M1B in which each of the target objects 92 to 94 detected by the detection unit 123 is overlaid with a "+" sign indicating a detection. By viewing image M1B, which is the detection result, a user can see that the target objects detected by the current detection parameter are target objects 92 to 94, and that target object 91 is not detected.

[0033] Next, the area calculation unit 125 determines the total area of ​​the target object regions identified by the recognition unit 123 in the captured image (step S5). In this case, the area calculation unit 125 calculates the sum of the areas of the three target objects 92 to 94 in image M1 as the first area.

[0034] Next, the binarization processing unit 124 performs binarization processing on the captured image using the threshold set in step S1 (step S6). If an "area that is a target for binarization processing in a captured image" is specified, the binarization processing unit 124 performs binarization processing on the specified area in the captured image. It is assumed that an "area that is a target for binarization processing in a captured image" is specified for the entire area of ​​the captured image. In image M1, target objects 91 to 94 are captured relatively brightly, and an area of ​​the bottom surface or lower surface of the palette is captured darkly. By performing binarization processing on image M1, a binarized image M2 can be obtained, as shown in Fig. Figure 5 illustrates this. In the example of image M2, an area containing four target objects 91 to 94 is extracted as a region of blob B1 with a pixel value of 1. Note that an area with a pixel value of 0, in this case, represents the bottom surface or the lower surface of the palette.

[0035] Next, the area calculation unit 125 determines the total area of ​​the blob-extracted regions in the binarized image as the second area (step S7). In the case of image M2, the area of ​​blob B1 is determined as the second area. Subsequently, the detection unit 126 determines whether an undetected target object is present based on the difference between the second area and the first area (step S8). For example, if the difference between the second area and the first area is equal to or greater than the "threshold for determining non-detection" set in step S1, the detection unit 126 determines that a non-detection has occurred. In this case, as described in Fig. Figure 6 schematically shows that the area of ​​the extracted blob B1 (the second area) is practically larger by the area of ​​a target object than the area of ​​the regions of the detected target objects 92 to 94 (the first area). Accordingly, for example, by setting the value of approximately half the area of ​​a target object to the "threshold for determining whether a non-detection has occurred," the determination of whether a non-detection has occurred can be carried out appropriately.

[0036] If an unrecognized target object is present (S9: YES), the recognition result storage unit 127 stores an image and information about the result of the recognition processing as history information, for example, in storage device 22 (step S10). In this case, the recognition result can include a history image, such as image M1B, a parameter used in the recognition processing, and the result of the recognition processing with respect to the unrecognized target object (e.g., a score value). The stored history information can be used to analyze the cause of the non-recognition and to improve the recognition processing. If no unrecognized target object is present (S9: NO), the processing proceeds to step S11.

[0037] Next, robot 10 performs a workpiece transfer operation to pick up the detected target objects and place them at a separate location according to the operating program (step S11). The pallet containing the target objects is then ejected (step S12). A series of processing operations from step S2 to S12 (loop processing) is repeated on subsequently supplied target objects. The processing ends when the loop processing has been executed a predetermined number of times.

[0038] The non-recognition detection processing described above enables the automatic and reliable determination of whether a non-recognition has occurred during the recognition processing by the recognition unit 123. If a non-recognition occurs, historical information, including a historical image and similar data, is automatically saved. This eliminates the need for a user to visually inspect whether a non-recognition has occurred. Furthermore, historical information can be efficiently saved only when a non-recognition has occurred. The historical information accumulated by the non-recognition detection processing can be used to analyze the cause of a non-recognition and to improve the recognition processing.

[0039] For example, the aforementioned non-detection detection processing can be applied not only to the case where target objects are placed in the lined-up state, as in Fig. 1 is shown, but also applies to a situation where target objects are present in a scattered state within a palette. Furthermore, the non-recognition detection processing mentioned above can be applied to a situation where the number of delivered target objects is not always the same.

[0040] Fig. Figure 7 presents an example of a user interface (UI) screen 300 provided by the setting unit 128. The UI screen 300 accepts the setting of various parameters in step S1 during non-detection detection processing and is designed as a user interface for displaying an image representing a detection result. The UI screen 300 includes an image display area 310 and a parameter setting area 320. The UI screen 300 may also include a program display area 330, which displays program instructions relating to image acquisition and image processing.

[0041] A captured image or a binarized image is displayed in image display area 310. Information indicating a recognition result can be added to the image displayed in image display area 310.

[0042] The parameter setting range 320 can include at least one or more of the following: (1) an input field 321 for entering a threshold value when performing the binarization processing, (2) an input field 322 for specifying an area that is a target for binarization processing in a captured image, and (3) an input field 323 for entering a threshold value to determine whether a non-detection has occurred.

[0043] A user can perform the non-recognition detection processing by entering the parameters and executing a predefined operation. When the non-recognition detection processing is executed with the input parameters, a binarized image can be displayed as part of the processing result in the image display area 310.

[0044] A numerical value can be entered into input field 321 as a threshold for performing binarization processing. For example, the coordinates (pixel values ​​of the x and y coordinates) of the upper left corner and the coordinates (pixel values ​​of the x and y coordinates) of the lower right corner of a rectangular area can be entered into input field 322 as the area to be targeted for binarization processing. It should be noted that, although an example of a setting technique for specifying an area to be targeted for binarization processing has been described using coordinate values, a graphical user interface may be provided that allows the target area to be specified graphically in an image using a pointing device. A numerical value can be entered into input field 323 as a threshold for determining whether a non-recognition has occurred.

[0045] For example, a user can make a setting via UI screen 300 as follows. For example, let's assume that the brightness value of a captured image ranges from 0 to 255. In this case, a user 231, for example, sets the "Threshold when performing binarization processing" (the brightness threshold) so that only one target object has a pixel value of "1" after binarization. For example, let's assume that the size and area of ​​a target object are 10 pixels by 30 pixels, or 300 pixels. In this case, the user can set 150 pixels, which is half the area of ​​300 pixels of a target object, as the "Threshold to determine if there is no detection".

[0046] Fig. Figure 7 shows an example of image M3 after binarization processing, displayed in image display area 310. In image M3, six target objects are extracted as blobs in a binarized image when the binarization processing is performed using the setting example mentioned above. Image M3 also shows an example of a "+" sign, indicating that a recognition is added to four of the six target objects, as information indicating a recognition result. In other words, image M3 indicates that six target objects are extracted as blobs (light areas), four of the target objects are recognized during the recognition processing, and two target objects remain unrecognized.

[0047] Further information about the result of the non-recognition detection processing can be displayed on UI screen 300. Examples of information about the result of the non-recognition detection processing include the presence of a non-recognition and the number of undetected target objects. For example, using known information about the area of ​​a target object, the detection unit 126 can determine the number of undetected target objects by calculating the ratio of the difference between the area of ​​the second and the area of ​​the first target object to the area of ​​the target object. Note that the area of ​​a target object can be specified as a parameter in parameter setting area 320.

[0048] In this way, the user can set parameters and efficiently confirm an image after binarization processing and a result of non-recognition determination processing via the UI screen 300.

[0049] Above, an example of accepting a user setting of the threshold during binarization processing and the threshold for determining the presence of non-detection by the function of the setting unit 128 and using the user-set parameters during non-detection detection processing has been described. The image processing device 20 may also have a function for automatically setting at least one of the parameters. A function for acquiring a suitable value of a parameter by learning is described as a function for automatically setting at least one of the parameters. As in Fig. As shown in Figure 2, the image processing device 20 can include a learning unit 129 which learns suitable values ​​of parameters, including the threshold for performing the binarization processing and the threshold for determining whether a non-recognition has occurred. The function of the learning unit 129 is described below.

[0050] As an example, learning unit 129 demonstrates how to learn to determine a suitable parameter value using machine learning. It describes an example of learning a suitable parameter value using supervised learning. A deep learning technique can be incorporated into the learning process.

[0051] The brightness and contrast of an image containing a target object can change for various reasons, such as the brightness of the workspace, even with the same target object. Accordingly, a correlation can be assumed between a captured image and parameters (the threshold for performing binarization processing and the threshold for determining non-recognition) if the presence of an unrecognized target object in the image is successfully determined. The training unit 129 accumulates training data (actual data) containing a captured image as input data and parameter values ​​as truth data when non-recognition is successfully determined for the image. For example, the training data can be stored in storage device 22.

[0052] Learning Unit 129 performs training using accumulated training data. For example, Learning Unit 129 instructs an estimator to learn training data using a captured image as input data and parameters. If a non-recognition for the image is successfully determined, the training unit uses these parameters as truth data. For example, the estimator is trained using a neural network (NN) or a convolutional neural network (CNN). In this way, Learning Unit 129 can build a learning model.

[0053] By entering any captured image into the estimation device (the learning model), parameters can be recorded that are considered suitable for the image.

[0054] As described above, the first embodiment enables an automatic and reliable determination of whether an undetected object, which was not detected during the recognition processing, is present in a recorded image. Second embodiment

[0055] In the following, an image processing device 20A according to a second embodiment (see Fig. 8) described. The image processing device 20A according to the second embodiment is configured to determine, using image information representing a three-dimensional point cloud of a target object, whether an undetected target object is present. A configuration of a robot system 100A that includes an image processing device 20A according to the second embodiment is similar to that described in Fig. 1 shown.

[0056] Fig. Figure 8 represents a graphical functional block representation of an image processing device 20A according to the second embodiment. Fig. 8 is a functional block corresponding to a functional block in the image processing device 20 according to the first embodiment, provided with the same symbol, and its description is omitted or simplified. As in Fig. As shown in Figure 8, the image processing device 20A includes the control unit 121 of the visual sensor, the image acquisition unit 122, the recognition unit 123, a plane calculation unit 131, an area calculation unit 125A, and the detection unit 126. The image processing device 20A may also include a storage unit 127 for recognition results, a setting unit 128A, and a learning unit 129A.

[0057] As in Fig. As shown in Figure 8, the functional blocks can be provided by the execution of software by the processor 21.

[0058] A visual sensor 70A according to the present embodiment has a function as a three-dimensional sensor which, in addition to an image acquisition function of a two-dimensional image, can acquire a three-dimensional point cloud representing three-dimensional position information of an image-acquisition target object. For example, a time-of-flight (TOF) camera that acquires a depth map by a time-of-flight method, or a stereo camera that includes two cameras, can be used as a three-dimensional sensor.

[0059] The image acquisition unit 122 captures image information from the visual sensor 70A, including a two-dimensional image and a three-dimensional point cloud of an image acquisition target object.

[0060] The Plane Calculation Unit 131 can provide a function for extracting a point cloud within a specific height range from a three-dimensional point cloud as a plane. Since a three-dimensional point cloud contains three-dimensional positional information for each point, a point cloud within the specific height range can be extracted as a plane. In other words, the Plane Calculation Unit 131 functions as a blob extraction unit, capable of extracting a blob-specified area into image information. For example, the "specific height range" is set by a user.

[0061] An area calculation unit 125A calculates the total area of ​​regions of target objects that have been detected in an image by the recognition unit 123 as the first area. The area calculation unit 125A then determines the total area of ​​planes calculated by the plane calculation unit 131 as the second area. Subsequently, the detection unit 126 can compare the first area with the second area to determine whether a detection by the recognition unit occurred or not.

[0062] The non-recognition detection processing performed by the image processing device 20A according to the second embodiment is described below. Fig. Figure 9 is a flowchart for the non-detection detection processing according to the second embodiment. The non-detection detection processing is executed under the control of processor 21. It should be noted that in Fig. 9 a step which is the same processing as a step in the non-detection investigation processing according to the in Fig. 3 is the first embodiment shown, is provided with the same step number and its description is omitted or simplified.

[0063] First, the setting unit 128A accepts a user setting of parameters used in the non-detection detection processing (step S1). The parameters to be set include one or more of the following parameters relating to a non-detection detection, in addition to parameters used in the detection processing by the detection unit 123 (e.g., a detection score). (1) a height range of a three-dimensional point cloud for area determination (2) an area which is a target of a level determination in a recorded image (3) a threshold for determining whether a failure to detect has occurred

[0064] A "Three-Dimensional Point Cloud Height Range for Area Determination" is a parameter that specifies a height range for extracting a point cloud representing a plane from a three-dimensional point cloud. For example, based on the size (height) of the target object, a user can specify a suitable height range for extracting a region of the target object as a blob. For instance, a specific area encompassing the height of a target object can be set as the "Three-Dimensional Point Cloud Height Range for Area Determination." The 128A setting unit can automatically set a "Three-Dimensional Point Cloud Height Range for Area Determination" based on model data of a target object.

[0065] An "area that is a target for plane detection in a captured image" is an area in an image that is the target of detecting a plane from a three-dimensional point cloud. By making an "area that is a target for plane detection in a captured image" specifiable, a user can, for example, avoid a situation where peripherals are extracted as blobs by processing a plane detection. Making an "area that is a target for plane detection in a captured image" specifiable can reduce the image processing load and speed up processing.

[0066] Next, a plurality of target objects are delivered into a work area of ​​the robot 10 and simultaneously into the field of view of the visual sensor (step S2). The target objects can be in a lined-up state, as shown in Fig. 1 shown, in which the workspace is placed, or can be delivered in a state where multiple target objects are placed in a palette (not shown). Step S2 in Fig. Figure 9 describes how a plurality of target objects are delivered, for example, in a pallet. In this phase, target objects 91 to 94 are placed in the working area of ​​robot 10, as shown in Figure 9. Fig. 1 is shown as an example.

[0067] Next, the visual sensor 70A captures the interior of the field of view according to a command from the visual sensor's control unit 121 and acquires an image and a three-dimensional point cloud (step S3a). Subsequently, the detection unit 123 performs the detection processing on the captured image based on model data of the target objects (step S4). The area calculation unit 125A determines the total area of ​​the target object regions that were first detected by the detection unit 123 in the captured image (step S5).

[0068] Next, the Level Calculation Unit 131 calculates a plane within a specified height range based on the "height range of a three-dimensional point cloud for area determination" specified by a user in step S1 (step S6a). For example, the Level Calculation Unit 131 extracts a point cloud within the specified height range in the three-dimensional point cloud as a plane. If an "area that is a target of a plane determination in a captured image" is specified, the Level Calculation Unit 131 extracts a plane in the specified area in an image represented by the three-dimensional point cloud.

[0069] Next, the area calculation unit 125A determines the total area of ​​the regions specified as planes in step S6a as the second area (step S7a). Subsequently, the detection unit 126 determines, based on the difference between the second area and the first area, whether an undetected target object is present (step S8). If an undetected target object is present (S9: YES), the storage unit 127 for recognition results stores a trend image and the recognition result as trend information, for example, in the storage device 22 (step S10).

[0070] Next, robot 10 performs a workpiece transfer operation to pick up the detected target objects and place them at a separate location according to the operating program (step S11). The pallet containing the target objects is then ejected (step S12). A series of processing operations (loop processing) from steps S2 to S11 is repeated on subsequently supplied target objects. The processing ends when the loop processing has been executed a predetermined number of times.

[0071] When the non-detection detection processing according to the present embodiment is performed on target objects 91 to 94, as described in Fig. As shown in 1, the captured image M1 is also recorded, as shown in Fig. 4A is shown, and the image M1B of a recognition result is also captured, as in Fig. 4B is shown. By applying the processing to determine a plane to a three-dimensional point cloud, the same area as the area shown as Blob B1 in image M2 can be found in Fig. 5 is extracted as a layer. Accordingly, in the non-recognition detection processing according to the present embodiment, it can also be automatically and reliably determined whether a non-recognition occurs during the recognition processing by the recognition unit 123. Furthermore, if a non-recognition occurs, historical information, including a historical image and the like, is automatically stored. In other words, the non-recognition detection processing according to the present embodiment can also provide similar effects to those provided by the non-recognition detection processing according to the first embodiment.

[0072] The setting unit 128A according to the second embodiment can also provide a UI screen which has a similar function to that of the UI screen 300 according to the first embodiment. Fig. Figure 10 represents a UI screen 300A, which is provided by the setting unit 128A. It should be noted that in Fig. 10A a part that performs the same function as that of the one in Fig. 7 shown UI screen, is marked with the same symbol and its description is omitted or simplified.

[0073] As in Fig. As shown in Figure 10, the UI screen 300A includes an image display area 310, a parameter setting area 320A, and a program display area 330. The parameter setting area 320A according to the present embodiment can include at least one or more of the following: (1) an input field 324 for specifying a “height range of a three-dimensional point cloud for area determination”, (2) an input field 325 for specifying an “area that is a target of a level detection in a captured image”, and (3) an input field 323 for entering a threshold value to determine whether a non-detection has occurred.

[0074] A user can perform the non-recognition detection processing by entering the parameters and executing a predefined operation. When the detection processing is executed with the input parameters, an image (image M3A, in which a layer is extracted as a blob) can be displayed in image display area 310 as the processing result.

[0075] A “height range of a three-dimensional point cloud for area determination” can be entered numerically in input field 324. Coordinates (pixel values ​​of the x and y coordinates) of the upper left corner and coordinates (pixel values ​​of the x and y coordinates) of the lower right corner of a rectangular area can be entered in input field 325 as the area to be targeted for plane extraction. It should be noted that, although an example of a setting technique for specifying an area to be targeted for plane extraction has been described by coordinate values, a graphical user interface may be provided that allows the target area to be specified graphically in an image using a pointing device. A numerical value can be entered in input field 323 as a threshold for determining whether an area is not detected.

[0076] The image M3A, which represents a blob extracted from a three-dimensional point cloud by defining a plane, can be displayed in the image display area 310 according to the present embodiment. A part shown in white in image M3A is a part extracted from a three-dimensional point cloud as an area representing a plane within a set height range (blob).

[0077] Information about the result of the non-detection investigation processing can be displayed on UI screen 300A. Examples of information about the result of the non-detection investigation processing include the presence of a non-detection and the number of undetected target objects. For example, using known information about the area of ​​a target object, the investigation unit 126 can determine the number of undetected target objects by calculating the ratio of the difference between the area of ​​the second and the area of ​​the first target object to the area of ​​the target object. Note that the area of ​​a target object can be specified as one of the parameters in parameter setting area 320A.

[0078] In this way, the user can set parameters and efficiently confirm an image after layer extraction and the result of the non-recognition determination processing via the UI screen 300A.

[0079] The learning unit 129A according to the present embodiment performs learning using a technique similar to that performed by the learning unit 129 according to the first embodiment. The learning unit 129A performs the learning with training data, which includes a captured image, as input data and with parameter values ​​(a "height range of a three-dimensional point cloud for area detection" and a "threshold for determining the presence of non-detection") as truth data when the presence of an undetected target object for the image has been successfully determined. By inputting any captured image into an estimation device (a learning model), parameters (a "height range of a three-dimensional point cloud for area detection" and a "threshold for determining the presence of non-detection") can be acquired that are deemed suitable for the image.

[0080] In this way, the second embodiment also enables an automatic and reliable determination of whether an unrecognized object, which was not recognized during the recognition processing, is present in a recorded image.

[0081] Although an example of specifying a "height range of a three-dimensional point cloud for area determination" as a parameter for extracting a plane from a three-dimensional point cloud is described in the embodiment mentioned above, a "height of a three-dimensional point cloud for area determination" can be specified as a parameter for extracting a plane from the three-dimensional point cloud. In this case, the plane calculation unit 131 can extract a point cloud with a height practically equal to the specified "height of a three-dimensional point cloud for area determination" from the three-dimensional point cloud as a plane. Alternatively, for example, the plane calculation unit 131 can extract a point cloud within a range of a specified threshold value from the value of the "height of a three-dimensional point cloud for area determination" as a plane.

[0082] The non-detection detection processing according to each of the embodiments mentioned above is a processing method that can determine whether an object that is not a target of the detection processing is present in the workspace (the field of view of the visual sensor) into which target objects are delivered. By setting the "non-detection threshold" to a relatively small value (e.g., a value sufficiently smaller than the area of ​​a target object), the presence of a relatively small, unconfirmed object in a captured image (i.e., in the workspace) can also be detected.

[0083] Each of the embodiments mentioned above describes a configuration example for the image processing device, which is arranged in the robot system as a separate device from the robot control unit. One possible example is a configuration of the image processing function that is integrally integrated into the robot control unit. In this case, the UI screen 300 and the UI screen 300A can be provided on the display unit 41 of the teaching device 40. Alternatively, the image processing function can be integrally integrated into the teaching device 40.

[0084] The functional blocks of the in Fig. 2 and Fig.The image processing device shown in Figure 8 can be provided by having the processor in the image processing device execute different types of software that is stored in the storage device, or can be provided by a configuration primarily based on hardware such as an application-specific integrated circuit (ASIC).

[0085] Programs that perform different types of processing according to the embodiments described above, such as non-detection detection processing, can be recorded on various computer-readable recording media (e.g., semiconductor memories such as ROM, EEPROM, and flash memory; magnetic recording media; and optical discs such as CD-ROM and DVD-ROM).

[0086] As described above, each embodiment enables automatic and reliable determination of whether an undetected object, which was not detected during recognition processing, is present in a captured image.

[0087] Although the present disclosure has been described in detail, it is not limited to each of the embodiments mentioned above. Various additions, replacements, modifications, deletions, and the like may be made to the embodiments without deviating from the essence of the present disclosure or from the scope of the present disclosure as defined by the content described in the claims and their equivalents. Furthermore, the embodiments may be implemented in combination. For example, the sequence of operations or the processing sequence in the embodiments mentioned above is described as an example and is not limited to it. The same applies if a numerical value or a mathematical expression is used in the description of the embodiments mentioned above.

[0088] The following additional remarks are further disclosed with regard to the embodiments mentioned above and their modified examples. Supplementary Note 1

[0089] An image processing device (20, 20A) comprising: an image acquisition unit (122) configured to acquire image information from a visual sensor (70, 70A), which is acquired by the visual sensor by taking an image of the interior of a field of view; a recognition unit (123) configured to perform recognition processing to identify a target object from the image information based on information representing a feature of the target object; a blob extraction unit (124, 131) configured to extract an area specified as a blob from the image information; an area calculation unit (125, 125A) configured to calculate the area of ​​the target object identified by the recognition unit (123) as the first area and the area of ​​an area specified as a blob as the second area;and an investigation unit (126) trained to determine, on the basis of a comparison between the first surface and the second surface, whether an undetected object, which was not detected during recognition processing, is present in the image information. Supplementary Note 2

[0090] The image processing device (20, 20A) according to Supplementary Note 1, wherein the area calculation unit (125, 125A) is configured to calculate as a first area a total sum of areas of one or more target objects detected by the recognition unit 123, and to calculate as a second area a total sum of areas of one or more regions specified as blobs. Supplementary Note 3

[0091] The image processing device (20) according to Supplementary Note 1 or 2, wherein the image information includes a two-dimensional image, the blob extraction unit (124) includes a binarization processing unit (124) configured to perform binarization processing on the two-dimensional image, wherein the binarization processing unit (124) is configured to extract an area having a specific pixel value in an image after binarization processing as an area specified as a blob, and wherein the area calculation unit (125) is configured to calculate a total sum of areas of one or more areas, each having the specific pixel value, in the image after binarization processing as a second area. Supplementary note 4

[0092] The image processing device (20) according to Supplementary Note 3, which further includes a setting unit (128) configured to accept a process for specifying a threshold value when performing the binarization processing and / or an area that is a target of the binarization processing in the two-dimensional image. Supplementary note 5

[0093] The image processing device (20A) according to supplementary note 1 or 2, wherein the image information includes a three-dimensional point cloud, the blob extraction unit (131) includes a plane calculation unit (131) configured to extract a plane present at a specific height or within a specific height range from the three-dimensional point cloud as a region specified as a blob, and The area calculation unit (125A) is designed to calculate a total sum of areas of one or more determined planes as a second area. Supplementary Note 6

[0094] The image processing device (20A) according to Supplementary Note 5, which further includes a setting unit (128A) configured to accept a process for specifying the specific height, specific height range and / or area which is a target for determining the plane in the three-dimensional point cloud. Supplementary note 7

[0095] The image processing device (20, 20A) according to any one of the supplementary notes 1 to 6, wherein the detection unit (126) is configured to determine that the undetected object is present when a difference between the second surface and the first surface is equal to or greater than a predetermined threshold. Supplementary Note 8

[0096] The image processing device (20, 20A) according to any one of Supplementary Notes 1 to 7, wherein the detection unit (126) is configured to identify a number of one or more unrecognized target objects that have not been detected by the recognition processing in the image information, based on information representing an area of ​​the one target object. Supplementary note 9

[0097] The image processing device (20, 20A) according to any one of the supplementary notes 1 to 8, which further includes a storage unit (127) for recognition results configured to store the image information and information about a result of the recognition processing when the detection unit (126) determines that an undetected object is present. Supplementary Note 10

[0098] The image processing device (20, 20A) according to Supplementary Note 1, 2, 3 or 5, further comprising a setting unit (128, 128A) configured to accept a process for specifying a detection threshold used for detection by the detection unit (126), wherein the detection unit (126) is configured to determine that the undetected object is present when a difference between the second surface and the first surface is equal to or greater than the specified detection threshold. Supplementary Note 11

[0099] The image processing device (20) according to Supplementary Note 3, which further includes a learning unit (129) trained to perform learning with training data which includes the image information and actual data with respect to a threshold when the binarization processing unit performs the binarization processing, and / or a detection threshold used for detection by the detection unit, and an estimate of a threshold to be applied to the binarization processing when the binarization processing is performed on any input image information, and / or a detection threshold. List of reference symbols 10 robots 20, 20A Image processing device 21 processor 22 Storage device 23 Control unit 24 display units 40 teaching device 41 Display unit 50 robot control unit 70, 70A visual sensor 100, 100A robot system 90, 91 to 94 Target object 121 Control unit of the visual sensor 122 image capture units 123 Recognition unit 124 Binary Processing Unit 125, 125A Area calculation unit 126 Investigation Unit 127 Recognition result storage unit 128, 128A setting unit 129, 129A Learning Unit 151 Operating control unit 300, 300A UI screen 310 image display area 320, 320A parameter setting range 330 Program display area Input fields 321 to 325 QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2020-21212 A

[0003] JP 2007-021635 A

[0003]

Claims

[1] Image processing device comprising: an image acquisition unit designed to acquire image information from a visual sensor, which is acquired by the visual sensor by taking an image of the interior of a field of view; a recognition unit that is trained to perform recognition processing to identify a target object from the image information based on information that represents a feature of the target object; a blob extraction unit that is trained to extract an area specified as a blob based on the image information; an area calculation unit configured to calculate an area of ​​the target object detected by the recognition unit as the first area and to calculate an area of ​​a region specified as a blob as the second area; and an investigative unit trained to determine, based on a comparison between the first surface and the second surface, whether an undetected object, which was not detected during recognition processing, is present in the image information. [2] Image processing device according to claim 1, wherein the area calculation unit is configured to calculate as a first area a total sum of areas of one or more target objects recognized by the recognition unit, and to calculate as a second area a total sum of areas of one or more regions specified as blobs. [3] Image processing device according to claim 1 or 2, wherein the image information includes a two-dimensional image, The blob extraction unit includes a binarization processing unit designed to perform binarization processing on the two-dimensional image. the binarization processing unit is designed to extract an area in an image, after binarization processing, that has a specific pixel value as an area specified as a blob, and The area calculation unit is designed to calculate a total sum of areas of one or more regions, each with a specific pixel value, as a second region in the image after binarization processing. [4] Image processing device according to claim 3, which further comprises a setting unit configured to accept a process for specifying a threshold value when performing the binarization processing and / or an area which is a target of the binarization processing in the two-dimensional image. [5] Image processing device according to claim 1 or 2, wherein the image information includes a three-dimensional point cloud, The blob extraction unit includes a plane calculation unit designed to extract a plane present at a specific height or within a specific height range from the three-dimensional point cloud as a region specified as a blob, and The area calculation unit is designed to calculate a total sum of areas of one or more determined levels as a second area. [6] Image processing device according to claim 5, which further comprises an adjustment unit configured to accept a process for specifying the specific height, the specific height range and / or a range which is a target of determining the plane in the three-dimensional point cloud. [7] Image processing device according to any one of claims 1 to 6, wherein the detection unit is configured to determine that the undetected target object is present when a difference between the second surface and the first surface is equal to or greater than a predetermined threshold. [8] Image processing device according to any one of claims 1 to 7, wherein the detection unit is configured to identify a number of one or more unrecognized target objects that have not been recognized by the recognition processing in the image information, based on information that represents an area of ​​one target object. [9] Image processing device according to any one of claims 1 to 8, which further comprises a storage unit for recognition results configured to store the image information and information about a result of the recognition processing when the detection unit determines that an unrecognized object is present. [10] Image processing device according to claim 1, 2, 3 or 5, further comprising a setting unit that is trained to accept a process for specifying a threshold for determination, which is used for determination by the determination unit, wherein The investigation unit is trained to determine that the undetected object is present if the difference between the second area and the first area is equal to or greater than the specified threshold for detection. [11] Image processing device according to claim 3, further comprising a learning unit configured to perform learning with training data comprising the image information and actual data with respect to a threshold when the binarization processing unit performs the binarization processing, and / or a threshold for detection used for detection by the detection unit, and providing an estimate of a threshold to be applied to the binarization processing when the binarization processing is performed on any input image information, and / or a threshold for detection.

Images