Image processing device, image processing method, computer program, machine learning device, and machine learning method

Abstract

A non-detected workpiece may occur in workpiece detection processing for detecting, on the basis of image data captured by a visual sensor, a workpiece depicted in the image data.　An image processing device 50 comprises: a workpiece detection unit 52 that executes workpiece detection processing for detecting, on the basis of image data captured by a visual sensor 14, a workpiece depicted in the image data; a pixel removal unit 54 that removes, from the image data, pixels that depict the workpiece detected in the workpiece detection processing; a blob detection unit 56 that detects, as a blob, an aggregate of pixels that depict a subject in the image data from which pixels have been removed by the pixel removal unit 54; and a non-detection determination unit 58 that determines, on the basis of the blob detected by the blob detection unit 56, whether a workpiece not detected in the workpiece detection processing is present.

Description

Image processing device, image processing method, computer program, machine learning device, and machine learning method

[0001] The present disclosure relates to an image processing device, an image processing method, a computer program, a machine learning device, and a machine learning method.

[0002] 2. Description of the Related Art A workpiece detection processing technique is known in which a workpiece shown in image data captured by a visual sensor is detected based on the image data (see, for example, Patent Document 1).

[0003] International Publication No. 2022 / 050169

[0004] In the workpiece detection process described above, there may be cases where an undetected workpiece occurs.

[0005] In one aspect of the present disclosure, an image processing device includes a work detection unit that performs a work detection process to detect workpieces that appear in image data captured by a visual sensor based on the image data; a pixel removal unit that removes pixels that depict workpieces detected in the work detection process from the image data; a blob detection unit that detects a collection of pixels that depict a subject as a blob in the image data from which the pixel removal unit has removed pixels; and an undetected determination unit that determines whether or not there are any workpieces that have not been detected in the work detection process based on the blobs detected by the blob detection unit.

[0006] In another aspect of the present disclosure, an image processing method performs a work detection process to detect workpieces that appear in image data captured by a visual sensor based on the image data, removes pixels that depict the workpieces detected in the work detection process from the image data, detects a collection of pixels that depict the subject as a blob in the image data from which the pixels have been removed, and determines whether there are any workpieces that have not been detected in the work detection process based on the detected blobs.

[0007] In yet another aspect of the present disclosure, a machine learning device that learns a change amount to modify parameters used in a work detection process that detects a work that appears in image data captured by a visual sensor based on the image data includes a state observation unit that observes the image data, data of blobs detected as a collection of pixels that depict a subject in image data obtained by removing pixels that depict a work that has been detected in the work detection process from the image data, and judgment data that indicates a judgment result that determines whether or not there is a work that has not been detected in the work detection process based on the detected blobs, as state variables that represent the current state of the environment in which the work detection process is performed, and a learning unit that uses the state variables to learn the change amount by associating it with the judgment result.

[0008] In yet another aspect of the present disclosure, a machine learning method for learning the amount of change to modify parameters used in a work detection process for detecting workpieces captured in image data based on the image data captured by a visual sensor observes the image data, data on blobs detected as a collection of pixels capturing a subject in image data obtained by removing pixels capturing workpieces detected in the work detection process from the image data, and judgment data indicating the judgment result obtained by determining whether or not there are any workpieces that have not been detected in the work detection process based on the detected blobs, as state variables representing the current state of the environment in which the work detection process is performed, and uses the state variables to learn the amount of change by associating it with the judgment result.

[0009] 1 is a schematic diagram of a robot system according to an embodiment. FIG. 1 is a block diagram of the robot system shown in FIG. 1. FIG. 1 is a perspective view showing a workpiece to be worked on and a visual sensor. FIG. 1 is a flowchart showing an example of an operation flow of the robot system shown in FIG. 1. FIG. 2 is a flowchart showing an example of the flow of step S2 in FIG. 4. FIG. 3 is a perspective view of the 3D point cloud image data shown in FIG. 6. FIG. 4 is a diagram showing an example of 2D point cloud image data. FIG. 5 is a diagram showing a state in which a detected workpiece surface is defined in a sensor coordinate system that defines the 3D point cloud image data shown in FIG. 7. FIG. 6 is a diagram showing a schematic diagram of the data structure of a position database. FIG. 7 is a diagram showing a state in which the position database shown in FIG. 10 has been updated so that its order is rearranged. FIG. 8 is a diagram showing 3D point cloud image data from which point clouds overlapping with the workpiece surface shown in FIG. 9 have been removed. FIG. 9 is also a diagram showing a schematic diagram of a blob detected in the 3D point cloud image data shown in FIG. 4. FIG. 9 is a diagram showing a schematic diagram of the workpiece surface shown in FIG. 3, the workpiece coordinate system shown in FIG. 9, and the blob shown in FIG. 13. FIG. 11 is a diagram showing a state in which the status of the position database shown in FIG. 11 has been updated. FIG. 12 is a flowchart showing another example of the flow of step S2 in FIG. 4. 20 is a flowchart showing yet another example of the flow of step S2 in Fig. 4. Fig. 21 is a block diagram showing other functions of the robot system shown in Fig. 1. Fig. 22 is a flowchart showing an example of a learning cycle executed by the machine learning device shown in Fig. 19. Fig. 23 is a flowchart showing an example of the flow of step S32 in Fig. 20.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In various embodiments described below, like elements will be designated by like reference numerals, and duplicated descriptions will be omitted. First, a robot system 10 according to one embodiment will be described with reference to FIGS. 1 and 2. The robot system 10 includes a robot 12, a visual sensor 14, and a control device 16.

[0011] In this embodiment, the robot 12 is a vertical articulated robot and includes a robot base 18, a rotating body 20, a lower arm 22, an upper arm 24, a wrist 26, and an end effector 28. The robot base 18 is fixed to the floor of a work cell or to an automated guided vehicle (AVG). The rotating body 20 is attached to the robot base 18 so as to be rotatable about a vertical axis. The lower arm 22 is attached to the rotating body 20 so as to have a base end rotatable about a horizontal axis. The upper arm 24 has a base end rotatably attached to a tip of the lower arm 22. The wrist 26 is attached to the tip of the upper arm 24 so as to be rotatable about two axes that are perpendicular to each other.

[0012] The end effector 28 is detachably attached to the tip of the wrist 26. In this embodiment, the end effector 28 is a robot hand that can grip a workpiece. The end effector 28 may be a robot hand that has a plurality of openable and closable fingers that grip the workpiece by pinching it between the fingers, or may be a robot hand that has an adsorption unit (negative pressure generator, suction cup, etc.) that can adsorb the surface of the workpiece and grip the workpiece by suction.

[0013] Each component of the robot 12 (robot base 18, rotating body 20, lower arm 22, upper arm 24, wrist 26) is provided with a servo motor 30 ( FIG. 2 ). These servo motors 30 rotate each movable element of the robot 12 (rotating body 20, lower arm 22, upper arm 24, wrist 26, end effector 28) around a drive axis in response to a command from the control device 16. As a result, the robot 12 positions the end effector 28 in any position and orientation.

[0014] The visual sensor 14 captures an image of the workpiece along a line of sight A. In this embodiment, the visual sensor 14 is a three-dimensional visual sensor that incorporates a stereo camera 32 and an image processing processor (not shown) and is capable of detecting the three-dimensional shape of the workpiece. The stereo camera 32 has a pair of image sensors (CMOS, CCD, etc.) and a pair of optical lenses (collimating lenses, focusing lenses, etc.) that respectively guide subject images to the pair of image sensors.

[0015] 1, a robot coordinate system C1 and a tool coordinate system C2 are set for the robot 12. The robot coordinate system C1 is a coordinate system C for automatically controlling the operation of each movable element of the robot 12. In this embodiment, the robot coordinate system C1 is set with respect to the robot base 18 so that its origin is located at the center of the robot base 18 and its z axis is parallel to (specifically, coincides with) the rotation axis of the rotating body 20.

[0016] The tool coordinate system C2 is a coordinate system C that defines the position and posture of the end effector 28 in the robot coordinate system C1. In this embodiment, the tool coordinate system C2 is set with respect to the end effector 28 so that its origin (so-called TCP) is located at the working position of the end effector 28 (in this embodiment, the workpiece gripping position).

[0017] When moving the end effector 28, the control device 16 sets a tool coordinate system C2 in the robot coordinate system C1, and generates commands (position commands, speed commands, torque commands, etc.) to each servo motor 30 of the robot 12 so as to place the end effector 28 at the position and orientation represented by the set tool coordinate system C2. In this way, the control device 16 positions the end effector 28 at any position and orientation in the robot coordinate system C1 by operating the robot 12.

[0018] On the other hand, a sensor coordinate system C3 is set for the visual sensor 14. The sensor coordinate system C3 is a coordinate system C that defines the position and orientation of the visual sensor 14 in the robot coordinate system C1 (i.e., the position and direction of the line of sight A). In this embodiment, the sensor coordinate system C3 is set for the visual sensor 14 so that its origin is located at the center of the stereo camera 32 and its z axis is parallel to (specifically, coincides with) the line of sight A.

[0019] The positional relationship between the robot coordinate system C1 and the sensor coordinate system C3 is known through calibration. Therefore, the coordinates of the robot coordinate system C1 and the coordinates of the sensor coordinate system C3 can be mutually converted via a known transformation matrix (e.g., a homogeneous transformation matrix). Note that the z-axis of the robot coordinate system C1 and the z-axis of the sensor coordinate system C3 may be arranged parallel to each other.

[0020] The control device 16 controls the operations of the robot 12 and the visual sensor 14. As shown in Fig. 2, the control device 16 is a computer having a processor 34, a memory 36, an I / O interface 38, a display device 40, and an input device 42. The processor 34 has a CPU or a GPU, etc., and is communicatively connected to the memory 36, the I / O interface 38, the display device 40, and the input device 42 via a bus 44, and performs arithmetic processing to realize the image processing function described below while communicating with these components.

[0021] The memory 36 includes RAM, ROM, or the like, and temporarily or permanently stores various data. The memory 36 may be configured from a computer-readable non-transitory storage medium, such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium. The I / O interface 38 includes, for example, an Ethernet (registered trademark) port, a USB port, an optical fiber connector, or an HDMI (registered trademark) terminal, and communicates data with external devices via wired or wireless communication under instructions from the processor 34. The servo motors 30 of the robot 12 and the visual sensor 14 are communicatively connected to the I / O interface 38.

[0022] The display device 40 has a liquid crystal display, an organic EL display, or the like, and visibly displays various data under instructions from the processor 34. The input device 42 has a push button, a switch, a keyboard, a mouse, a touch panel, or the like, and receives data input from an operator. The display device 40 and the input device 42 may be integrally incorporated into the housing of the control device 16, or may be connected to the I / O interface 38 as a single computer (PC, etc.) separate from the housing of the control device 16.

[0023] In this embodiment, the processor 34 operates the robot 12 to sequentially pick up a plurality of workpieces W1, W2, W3, W4, and W5 (e.g., packaging materials for containing products) placed as shown in Fig. 3 using the end effector 28 and transport them to a predetermined storage location. The workpieces W1, W2, W3, W4, and W5 have upper surfaces SF1, SF2, SF3, SF4, and SF5 facing vertically upward, respectively.

[0024] The visual sensor 14 is disposed at an imaging position where the workpiece Wi (i=1, 2, 3, 4, 5) can be accommodated within its field of view. The visual sensor 14 may be fixed at a fixed point at a predetermined imaging position, or may be fixed to the wrist 26 or the end effector 28 and positioned at the imaging position by the robot 12.

[0025] The operation of the robot system 10 will be described below with reference to Figure 4. The processor 34 starts the flow of Figure 4 when it receives a work start command from an operator, a higher-level controller, or the computer program PG1. In step S1, the processor 34 sets "n," the number of times the workpiece detection process WD was executed in step S2 (described later), to "0." In step S2, the processor 34 executes the workpiece detection process WD to detect the workpiece W shown in the image data ID captured by the visual sensor 14, based on the image data ID. This step S2 will be described with reference to Figure 5.

[0026] In step S11, the processor 34 operates the visual sensor 14 to capture an image of the workpiece Wi. Specifically, the processor 34 sends an imaging command to the visual sensor 14. In accordance with the imaging command, the visual sensor 14 operates the stereo camera 32 using imaging parameters PR1 to capture a pair of two-dimensional image data ID1 and ID2. The imaging parameters PR1 include an exposure time τ (or shutter speed, aperture value), a focal length F, and an illuminance B of a lighting fixture that illuminates the workpiece Wi during imaging.

[0027] The visual sensor 14 supplies the captured two-dimensional image data ID1 and ID2 to the control device 16. Each of the two-dimensional image data ID1 and ID2 is defined by two-dimensional coordinates on the x-axis and y-axis of a sensor coordinate system C3, and includes pixels PX arranged on the xy plane of the sensor coordinate system C3 as pixels IE that represent a subject including the workpiece Wi.

[0028] In step S12, the processor 34 generates three-dimensional point cloud image data ID3. Specifically, the processor 34 calculates the parallax DP of the stereo camera 32 from the two-dimensional image data ID1 and ID2 acquired in step S11. The processor 34 then calculates the distance d1 from the visual sensor 14 to the workpieces W1 to W5 based on the parallax DP, and generates three-dimensional point cloud image data ID3 based on the distance d1 and the two-dimensional image data ID1 and ID2. The three-dimensional point cloud image data ID3 includes, as pixels IE, a three-dimensional point cloud PT that depicts the visual features (faces, edges, vertices, etc.) of the captured subject.

[0029] Examples of the three-dimensional point cloud image data ID3 are shown in Figures 6 and 7. In this embodiment, the three-dimensional point cloud image data ID3 includes a three-dimensional point cloud PT1 representing the top surface SF1 of the workpiece W1, a three-dimensional point cloud PT2 representing the top surface SF2 of the workpiece W2, a three-dimensional point cloud PT3 representing the top surface SF3 of the workpiece W3, a three-dimensional point cloud PT4 representing the top surface SF4 of the workpiece W4, and a three-dimensional point cloud PT5 representing the top surface SF5 of the workpiece W5. The three-dimensional point cloud image data ID3 is defined by a sensor coordinate system C3, and the three-dimensional point clouds PTi (i = 1, 2, 3, 4, 5) are distributed and arranged within a three-dimensional virtual space defined by the sensor coordinate system C3. Each point constituting the three-dimensional point cloud PTi is represented by coordinates (x, y, z) in the sensor coordinate system C3.

[0030] In step S13, the processor 34 detects the workpiece W that appears in one of the pair of two-dimensional image data ID1 and ID2 captured in step S11, based on that one of the pair of two-dimensional image data ID1 and ID2. An example of the two-dimensional image data ID1 is shown in Fig. 8. The two-dimensional image data ID1 includes a two-dimensional pixel group PX1 that captures the contour of the top surface SF1 of the workpiece W1, a pixel group PX2 that captures the contour of the top surface SF2 of the workpiece W2, a pixel group PX3 that captures the contour of the top surface SF3 of the workpiece W3, a pixel group PX4 that captures the contour of the top surface SF4 of the workpiece W4, and a pixel group PX5 that captures the contour of the top surface SF5 of the workpiece W5.

[0031] Meanwhile, two-dimensional surface models SMi that model the upper surfaces SFi (i = 1, 2, 3, 4, 5) are prepared in advance and stored in memory 36. The processor 34 uses the detection parameters PR2 to sequentially match the surface models SMi to pixel groups PXi included in the two-dimensional image data ID1, and determines the degree of match MD1 between the feature points of the surface models SMi and the feature points of the pixel groups PXi. If the determined degree of match MD1 exceeds a predetermined threshold value α1, the processor 34 determines that the surface models SMi and the pixel groups PXi are highly matched, and thereby can detect the workpiece Wi depicted in the two-dimensional image data ID1.

[0032] The detection parameters PR2 for detecting the workpiece Wi from the two-dimensional image data ID1 include the above-mentioned threshold value α1, the image roughness ρ1 of the surface model SMi or pixel group PXi when matching, the displacement amount δ1 by which the surface model SMi is displaced within the xy plane of the sensor coordinate system C3 when matching, and the window size λ1 that defines the image area when matching.

[0033] Here, when step S13 is executed, it may not be possible to detect all of the workpieces W1 to W5 that appear in the two-dimensional image data ID1 due to reflected light, halation, noise, or the like that appear in the two-dimensional image data ID1. Below, a case will be described in which, in step S13, workpieces W1, W3, W4, and W5 (that is, upper surfaces SF1, SF3, SF4, and SF5) are detected among the workpieces W1 to W5 that appear in the two-dimensional image data ID1, but workpiece W2 (upper surface SF2) is not detected.

[0034] In step S14, the processor 34 determines whether or not the work Wi detected in the immediately preceding step S13 is present. If the processor 34 determines YES, it proceeds to step S15, whereas if the processor 34 determines NO, it ends the flow of step S2 shown in Fig. 5, thereby ending the flow of Fig. 4. In this embodiment, since the works W1, W3, W4, and W5 were detected in step S13, the processor 34 determines YES.

[0035] In step S15, the processor 34 defines the surface SFi of the workpiece Wi detected in the most recent step S13 in the sensor coordinate system C3. Specifically, the processor 34 calculates the scale SC between the two-dimensional image data ID1 and the three-dimensional point cloud image data ID3 based on the above-mentioned parallax DP. Then, the processor 34 reduces or enlarges the surface model SMi corresponding to the top surface SFi detected in step S13 in accordance with the calculated scale SC, and defines it in the sensor coordinate system C3.

[0036] As a result, as shown in Figure 9, in the sensor coordinate system C3 that defines the three-dimensional point cloud image data ID3, the surface model SM1 is defined to overlap with the three-dimensional point cloud PT1, the surface model SM3 is defined to overlap with the three-dimensional point cloud PT3, the surface model SM4 is defined to overlap with the three-dimensional point cloud PT4, and the surface model SM5 is defined to overlap with the three-dimensional point cloud PT5.

[0037] On the other hand, a workpiece coordinate system CWi is set in advance for each surface model SMi so as to have a predetermined positional relationship. Therefore, a workpiece coordinate system CW1 is set for the surface model SM1 defined in the sensor coordinate system C3, a workpiece coordinate system CW3 is set for the surface model SM3, a workpiece coordinate system CW4 is set for the surface model SM4, and a workpiece coordinate system CW5 is set for the surface model SM5. These workpiece coordinate systems CWi are coordinate systems C that define the position and orientation of the workpiece Wi in the robot coordinate system C1 and the sensor coordinate system C3.

[0038] In step S16, the processor 34 detects the position and orientation of the workpiece Wi. Specifically, the processor 34 acquires coordinates Qi in the sensor coordinate system C3 of the workpiece coordinate system CWi (i = 1, 3, 4, 5) shown in Fig. 9, and then converts the coordinates Qi into coordinates Oi in the robot coordinate system C1. These coordinates Oi (xi, yi, zi, wi, pi, ri) represent the position (xi, yi, zi) and orientation (wi, pi, ri) in the robot coordinate system C1 of the workpiece Wi detected in step S13.

[0039] The processor 34 then stores the acquired coordinates Oi in the position database DB. An example of the data structure of the position database DB is shown in FIG. 10. The position database DB shown in FIG. 10 stores the acquired four coordinates O1 (x1, y1, z1, w1, p1, r1), coordinate O3 (x3, y3, z3, w3, p3, r3), coordinate O4 (x4, y4, z4, w4, p4, r4), and coordinate O5 (x5, y5, z5, w5, p5, r5). Note that "No." in the position database DB indicates the work order, and "status" indicates the work status.

[0040] In this embodiment, when multiple workpieces Wi are detected, work is performed on the workpieces Wi in descending order of vertical height (in other words, the z coordinates zi of the coordinates Qi). In the example shown in FIG. 3, the vertical heights are greatest in the order of the top surface SF4 of workpiece W4, the top surface SF3 of workpiece W3, the top surface SF1 of workpiece W1, and the top surface SF5 of workpiece W5. In other words, the z coordinates zi of the coordinates Qi satisfy the relationship z4 > z3 > z1 > z5. The processor 34 updates the position database DB so that the coordinates Qi are arranged in descending order of z coordinates (i.e., vertical heights). The position database DB updated in this manner is shown in FIG. 11. In this manner, the position database DB for the workpiece Wi to be worked on is created.

[0041] In step S17, the processor 34 increments the number of times "n" that the workpiece detection process WD has been executed by "1" (n=n+1), ends step S2 shown in Fig. 5, and proceeds to step S3 in Fig. 4. Thus, in step S2, the processor 34 executes the workpiece detection process WD to detect the workpiece Wi based on the image data ID (ID1, ID2, ID3). Therefore, the processor 34 functions as the workpiece detection unit 52 (Fig. 2) that executes the workpiece detection process WD.

[0042] 4 again, in step S3, the processor 34 removes from the image data ID pixels IE that represent the workpiece Wi detected in the workpiece detection process WD. Specifically, the processor 34 removes the three-dimensional point cloud PTi that overlaps with the surface SFi defined in the sensor coordinate system C3 in the most recent step S15. In this embodiment, as shown in FIG. 9, the surface models SM1, SM3, SM4, and SM5 are defined in the sensor coordinate system C3 so as to overlap with the three-dimensional point clouds PT1, PT3, PT4, and PT5, respectively.

[0043] Therefore, the processor 34 removes the three-dimensional point groups PT1, PT3, PT4, and PT5 from the three-dimensional point cloud image data ID3. At this time, the processor 34 may determine that the three-dimensional point group PTi that is within a predetermined distance d2 from the surface model SMi in the sensor coordinate system C3 overlaps with the surface model SMi and may set the three-dimensional point group PTi as a target for removal. This distance d2 range may be defined, for example, as a range of z coordinates in the robot coordinate system C1 or the sensor coordinate system C3 (for example, a range of z coordinates ±5 from the surface model SMi).

[0044] In this way, the processor 34 removes the three-dimensional point clouds PTi as pixels IE that represent the workpieces Wi detected in the workpiece detection process WD from the three-dimensional point cloud image data ID3. Therefore, the processor 34 functions as a pixel removal unit 54 (FIG. 2) that removes the pixels IE (in this embodiment, the three-dimensional point clouds PT1, PT3, PT4, and PT5) that represent the workpieces Wi detected in the workpiece detection process WD from the image data ID.

[0045] The 3D point cloud image data ID3 after the point cloud has been removed in step S3 is shown in Fig. 12. As a result of step S3, only the 3D point cloud PT2 as pixels IE representing the subject (in this embodiment, the top surface SF2 of the workpiece W2) that was not detected as the workpiece Wi in step S2 remains in the sensor coordinate system C3 as shown in Fig. 12.

[0046] In step S4, the processor 34 detects a collection of pixels IE that represent the subject as a blob BL in the image data ID from which the pixels IE have been removed in the immediately preceding step S3. Specifically, the processor 34 connects points that make up the three-dimensional point cloud PT2 that represents the workpiece W2 as the subject in the three-dimensional point cloud image data ID3 ( FIG. 12 ) from which the three-dimensional point clouds PT1, PT3, PT4, and PT5 have been removed in step S3 to form a collection of the points.

[0047] In this case, the processor 34 may connect, in the sensor coordinate system C3, a plurality of points that are within a predetermined distance from each other among the points that make up the three-dimensional point cloud PT2. As a result, a blob BL is defined in the sensor coordinate system C3 as a collection of points in the three-dimensional point cloud PT2, as shown in Figure 13. A blob surface BLs is defined by three or more points that make up this blob BL.

[0048] In this way, the processor 34 detects a collection of pixels IE (three-dimensional point cloud PT2) that represent the subject (workpiece W2) as a blob BL in the image data ID (three-dimensional point cloud image data ID3) from which the pixels IE have been removed in step S3. Therefore, the processor 34 functions as a blob detection unit 56 (FIG. 2) that detects the blob BL.

[0049] In step S5, the processor 34 determines whether or not there is a workpiece W that was not detected in the workpiece detection process WD in step S2, based on the blob BL detected in the immediately preceding step S4. For example, if the sum of the dimensions (vertical dimension and horizontal dimension, or area) of the multiple blob surfaces BLs defined in the blob BL ( FIG. 13 ) detected in the immediately preceding step S4 exceeds a predetermined threshold value β1, the processor 34 determines that there is an undetected workpiece W (in this embodiment, workpiece W2) (i.e., YES).

[0050] Alternatively, the processor 34 may determine "YES" if the volume of the blob BL exceeds a predetermined threshold value β2. In this way, the processor 34 functions as an undetected determination unit 58 (FIG. 2) that determines whether or not there is any undetected workpiece W based on the blob BL. If the processor 34 determines "YES," the process proceeds to step S6, whereas if the processor 34 determines "NO," the process proceeds to step S7.

[0051] In step S6, the processor 34 executes an undetected process. Step S6 will be described with reference to FIG. 14. In step S21, the processor 34 stores image data ID in the memory 36. Specifically, the processor 34 stores in the memory 36 the two-dimensional image data ID1 and ID2 acquired in the most recent step S11 and the three-dimensional point cloud image data ID3 acquired in the most recent step S12.

[0052] In this manner, in the present embodiment, when it is determined in step S5 that there is an undetected workpiece W, the processor 34 functions as an image storage unit 60 (FIG. 2) that stores image data IDs (ID1, ID2, ID3) in the memory 36. The processor 34 may store one (or two) of the image data ID1 and ID2 and the three-dimensional point cloud image data ID3 in the memory 36.

[0053] In step S22, the processor 34 determines whether the number of times "n" that the workpiece detection process WD has been executed, which is set at this time, is equal to or exceeds a predetermined maximum number of times: n ＭＡＸ (e.g., n ＭＡＸ = 2 times) or (n = n ＭＡＸ If processor 34 determines YES, the process proceeds to step S24, whereas if processor 34 determines NO, the process proceeds to step S23.

[0054] In step S23, the processor 34 adjusts the parameters PR used in the workpiece detection process WD. The parameters PR include the imaging parameters PR1 and the detection parameters PR2. The processor 34 changes at least one of the imaging parameters PR1 (exposure time τ, focal length F, and illuminance B) and the detection parameters PR2 (threshold value α1, image roughness ρ1, displacement amount δ1, and window size λ1).

[0055] At this time, the processor 34 changes the parameters PR (exposure time τ, focal length F, illuminance B, threshold value α1, image roughness ρ1, displacement amount δ1, or window size λ1) by a predetermined change amount Δp. This change amount Δp may be predetermined by an operator, or may be determined randomly by the processor 34, for example, according to a random number table.

[0056] Alternatively, the processor 34 may determine the optimal change amount Δp using the machine learning model LM1. The machine learning model LM1 will be described later. In this manner, the processor 34 functions as a parameter change unit 62 ( FIG. 2 ) that changes the parameters PR (PR1, PR2) when it is determined in step S5 that there is an undetected workpiece W.

[0057] After step S23, the processor 34 returns to step S2 in Fig. 4 and executes the next workpiece detection process WD using the changed parameters PR. As a result, it may be possible to detect the workpiece W2 that was not detected in the previous workpiece detection process WD. Thus, while the processor 34 determines NO in step S22, it changes the parameters PR in step S23 and re-executes the workpiece detection process WD in step S2. Then, when the processor 34 executes step S16 in Fig. 5, it updates the position database DB (Fig. 11) stored in the memory 36 at this time to the latest position database DB.

[0058] Referring again to Fig. 14, if the determination in step S22 is YES, in step S24, the processor 34 determines whether the working conditions CD are satisfied. The working conditions CD determine the positional relationship between the workpiece Wi to be worked on in step S25 (described later) and the blob BL (Fig. 13) detected in step S4. These working conditions CD will be described with reference to Fig. 15. Fig. 15 schematically illustrates the top surface SFi of the workpiece Wi shown in Fig. 3, the workpiece coordinate system CWi shown in Fig. 9, and the blob BL shown in Fig. 13, along with the robot coordinate system C1 and the sensor coordinate system C3.

[0059] In this embodiment, the work conditions CD include a condition CD1 that the workpiece Wi to be worked on is located closer to the visual sensor 14 than the blob BL, and a condition CD2 that the workpiece Wi to be worked on is located a predetermined distance d3 away from the blob BL in a direction DR that intersects with the line of sight A of the visual sensor 14.

[0060] In this embodiment, the top surface SF3 of workpiece W3 and the top surface SF4 of workpiece W4 are located closer to the visual sensor 14 (i.e., the origin of the sensor coordinate system C3) than the top surface SF2 of workpiece W2. Therefore, in the robot coordinate system C1, the coordinate O3 of the workpiece coordinate system CW3 representing the top surface SF3 and the coordinate O4 of the workpiece coordinate system CW4 representing the top surface SF4 are located closer to the visual sensor 14 than the blob BL corresponding to the top surface SF2. Therefore, workpieces W3 and W4 (i.e., the coordinates O3 and O4) satisfy the condition CD1.

[0061] On the other hand, the top surface SF1 of workpiece W1 and the top surface SF5 of workpiece W5 are located farther from the visual sensor 14 than the top surface SF2 of workpiece W2. Therefore, in the robot coordinate system C1, the coordinate O1 of the workpiece coordinate system CW1 representing the top surface SF1 and the coordinate O5 of the workpiece coordinate system CW5 representing the top surface SF5 are located farther from the visual sensor 14 than the blob BL. Therefore, workpieces W1 and W5 (i.e., coordinates O1 and O5) do not satisfy condition CD1.

[0062] On the other hand, the top surface SF5 of workpiece W5 is spaced a distance d3 or more from the top surface SF2 of workpiece W2 in a direction DR that intersects with the line of sight A (in other words, a direction along the xy plane of the sensor coordinate system C3). Therefore, in the robot coordinate system C1, the coordinate O5 of the workpiece coordinate system CW5 that represents the top surface SF5 is spaced a distance d3 from the blob BL in the direction DR. Therefore, the workpiece W5 (coordinate O5) satisfies the condition CD2.

[0063] On the other hand, the distance in the direction DR between the top surface SF1 of workpiece W1 and the top surface SF2 of workpiece W2 is less than distance d3. Therefore, in the robot coordinate system C1, the coordinate O1 of the workpiece coordinate system CW1 representing the top surface SF1 is not separated from the blob BL by the distance d3 or more in the direction DR. Therefore, the workpiece W1 (coordinate O1) does not satisfy the condition CD2.

[0064] In step S24, the processor 34 refers to the position database DB (FIG. 11) stored in the memory 36 at this time point, and determines whether the conditions CD1 and CD2 are met for each of the coordinates Oi of the workpiece Wi stored in the position database DB. Specifically, the processor 34 converts the coordinates Qb in the sensor coordinate system C3 of each point (or each surface) constituting the blob BL detected in the most recent step S4 into coordinates Ob in the robot coordinate system C1.

[0065] Then, based on the coordinates Oi and Ob, the processor 34 determines whether each of the coordinates Oi satisfies at least one of the conditions CD1 and CD2. As described above, in this embodiment, the coordinates O3 and O4 satisfy the condition CD1, and the coordinate O5 satisfies the condition CD2. In this case, the processor 34 maintains the "status" of the coordinates O3, O4, and O5 in the position database DB as "waiting for work." This "waiting for work" means that work can be performed on the work Wi at the coordinate Oi.

[0066] On the other hand, the coordinate O1 of the workpiece W1 does not satisfy both conditions CD1 and CD2. In this case, the processor 34 changes the "status" of the coordinate O1 to "work pending" in the position database DB. This "work pending" means that work on the workpiece Wi at the coordinate Oi cannot be performed. As a result, the position database DB is updated as shown in FIG. 16.

[0067] In step S24, the processor 34 determines YES if at least one of the coordinates Oi stored in the position database DB satisfies at least one of the conditions CD1 and CD2 (in other words, if there is at least one "waiting for work" in the "status" of the position database DB). On the other hand, the processor 34 determines NO if all of the coordinates Oi stored in the position database DB do not satisfy both the conditions CD1 and CD2 (in other words, if all of the "status" of the position database DB are "pending work").

[0068] If the processor 34 determines YES, it proceeds to step S25, but if the processor 34 determines NO, it proceeds to step S27. Thus, in step S24, the processor 34 determines whether the work conditions CD (CD1, CD2) are satisfied. Therefore, the processor 34 functions as a condition determination unit 64 (FIG. 2) that determines whether the work conditions CD are satisfied. Note that the work conditions CD may include any condition that defines the positional relationship between the workpiece Wi and the blob BL, other than the above conditions CD1 and CD2.

[0069] In step S25, the processor 34 performs work on the workpieces Wi. Specifically, the processor 34 refers to the position database DB stored in the memory 36 at this time, and performs work on the workpieces Wi at the coordinates Oi whose "status" is "waiting for work" in the order indicated by "No" (i.e., in descending order of the vertical height of the top surface SFi of the workpieces Wi).

[0070] For example, when the position database DB shown in Fig. 16 is stored, the processor 34 reads out the coordinates O4 (x4, y4, z4, w4, p4, r4) of No. 1, operates the robot 12, positions the end effector 28 (i.e., the tool coordinate system C2) at the coordinates O4, and activates the end effector 28 to pick up the workpiece W4.

[0071] Thereafter, the processor 34 reads out the coordinate O3 of No. 2 and performs the work on the workpiece W3, and then reads out the coordinate O5 of No. 4 and performs the work on the workpiece W5. In this manner, the processor 34 operates the robot 12 to perform the work on the workpiece Wi (in this embodiment, the workpieces W4, W3, and W5) that were determined to satisfy the work condition CD in step S24. Therefore, the processor 34 functions as the work execution unit 66 (FIG. 2) that executes the work.

[0072] After step S25 is completed, in step S26, the processor 34 resets the number of times "n" that the workpiece detection process WD has been executed to "0" and returns to step S2 in Fig. 4. Then, the processor 34 executes the workpiece detection process WD of step S2 again for the unprocessed workpiece Wi whose position has changed as a result of step S25.

[0073] On the other hand, if the determination in step S24 is NO, the processor 34 generates an alarm AL in step S27. For example, the processor 34 generates an alarm AL in the form of a visual or audio message stating, "Work cannot be performed because there is undetected work," and outputs the message through the display device 40 or a speaker (not shown). Thereafter, the processor 34 ends the flow of step S6 shown in FIG. 14, thereby ending the flow of FIG. 4. In this way, the processor 34 functions as the alarm generation unit 68 (FIG. 2) that generates the alarm AL.

[0074] 4 again, if the determination in step S5 is NO, in step S7, the processor 34 functions as the work execution unit 66 and executes work on the work Wi. Specifically, similar to step S25 described above, the processor 34 refers to the position database DB (FIG. 11) stored in the memory 36 at this time, and executes work on the work Wi at the coordinates Oi whose "status" is "waiting for work" in the order indicated by "No."

[0075] As described above, in this embodiment, the processor 34 processes image data IDs (ID1, ID2, ID3) by functioning as the workpiece detection unit 52, pixel removal unit 54, blob detection unit 56, undetection determination unit 58, image storage unit 60, parameter change unit 62, condition determination unit 64, task execution unit 66, and alarm generation unit 68. Therefore, the workpiece detection unit 52, pixel removal unit 54, blob detection unit 56, undetection determination unit 58, image storage unit 60, parameter change unit 62, condition determination unit 64, task execution unit 66, and alarm generation unit 68 constitute the image processing device 50 ( FIG. 2 ).

[0076] In this image processing device 50, the work detection unit 52 executes the work detection process WD (step S2), and the pixel removal unit 54 removes the pixels IE (specifically, the three-dimensional point clouds PT1, PT3, PT4, and PT5) that represent the works W1, W3, W4, and W5 detected in the work detection process WD from the image data ID (specifically, the three-dimensional point cloud image data ID3) (step S3).

[0077] Furthermore, the blob detection unit 56 detects a collection of pixels IE (specifically, the three-dimensional point cloud PT2) representing the subject (workpiece W2) as a blob BL in the image data ID (ID3) from which the pixels IE (PT1, PT3, PT4, and PT5) have been removed by the pixel removal unit 54 (step S4).The undetection determination unit 58 then determines whether or not there is a workpiece W2 that was undetected in the workpiece detection process WD based on the blob BL detected by the blob detection unit 56 (step S5).This configuration makes it possible to automatically and accurately determine whether or not there is a workpiece Wi that was undetected in the workpiece detection process WD, which was executed based on the image data ID captured by the visual sensor 14.

[0078] In addition, in the image processing device 50, the image data ID has three-dimensional point cloud image data ID3 that includes a three-dimensional point cloud PT that captures the subject as pixels IE, and the pixel removal unit 54 removes from the three-dimensional point cloud image data ID3 the three-dimensional point clouds PT1, PT3, PT4, and PT5 that capture the workpieces W1, W3, W4, and W5 detected in the workpiece detection process WD.

[0079] The blob detection unit 56 then detects, as a blob BL, a collection of connected 3D point clouds PT2 that represent the subject (workpiece W2) in the 3D point cloud image data ID3 from which the 3D point clouds PT1, PT3, PT4, and PT5 have been removed by the pixel removal unit 54. With this configuration, the blob BL can be generated from the 3D point cloud PT2, allowing the blob BL to be detected quickly and with high accuracy.

[0080] In the image processing device 50, the image data ID has a pair of two-dimensional image data ID1 and ID2 captured by the stereo camera 32 built into the visual sensor 14, and the workpiece detection unit 52 detects workpieces W1, W3, W4, and W5 reflected in one of the pair of two-dimensional image data ID1 and ID2 based on one (ID1) of the pair of two-dimensional image data ID1 and ID2 in the workpiece detection process WD. Then, the workpiece detection unit 52 defines the surfaces SF1, SF3, SF4, and SF5 (specifically, surface models SM1, SM3, SM4, and SM5) of the detected workpieces W1, W3, W4, and W5 in a sensor coordinate system C3 that defines the three-dimensional point cloud image data ID3 ( FIG. 9 ).

[0081] Then, the pixel removal unit 54 removes the 3D point groups PT1, PT3, PT4, and PT5 that overlap with the surfaces SF1, SF3, SF4, and SF5 defined in the sensor coordinate system C3. With this configuration, the 3D point group PTi that captures the workpiece Wi detected in the workpiece detection process WD can be reliably removed from the 3D point cloud image data ID3, and then the blob BL can be detected. This makes it easier to detect the blob BL.

[0082] Furthermore, in the image processing device 50, when the undetected workpiece W2 is determined to exist by the undetected determination unit 58 (YES in step S5), the parameter change unit 62 changes the parameters PR (PR1, PR2) used in the workpiece detection process WD (step S23). Then, the workpiece detection unit 52 executes the next workpiece detection process WD using the changed parameters PR (step S2 after step S23). This configuration improves the detection accuracy of the next workpiece detection process WD, thereby increasing the possibility of detecting a workpiece Wi that was undetected the previous time.

[0083] Furthermore, in the image processing device 50, the parameter changing unit 62 changes the parameter PR by a predetermined change amount Δp, a randomly determined change amount Δp, or a change amount Δp determined using the machine learning model LM1. When the change amount Δp is determined in advance, the operator can arbitrarily set the change amount Δp by an experimental method or simulation.

[0084] Furthermore, when the parameter change unit 62 randomly determines the amount of change Δp, the processing speed of the loop of adjusting the parameter PR in step S23 and re-executing step S2 (workpiece detection process WD) can be increased. On the other hand, when the parameter change unit 62 determines the amount of change Δp using the machine learning model LM1, the optimal parameter PR can be determined, and the possibility of detecting the workpiece Wi in the subsequent step S2 can be effectively increased.

[0085] Furthermore, in the image processing device 50, the parameters PR include an imaging parameter PR1 used when the visual sensor 14 captures the image data ID (ID1, ID2), or a detection parameter PR2 for detecting the workpiece Wi from the image data ID. According to this configuration, by changing the imaging parameter PR1 or the detection parameter PR2 as the parameter PR, it is possible to effectively improve the detection accuracy of the next workpiece detection process WD.

[0086] Furthermore, in the image processing device 50, when the undetected workpiece W2 is determined to exist by the undetected determination unit 58, the image storage unit 60 stores the image data IDs (ID1, ID2, ID3) in the memory 36 (step S21). With this configuration, for example, an operator can investigate the reason why the workpiece Wi was not detected in the workpiece detection process WD by verifying the image data ID stored in the memory 36. Furthermore, it is also possible to consider optimizing changes to the parameters PR (i.e., the imaging parameters PR1 or the detection parameters PR2) used in the workpiece detection process WD.

[0087] In the image processing device 50, the alarm generating unit 68 generates an alarm AL when the undetected workpiece W2 is determined to be present by the undetected workpiece determining unit 58 (step S27). With this configuration, the operator can easily recognize that an undetected workpiece Wi has occurred in the workpiece detection process WD.

[0088] In addition, in the image processing device 50, when the undetected determination unit 58 determines that there is an undetected work W2, the condition determination unit 64 determines whether or not the work conditions CD (CD1, CD2) that define the positional relationship between the work W1, W3, W4, and W5 to be worked on and the blob BL detected by the blob detection unit 56 are satisfied (step S24).

[0089] The task execution unit 66 then operates the robot 12 to execute the task on the workpieces W3, W4, and W5 that the condition determination unit 64 has determined satisfy the task condition CD (step S25). With this configuration, even if a blob BL occurs, the task on the workpieces W3, W4, and W5 can be executed while avoiding interference with the object represented by the blob BL (e.g., the undetected workpiece W2). This improves work efficiency.

[0090] Furthermore, in the image processing device 50, the work conditions CD include a condition CD1 that the workpieces W1, W3, W4, and W5 to be worked on are located closer to the visual sensor 14 than the blob BL, or a condition CD2 that the workpieces W1, W3, W4, and W5 to be worked on are spaced a predetermined distance d3 in the direction DR from the blob BL. With this configuration, when working on the workpieces W3, W4, and W5, it is possible to reliably avoid interference with the object represented by the blob BL (such as the undetected workpiece W2).

[0091] It should be noted that various modifications can be made to the flow shown in Fig. 4, Fig. 5, or Fig. 14. For example, step S23 may be omitted from the flow of step S6 shown in Fig. 14, and the processor 34 may proceed to step S2 in Fig. 4 when determining NO in step S22. In other words, in this case, the parameter change unit 62 can be omitted from the image processing device 50.

[0092] Furthermore, steps S6 to S21 may be omitted. That is, in this case, the image storage unit 60 may be omitted from the image processing device 50. Furthermore, steps S6 to S27 may be omitted, and the processor 34 may end the flow of FIG. 4 if it determines NO in step S24. That is, in this case, the alarm generation unit 68 may be omitted from the image processing device 50. Furthermore, step S6 may be omitted from the flow shown in FIG. 4, and the processor 34 may end the flow of FIG. 4 if it determines YES in step S5. That is, in this case, the image storage unit 60, the parameter change unit 62, the condition determination unit 64, and the alarm generation unit 68 may be omitted from the image processing device 50.

[0093] Next, another flow of step S2 (workpiece detection processing WD) will be described with reference to Fig. 17. In the flow shown in Fig. 17, the same step numbers are used for processes that are the same as those in the flow of Fig. 5, and duplicated explanations will be omitted. The flow shown in Fig. 17 differs from the flow of Fig. 5 in steps S13', S14', and S16'.

[0094] Specifically, after step S12, in step S13′, the processor 34 detects the workpieces Wi that appear in the three-dimensional point cloud image data ID3 (FIGS. 7 and 8) based on the three-dimensional point cloud image data ID3 generated in the immediately preceding step S12. Here, in this embodiment, three-dimensional workpiece models WMi that model each of the workpieces Wi are prepared in advance and stored in the memory 36.

[0095] As one example, the workpiece model WMi is a three-dimensional CAD model WM1i. The three-dimensional CAD model WM1i is created by an operator using a design support device (such as a CAD / CAM device). As another example, the workpiece model WMi is a three-dimensional point cloud model WM2i. This three-dimensional point cloud model WM2i represents, as a three-dimensional point cloud, the model components (edges, faces, etc.) of the three-dimensional CAD model WM1i when the three-dimensional CAD model WM1i is viewed from various viewpoints, and is created based on the three-dimensional CAD model WM1i.

[0096] In this embodiment, five workpiece models WM1, WM2, WM3, WM4, and WM5 are prepared as three-dimensional CAD models WM11, WM12, WM13, WM14, and WM15, or three-dimensional point cloud models WM21, WM22, WM23, WM24, and WM25. The workpiece models WMi include surface models SMi, and a workpiece coordinate system CWi is set for each workpiece model WMi so as to establish a predetermined positional relationship. The workpiece coordinate system CWi defines the position and orientation of the workpiece model WMi.

[0097] The processor 34 uses the detection parameter PR3 to sequentially match the workpiece model WMi to the three-dimensional point cloud PTi included in the three-dimensional point cloud image data ID3 in the sensor coordinate system C3, and finds the degree of match MD2 between the feature points of the workpiece model WMi and the feature points of the three-dimensional point cloud PTi. When the found degree of match MD2 exceeds a predetermined threshold value α2, the processor 34 determines that the workpiece model WMi and the three-dimensional point cloud PTi are highly matched, and is thereby able to detect the workpiece Wi depicted in the three-dimensional point cloud image data ID3.

[0098] The detection parameters PR3 for detecting the workpiece Wi from the three-dimensional point cloud image data ID3 include the above-mentioned threshold value α2, the image roughness ρ2 of the workpiece model WMi or the three-dimensional point cloud PTi when matching, the displacement amount δ2 by which the workpiece model WMi is displaced within the three-dimensional virtual space defined by the sensor coordinate system C3 when matching, and the window size λ2 that defines the image area when matching.

[0099] As a result of this step S13', the workpieces Wi (e.g., workpieces W1, W3, W4, and W5) that appear in the three-dimensional point cloud image data ID3 are detected. Here, when the three-dimensional point cloud image data ID3 is created in step S12, there are cases where the three-dimensional point cloud PTi that represents the workpieces Wi cannot be sufficiently generated in the three-dimensional point cloud image data ID3 due to the above-mentioned reflected light, halation, noise, or the like. In such cases, in this step S13', it may be impossible to detect all of the workpieces W1 to W5 that appear in the three-dimensional point cloud image data ID3.

[0100] In step S14', the processor 34 determines whether or not there is a workpiece Wi detected from the 3D point cloud image data ID3 in the immediately preceding step S13'. If the processor 34 determines YES, the process proceeds to step S16', whereas if the processor 34 determines NO, the process ends the flow of step S2 shown in Figure 17, thereby ending the flow of Figure 4.

[0101] In step S16', the processor 34 detects the position and orientation of the workpiece Wi. Specifically, the processor 34 acquires coordinates Qi in the sensor coordinate system C3 of the workpiece coordinate system CWi set in the workpiece model WMi matched to the three-dimensional point cloud PTi in the most recent step S13'. Next, the processor 34 converts the coordinates Qi into coordinates Oi in the robot coordinate system C1, and stores the acquired coordinates Oi in the position database DB, as in the above-described embodiment. In this way, the position database DB shown in FIG. 11 is created. After step S16', the processor 34 executes step S17, as in the flow of FIG. 5, and then proceeds to step S3 in FIG. 4.

[0102] Then, in step S3, the processor 34 removes from the image data ID the pixels IE that represent the workpiece W detected in the workpiece detection process WD. Here, when the workpiece Wi was detected in the most recent step S13', a workpiece model WMi that highly matches the three-dimensional point cloud PTi is located in the sensor coordinate system C3. The processor 34 removes the three-dimensional point cloud PTi that overlaps with the surface model SMi included in the workpiece model WMi.

[0103] In addition, when the workpiece model WMi is a three-dimensional point cloud model, the processor 34 may remove points of the three-dimensional point cloud PTi that are within a predetermined distance from each point that constitutes the surface model SMi as overlapping with the surface model SMi. In this way, the processor 34 functions as the workpiece detection unit 52 and executes step S2 shown in Fig. 17, and detects the workpiece Wi that appears in the three-dimensional point cloud image data ID3 by matching the three-dimensional workpiece model WMi to the three-dimensional point cloud image data ID3.

[0104] In addition, when step S23 in FIG. 14 is executed after step S2 shown in FIG. 17, the processor 34 may function as a parameter change unit 62 and change at least one of the exposure time τ, focal length F, and illuminance B as the imaging parameters PR1, and the threshold value α2, image roughness ρ2, displacement amount δ2, and window size λ2 as the detection parameters PR3.

[0105] In the flow of step S2, the flow of FIG. 5 and the flow of FIG. 17 can also be combined. Hereinafter, with reference to FIG. 18, another example of the flow of step S2 will be described. In the flow shown in FIG. 18, the same steps as those in the flow of FIG. 5 or FIG. 17 are numbered the same, and duplicated descriptions will be omitted. In the flow shown in FIG. 18, the processor 34 selectively executes a first detection process WD1 (steps S13 to S16) for detecting a workpiece Wi from two-dimensional image data ID1 as in the flow of FIG. 5, and a second detection process WD2 (steps S13' to S16') for detecting a workpiece Wi from three-dimensional point cloud image data ID3 as in the flow of FIG. 17.

[0106] Specifically, after step S12, in step S18, the processor 34 determines whether or not a second detection process WD2 is necessary based on the three-dimensional point cloud image data ID3 generated in the immediately preceding step S12. Here, there may be a workpiece Wi that cannot be detected by the first detection process WD1 executed in step S13.

[0107] For example, if the workpieces Wi have a relatively complex shape or are piled up in various positions, it becomes difficult to detect the workpieces Wi from the two-dimensional image data ID1. In such cases, it is necessary to execute the second detection process WD2 in step S13'. On the other hand, the first detection process WD1 executed in step S13 has the advantage of being able to improve the accuracy of detecting the workpieces Wi from the image data ID1 and shorten the time required for detection compared to the second detection process WD2.

[0108] Therefore, in this embodiment, in step S18, the processor 34 analyzes the three-dimensional point cloud PTi included in the three-dimensional point cloud image data ID3 generated in the immediately preceding step S12, and determines whether or not it is necessary to execute the second detection process WD2. As an example, the processor 34 estimates the top surface SFi of the workpiece Wi from the three-dimensional point cloud PTi. At this time, the processor 34 may detect a blob BL' as a collection of the three-dimensional point cloud PTi using a method similar to that of step S4 described above, and estimate the top surface SFi based on a blob surface BLs' defined in the blob BL'.

[0109] Next, the processor 34 calculates the normal directions Ni of the estimated upper surfaces SFi, and calculates the angles θi of each normal direction Ni from the z-axis of the sensor coordinate system C3 (or the robot coordinate system C1). Then, the processor 34 calculates the angle θi of each normal direction Ni from the z-axis of the sensor coordinate system C3 (or the robot coordinate system C1) when at least one of the calculated angles θi is smaller than a predetermined threshold θ ｔｈ If this is the case (in other words, if the top surface SFi of at least one workpiece Wi is inclined with respect to the vertical direction), the determination may be YES.

[0110] As another example, the processor 34 may determine whether or not it is necessary to execute the second detection process WD2 by analyzing the distribution of the three-dimensional point cloud PTi in the sensor coordinate system C3 (or the robot coordinate system C1). For example, the processor 34 may compare the distribution of the three-dimensional point cloud PTi in the z-axis direction of the sensor coordinate system C3 (or the robot coordinate system C1) with a predetermined reference distribution (such as a normal distribution), and determine YES if the two differ.

[0111] As yet another example, the processor 34 may use a machine learning model LM2 to determine whether or not it is necessary to perform the second detection process WD2. This machine learning model LM2 represents the correlation between the 3D point cloud image data ID3 and whether or not the second detection process WD2 is necessary. Such a machine learning model LM2 can be constructed, for example, by providing various 3D point cloud image data ID3 and label information LB indicating whether or not the second detection process WD2 is necessary as a training dataset DS to the machine learning device LD. Note that the processor 34 may be configured to function as the machine learning device LD and import the training dataset DS to generate the machine learning model LM2.

[0112] If the machine learning model LM2 has been generated, in step S18, the processor 34 inputs the 3D point cloud image data ID3 generated in the immediately preceding step S12 to the machine learning model LM2. The machine learning model LM2 then outputs label information LB that has a correlation with the input 3D point cloud image data ID3. The processor 34 can determine from the label information LB output by the machine learning model LM2 whether or not it is necessary to execute the second detection process WD2.

[0113] In this way, the processor 34 determines whether or not it is necessary to execute the second detection process WD2 based on the three-dimensional point cloud image data ID3. If the processor 34 determines YES, the process proceeds to step S13', and, similar to the flow of Figure 17, executes the second detection process WD2 including steps S13', S14', and S16', and then proceeds to step S17. On the other hand, if the processor 34 determines NO, the process proceeds to step S13, and, similar to the flow of Figure 5, executes the first detection process WD1 including steps S13 to S16, and then proceeds to step S17.

[0114] Thus, in this embodiment, the processor 34 functions as the work detection unit 52 to execute step S2 shown in FIG. 18, and selectively executes one of the first detection process WD1 (steps S13 to S16) and the second detection process WD2 (steps S13' to S16') depending on the three-dimensional point cloud image data ID3 generated in step S12.

[0115] According to this configuration, if the workpieces Wi have complex shapes or are piled up in various positions, the second detection process WD2 can be executed to reliably detect the workpieces Wi. Conversely, if the workpieces Wi have simple shapes or are uniform in position, the first detection process WD1 can be executed to improve the detection accuracy of the workpieces Wi and shorten the detection time.

[0116] In addition, when executing step S23 in FIG. 14 after executing step S13 in FIG. 18, the processor 34 may function as a parameter change unit 62 and change at least one of the exposure time τ, focal length F, and illuminance B as the imaging parameters PR1, and the threshold value α1, image roughness ρ1, displacement amount δ1, and window size λ1 as the detection parameters PR2.

[0117] On the other hand, when step S23 in FIG. 14 is executed after step S13′ shown in FIG. 18, the processor 34 may function as a parameter change unit 62 and change at least one of the exposure time τ, focal length F, and illuminance B as the imaging parameters PR1, and the threshold value α2, image roughness ρ2, displacement amount δ2, and window size λ2 as the detection parameters PR3.

[0118] In the above embodiment, the working conditions CD include the conditions CD1 and CD2. However, the working conditions CD may include only one of the conditions CD1 and CD2. In addition, the operator can operate the input device 42 to set the threshold values ​​α1, α2, β1, β2, or θ ｔｈ The processor 34 inputs the thresholds α1, α2, β1, β2, or θ through the input device 42. ｔｈ may be accepted.

[0119] In addition, when generating the above-mentioned three-dimensional point cloud image data ID3, the processor 34 may remove point clouds that depict subjects other than the work Wi (for example, the floor surface on which the work Wi is placed, or a container that contains the work Wi, etc.) from the three-dimensional point cloud image data ID3.

[0120] Furthermore, each time the processor 34 executes the above-described step S16, the processor 34 may accumulate position databases DB1, DB2, DB3, ... DBm (m = 1, 2, 3, ...) storing the acquired coordinates Oi in the memory 36. In this case, when the processor 34 executes the above-described steps S24 and S25, the processor 34 may execute the steps S24 and S25 based on the position database DBm that stores the largest number of coordinates Oi.

[0121] 4 in accordance with a computer program PG1 stored in the memory 36. The functions of the image processing device 50 (workpiece detection unit 52, pixel removal unit 54, blob detection unit 56, undetection determination unit 58, image storage unit 60, parameter change unit 62, condition determination unit 64, task execution unit 66, and alarm generation unit 68) executed by the processor 34 may be functional modules realized by the computer program PG1.

[0122] Next, other functions of the robot system 10 will be described with reference to Fig. 19. In this embodiment, the processor 34 functions as an image processing device 50 including a workpiece detection unit 52, a pixel removal unit 54, a blob detection unit 56, a non-detection determination unit 58, and a parameter modification unit 62. In addition, the processor 34 functions as a machine learning device 70 including a state observation unit 72, a learning unit 74, and a decision-making unit 76.

[0123] The machine learning device 70 learns a change amount Δp that changes the parameters PR (for example, exposure time τ, focal length F, illuminance B, threshold value α1, image roughness ρ1, displacement amount δ1, or window size λ1) used in the workpiece detection process WD. In the present embodiment, the processor 34 functions as the machine learning device 70 and advances learning of the change amount Δp by repeatedly executing the learning cycle shown in FIG.

[0124] In step S31, the processor 34 functions as the decision-making unit 76, refers to the value function EQ at this time point, and randomly selects a change amount Δp as an action to be taken in the current state indicated by the state variable SV observed by the state observing unit 72. The processor 34 then generates a command value CΔ for the selected change amount Δp, and changes the parameters PR (exposure time τ, focal length F, illuminance B, threshold value α1, image roughness ρ1, displacement amount δ1, or window size λ1) in accordance with the command value CΔ.

[0125] In step S32, the processor 34 functions as the state observing unit 72 and observes the state variable SV of the current state. This step S32 will be described with reference to Fig. 21. As shown in Fig. 21, after starting step S32, the processor 34 executes the next workpiece detection process WD using the parameter PR after it has been changed in accordance with the command value CΔ in the most recent step S31.

[0126] Specifically, as the work detection process WD, the processor 34 sequentially executes the above-mentioned steps S11 (imaging the work Wi), S12 (generating three-dimensional point cloud image data ID3), S13 (detecting the work Wi), S15 (defining the surface SFi), S16 (detecting the position and orientation of the work Wi), S3 (removing the pixel IE), and S4 (detecting the blob BL).

[0127] Then, in step S37, the processor 34 functions as the state observing unit 72 and acquires the state variable SV. For example, the processor 34 acquires, as the state variable SV, at least one of the two-dimensional image data ID1 and ID2 captured in step S11 and the three-dimensional point cloud image data ID3 acquired in step S12 as the image data ID captured by the visual sensor 14.

[0128] In addition, the processor 34 takes in, as state variables SV, the three-dimensional point cloud image data ID3 including the blob BL shown in FIG. 13 and at least one of the coordinates in the sensor coordinate system C3 (or the robot coordinate system C1) of the points or blob surfaces BLs that make up the blob BL, as data of the blob BL detected in step S4 in FIG. 21.

[0129] Meanwhile, in this step S37, similar to the above-mentioned step S5, the processor 34 determines whether or not there is any workpiece Wi that has not been detected in the workpiece detection process WD based on the blob BL detected in the immediately preceding step S4. Then, the processor 34 takes in, as a state variable SV, determination data DD that indicates the determination result of whether or not there is any workpiece Wi that has not been detected.

[0130] For example, when the processor 34 determines that there is an undetected workpiece Wi, it acquires determination data DD1 indicating the determination result that an undetected workpiece Wi has occurred. On the other hand, when the processor 34 determines that there is no undetected workpiece Wi, it acquires determination data DD2 indicating the determination result that an undetected workpiece Wi has not occurred.

[0131] In this way, the processor 34 observes, as state variables SV representing the current state of the environment in which the work detection process WD is performed, the image data ID (ID1, ID2, and ID3), data on blobs BL detected as a collection of pixels IE that represent the subject in the image data ID (for example, the three-dimensional point cloud image data ID3 shown in Figure 12) from which the pixels IE that represent the work Wi detected in the work detection process WD have been removed, and judgment data DD1 or DD2 indicating the judgment result based on the detected blobs BL as to whether or not there is any work Wi that has not been detected in the work detection process WD.

[0132] The processor 34 may also take in, as the state variable SV, the data of the surface SF of the workpiece Wi defined in the sensor coordinate system C3 in step S15 in Fig. 21 (for example, the coordinates of the sensor coordinate system C3 or the three-dimensional point cloud image data ID3 shown in Fig. 9). The processor 34 may also take in, as the state variable SV, the coordinates Qi or Oi of the workpiece Wi acquired in step S16 in Fig. 21. The processor 34 may also take in, as the state variable SV, the three-dimensional point cloud image data ID3 (Fig. 12) from which the three-dimensional point cloud PTi has been removed in step S3 in Fig. 21.

[0133] 20 again, in step S33, the processor 34 determines whether or not an undetected workpiece Wi has occurred. Specifically, if the processor 34 has acquired judgment data DD1 (a judgment result indicating the presence of an undetected workpiece) as the state variable SV in the most recent step S37, the processor 34 determines YES and proceeds to step S35. On the other hand, if the processor 34 has acquired judgment data DD2 (a judgment result indicating the absence of an undetected workpiece) as the state variable SV in the most recent step S37, the processor 34 determines NO and proceeds to step S34.

[0134] In step S34, the processor 34 functions as the learning unit 74, calculates a positive reward +R, and applies the calculated positive reward +R to the update equation for the function EQ. On the other hand, if the determination in step S33 is YES, in step S35 the processor 34 calculates a negative reward −R and applies it to the update equation for the function EQ. Note that in step S35, instead of giving the negative reward −R, the processor 34 may apply a reward R=0 to the update equation for the function EQ.

[0135] In this way, in this embodiment, the processor 34 functions as a reward calculation unit 78 (FIG. 19) that calculates a reward related to the determination result (or the determination data DD1 or DD2). By providing a reward R according to the determination result in this way, the learning of the change amount Δp is guided in the direction of selecting an action that will eliminate undetected workpieces Wi.

[0136] In step S36, processor 34 functions as learning unit 74 and updates value function EQ based on reward R. Thus, in this embodiment, processor 34 functions as function update unit 80 ( FIG. 19 ) that updates function EQ that represents the value of change Δp. By repeating the learning cycle of steps S31 to S36, processor 34 iteratively updates value function EQ, and as a result, can progress learning of change Δp associated with the determination result.

[0137] The value function EQ obtained as a result of such learning constitutes the machine learning model LM1. After learning the value function EQ as the machine learning model LM1, the processor 34 executes the flow shown in FIG. 4 and, in the above-mentioned step S23, determines the change amount Δp using the machine learning model LM1 (value function EQ). Specifically, in step S23, the processor 34 applies the image data ID1 or ID2 acquired in the most recent step S11 and the 3D point cloud image data ID3 acquired in the most recent step S12 to the machine learning model LM1 as state variables SV. The processor 34 also applies the data of the blob BL detected in the most recent step S4 to the machine learning model LM1 as state variables SV.

[0138] As a result, the machine learning model LM1 outputs a change Δp corresponding to the input state variable SV. The processor 34 generates a command value CΔ for the change Δp output by the machine learning model LM1, and changes the parameter PR by the change Δp in accordance with the command value CΔ. In this way, the processor 34 functions as the machine learning device 70, learns the machine learning model LM1 (value function EQ), and can obtain an optimal solution for the change Δp from the state variable SV obtained during execution of the flow in FIG. 4.

[0139] As described above, in the machine learning device 70, the state observation unit 72 observes the image data ID (ID1, ID2, ID3), data of the blob BL detected as a collection of pixels IE that represent the subject in the image data ID from which the pixels IE that represent the work Wi detected in the work detection process WD have been removed (three-dimensional point cloud image data ID3, coordinates of the points or blob surface BLs that constitute the blob BL), and judgment data DD1 or DD2 that indicates the judgment result of determining whether or not there is any work that has not been detected in the work detection process WD based on the detected blob BL, as a state variable SV that represents the current state of the environment in which the work detection process WD is performed (step S32).

[0140] Then, the learning unit 74 uses the state variable SV to learn the change amount Δp in association with the judgment result (steps S34 and S35). With this configuration, in the above-mentioned step S23, it is possible to automatically learn the optimal change amount Δp for preventing the occurrence of undetected workpieces Wi in the next workpiece detection process WD. This reduces the number of workpieces Wi that go undetected in the workpiece detection process WD, thereby improving work efficiency.

[0141] Furthermore, in the machine learning device 70, the learning unit 74 has a reward calculation unit 78 that calculates a reward R associated with the determination result, and a function update unit 80 that updates a function EQ that represents the value of the change amount Δp using the reward R. With this configuration, the learning of the change amount Δp can be guided in the direction of selecting an action that eliminates undetected work Wi, thereby making the learning cycle more efficient.

[0142] Furthermore, in the machine learning device 70, the decision-making unit 76 generates a command value CΔ for the amount of change Δp based on the learning results of the learning unit 74. Then, the state observing unit 72 observes, as the state variable SV in the next learning cycle, the judgment data DD1 or DD2 obtained when the next workpiece detection process WD is executed using the parameters PR after being changed according to the command value CΔ. With this configuration, the amount of change Δp can be automatically changed and trial learning tasks can be repeated, thereby automating the learning cycle.

[0143] 21, steps S13' and S16' (i.e., second detection processing WD2) shown in Fig. 17 may be executed instead of steps S13, S15, and S16 (i.e., first detection processing WD1). In this case, in step S31, the processor 34 selects, as the parameter PR, for example, the exposure time τ, the focal length F, the illuminance B, the threshold value α2, the image roughness ρ2, the displacement amount δ2, or the change amount Δp for changing the window size λ2.

[0144] 20 in accordance with a computer program PG2 stored in the memory 36. Furthermore, the functions of the machine learning device 70 (the state observing unit 72, the learning unit 74 (the reward calculating unit 78, the function updating unit 80), and the decision making unit 76) executed by the processor 34 may be functional modules realized by the computer program PG2.

[0145] In the above-described embodiment, the processor 34 may acquire distance image data ID4 instead of the three-dimensional point cloud image data ID3. The distance image data ID4 includes pixels PX' arranged two-dimensionally within the xy plane of the sensor coordinate system C3 as pixels IE that represent the subject, including the workpiece Wi. Each pixel PX' represents the subject, including information about the distance d1 from the visual sensor 14 to the workpiece Wi, with brightness corresponding to the distance d1.

[0146] The processor 34 can acquire distance image data ID4 based on the pair of two-dimensional image data ID1 and ID2 captured by the stereo camera 32 and the above-mentioned parallax DP. When the distance image data ID4 has been acquired, in step S3 in Fig. 4, pixels PXi' of the distance image data ID4 that overlap with the surface SFi defined in the sensor coordinate system C3 in the most recent step S15 (e.g., pixel PX1' capturing the top surface SF1 of workpiece W1, pixel PX3' capturing the top surface SF3 of workpiece W3, pixel PX4' capturing the top surface SF4 of workpiece W4, and pixel PX5' capturing the top surface SF5 of workpiece W5) are removed (e.g., brightness is set to zero).

[0147] Then, in step S4, the processor 34 connects the points that make up the pixel PX2' that represents the subject (e.g., workpiece W2) in the range image data ID4 after pixel PXi' has been removed, and detects the collection of points as a blob BL. At this time, the processor 34 may binarize the pixel PX2' that represents the subject (workpiece W2) and the other pixels PX' (e.g., set the brightness of pixel PX2' to "1" and the brightness of the other pixels PX' to "0"), and form a blob BL from the pixel PX2'.

[0148] In the above embodiment, the visual sensor 14 is described as including the stereo camera 32. However, the present invention is not limited to this, and the visual sensor 14 may be, for example, a scanner-type visual sensor including a light-emitting unit that emits electromagnetic waves (visible light, infrared light, or laser light) and a light-receiving unit that receives reflected waves from a subject.

[0149] In the above embodiment, the end effector 28 is a robot hand, and the robot 12 performs the task of picking up a workpiece Wi with the end effector 28. However, the present invention is not limited to this. The end effector 28 may be, for example, a welding torch, a cutting tool, a laser processing head, or a paint applicator, and the robot 12 may be configured to perform tasks such as welding, cutting, laser processing, or coating with the end effector 28. Furthermore, the robot 12 is not limited to a vertical articulated robot, and may be any type of robot, such as a horizontal articulated robot or a parallel link robot.

[0150] Although the present disclosure has been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0151] As described above, the present disclosure describes the following aspects: (Aspect 1) An image processing device 50 including a workpiece detection unit 52 that executes a workpiece detection process WD that detects workpieces Wi that appear in image data ID (ID1, ID2, ID3, ID4) captured by a visual sensor 14, based on the image data ID, a pixel removal unit 54 that removes pixels IE that represent the workpieces Wi detected in the workpiece detection process WD from the image data ID, a blob detection unit 56 that detects, as blobs BL, a collection of pixels IE that represent a subject in the image data ID from which the pixels IE have been removed by the pixel removal unit 54, and an undetection determination unit 58 that determines whether or not there are any workpieces W2 that have not been detected in the workpiece detection process WD, based on the blobs BL detected by the blob detection unit 56. (Aspect 2) The image data ID has three-dimensional point cloud image data ID3 that includes a three-dimensional point cloud PT that represents the subject as pixels IE, and the pixel removal unit 54 removes the three-dimensional point cloud PTi that represents the work Wi detected in the work detection process WD from the three-dimensional point cloud image data ID3, and the blob detection unit 56 detects a collection of connected three-dimensional point clouds PT that represent the subject W2 as a blob BL in the three-dimensional point cloud image data ID3 from which the pixel removal unit 54 has removed the three-dimensional point cloud PTi. (Aspect 3) The image processing device 50 according to Aspect 1 or 2, wherein the image data ID further includes a pair of two-dimensional image data ID1 and ID2 captured by a stereo camera 32 built into the visual sensor 14, and the workpiece detection unit 52, in the workpiece detection process WD, detects a workpiece Wi reflected in one ID1 of the pair of two-dimensional image data ID1 and ID2 based on the one ID1, and defines a surface SFi of the detected workpiece Wi in a sensor coordinate system C3 that defines the three-dimensional point cloud image data ID3, and the pixel removal unit 54 removes the three-dimensional point cloud PTi that overlaps with the surface SFi defined in the sensor coordinate system C3. (Aspect 4) The image processing device 50 according to any of Aspects 1 to 3, further including a parameter change unit 62 that changes parameters PR used in the workpiece detection process WD when the undetection determination unit 58 determines that there is an undetected workpiece W2, and the workpiece detection unit 52 executes the next workpiece detection process WD using the changed parameters PR.(Aspect 5) The image processing device 50 according to Aspect 4, wherein the parameter change unit 62 changes the parameter PR by a predetermined amount of change Δp, a randomly determined amount of change Δp, or an amount of change Δp determined using the machine learning model LM1. (Aspect 6) The image processing device 50 according to Aspect 4 or 5, wherein the parameter PR includes an imaging parameter PR1 used when the visual sensor 14 captures the image data ID, or a detection parameter PR2 for detecting the workpiece Wi from the image data ID. (Aspect 7) The image processing device 50 according to any one of Claims 1 to 6, further comprising an image storage unit 60 that stores the image data ID in a memory when the undetection determination unit 58 determines that there is an undetected workpiece W2. (Aspect 8) The image processing device 50 according to any one of Claims 1 to 7, further comprising an alarm generation unit 68 that generates an alarm AL when the undetection determination unit 58 determines that there is an undetected workpiece W2. (Aspect 9) The image processing device 50 according to any one of Aspects 1 to 8, further comprising: a condition determination unit 64 that, when the undetection determination unit 58 determines that there is an undetected workpiece W2, determines whether or not a work condition CD defining a positional relationship between a workpiece Wi to be worked on and a blob BL detected by the blob detection unit 56 is satisfied, and a work execution unit 66 that operates the robot 12 that performs the work on the workpiece Wi, and executes the work on the workpiece Wi that the condition determination unit 64 determines to satisfy the work condition CD. (Aspect 10) The image processing device 50 according to Aspect 9, wherein the work condition CD includes a condition CD1 that the workpiece Wi to be worked on is located closer to the visual sensor 14 than the blob BL, or a condition CD2 that the workpiece Wi to be worked on is separated from the blob BL by a predetermined distance d3 in a direction DR that intersects with the line of sight A of the visual sensor 14. (Mode 11) An image processing method in which, based on image data ID captured by a visual sensor 14, a work detection process WD is executed to detect a work Wi that appears in the image data ID, pixels IE that depict the work Wi detected by the work detection process WD are removed from the image data ID, a collection of pixels IE that depict a subject is detected as a blob BL in the image data ID from which the pixels IE have been removed, and based on the detected blob BL, it is determined whether there is any work Wi that has not been detected by the work detection process WD.(Aspect 12) A computer program PG1 that causes a processor 34 to execute the image processing method according to Aspect 11. (Aspect 13) A machine learning device 70 that learns a change amount Δp to change a parameter PR used in a workpiece detection process WD that detects a workpiece Wi that appears in image data ID captured by a visual sensor 14, based on the image data ID, the machine learning device 70 comprising: a state observation unit 72 that observes, as state variables SV that represent a current state of an environment in which the workpiece detection process WD is performed, data of blobs BL detected as a collection of pixels IE that depict a subject in the image data ID obtained by removing from the image data ID pixels IE that depict the workpiece Wi detected by the work detection process WD, and determination data DD (DD1, DD2) that indicate a determination result obtained by determining whether or not there is a workpiece W2 that has not been detected by the work detection process WD, based on the detected blobs BL; and a learning unit 74 that uses the state variables SV to learn the change amount Δp in association with the determination result. (Aspect 14) The machine learning device 70 according to aspect 13, wherein the learning unit 74 has a reward calculation unit 78 that calculates a reward R associated with the determination result, and a function update unit 80 that updates a function EQ that represents the value of the change amount Δp using the reward R. (Aspect 15) The machine learning device 70 according to aspect 13 or 114, further comprising a decision making unit 76 that generates a command value CΔ for the change amount Δp based on the learning result by the learning unit 74, and wherein the state observation unit 72 observes, as a state variable SV in the next learning cycle, the determination data DD when the next workpiece detection process WD is executed using the parameters PR after being changed according to the command value CΔ.(Mode 16) A machine learning method for learning a change amount Δp to modify a parameter PR used in a work detection process WD that detects a work Wi that appears in image data ID based on the image data ID captured by a visual sensor 14, wherein the machine learning method observes the image data ID, data on blobs BL detected as a collection of pixels IE that depict a subject in the image data ID from which pixels IE that depict the work Wi detected by the work detection process WD have been removed, and judgment data DD that indicates the judgment result of determining whether or not there is a work Wi that has not been detected by the work detection process WD based on the detected blobs BL as a state variable SV that represents the current state of the environment in which the work detection process WD is performed, and uses the state variable SV to learn the change amount Δp in association with the judgment result.

[0152] REFERENCE SIGNS LIST 10 Robot system 12 Robot 14 Visual sensor 16 Control device 34 Processor 50 Image processing device 52 Work detection unit 54 Pixel removal unit 56 Blob detection unit 58 Non-detection determination unit 60 Image storage unit 62 Parameter change unit 64 Condition determination unit 66 Work execution unit 68 Alarm generation unit 70 Machine learning device 72 State observation unit 74 Learning unit 78 Reward calculation unit 80 Function update unit 76 Decision-making unit

Claims

1. An image processing device comprising: a work detection unit that executes a work detection process to detect workpieces that appear in image data captured by a visual sensor based on the image data; a pixel removal unit that removes pixels that depict the workpieces detected in the work detection process from the image data; a blob detection unit that detects, as blobs, a collection of pixels that depict a subject in the image data from which the pixels have been removed by the pixel removal unit; and an undetection determination unit that determines whether or not there are any workpieces that have not been detected in the work detection process based on the blobs detected by the blob detection unit.

2. The image processing device of claim 1, wherein the image data has three-dimensional point cloud image data containing three-dimensional point clouds that represent a subject as the pixels, the pixel removal unit removes the three-dimensional point clouds that represent the workpiece detected in the workpiece detection process from the three-dimensional point cloud image data, and the blob detection unit detects, as the blob, the collection of connected three-dimensional point clouds that represent the subject in the three-dimensional point cloud image data from which the pixel removal unit has removed the three-dimensional point clouds.

3. The image processing device according to claim 2, wherein the image data further includes a pair of two-dimensional image data captured by a stereo camera built into the visual sensor, wherein the work detection unit, in the work detection process, detects a work appearing in one of the pair of two-dimensional image data based on the one piece of the two-dimensional image data, and defines the surface of the detected work in a sensor coordinate system that defines the three-dimensional point cloud image data, and wherein the pixel removal unit removes the three-dimensional point cloud that overlaps with the surface defined in the sensor coordinate system.

4. An image processing device as described in claim 1, further comprising a parameter change unit that changes the parameters used in the work detection process when the undetected work determination unit determines that there is an undetected work, and the work detection unit executes the next work detection process using the changed parameters.

5. The image processing device according to claim 4, wherein the parameter change unit changes the parameters by a predetermined amount of change, a randomly determined amount of change, or an amount of change determined using a machine learning model.

6. An image processing device according to claim 4, wherein the parameters include imaging parameters used when the visual sensor captures the image data, or detection parameters for detecting the workpiece from the image data.

7. The image processing device according to claim 1, further comprising an image storage unit that stores the image data in a memory when the undetected workpiece determination unit determines that there is an undetected workpiece.

8. The image processing device according to claim 1, further comprising an alarm generating unit that generates an alarm when the undetected workpiece is determined to exist by the undetected workpiece determining unit.

9. An image processing device as described in claim 1, further comprising: a condition determination unit that, when the undetection determination unit determines that there is an undetected workpiece, determines whether or not a work condition that defines the positional relationship between the workpiece to be worked on and the blob detected by the blob detection unit is satisfied; and a work execution unit that operates a robot that performs work on the workpiece and performs the work on the workpiece that the condition determination unit determines to satisfy the work condition.

10. An image processing device as described in claim 9, wherein the work conditions include a condition that the work to be worked on is located closer to the visual sensor than the blob, or a condition that the work to be worked on is separated from the blob by a predetermined distance in a direction intersecting the line of sight of the visual sensor.

11. An image processing method comprising: performing a workpiece detection process based on image data captured by a visual sensor to detect workpieces that appear in the image data; removing from the image data pixels that depict the workpieces detected in the workpiece detection process; detecting a group of pixels that depict a subject as a blob in the image data from which the pixels have been removed; and determining whether or not there are any workpieces that have not been detected in the workpiece detection process based on the detected blobs.

12. A computer program causing a processor to execute the image processing method according to claim 11.

13. A machine learning device that learns a change amount to change parameters used in a work detection process that detects a workpiece that appears in image data based on the image data captured by a visual sensor, the machine learning device comprising: a state observation unit that observes the image data, data on blobs detected as a collection of pixels that depict a subject in image data obtained by removing from the image data pixels that depict the workpiece detected in the work detection process, and judgment data that indicates a judgment result that determines whether or not there is a workpiece that has not been detected in the work detection process based on the detected blobs, as state variables that represent the current state of the environment in which the work detection process is performed; and a learning unit that uses the state variables to learn the change amount in association with the judgment result.

14. The machine learning device described in claim 13, wherein the learning unit has: a reward calculation unit that calculates a reward related to the judgment result; and a function update unit that uses the reward to update a function that represents the value of the amount of change.

15. The machine learning device of claim 13, further comprising a decision-making unit that generates a command value for the amount of change based on the learning results of the learning unit, and the state observation unit observes the judgment data when the next work detection process is executed using the parameters after being changed according to the command value as the state variable in the next learning cycle.

16. A machine learning method for learning a change amount to modify parameters used in a workpiece detection process that detects a workpiece that appears in image data based on the image data captured by a visual sensor, the machine learning method observing the image data, data on blobs detected as a collection of pixels that depict a subject in image data obtained by removing from the image data pixels that depict the workpiece detected in the workpiece detection process, and judgment data indicating the judgment result of determining whether or not there is a workpiece that has not been detected in the workpiece detection process based on the detected blobs, as state variables that represent the current state of the environment in which the workpiece detection process is performed, and using the state variables to learn the change amount in association with the judgment result.

Images