Device for setting detection range for detecting workpiece, robot system, method, and computer program

Abstract

It is needed to shorten a cycle time of work by reducing arithmetic processing required for a search process for searching for a workpiece appearing in image data.　A device 50 for setting detection ranges 132, 136, 138 for detecting a workpiece 200 in image data 100n+1 is provided with: a difference acquisition unit 52 for obtaining a difference δn between a distance d included in first image data 100n obtained by imaging the workpiece 200 by a visual sensor 14 before work, and a distance d included in second image data 100n+1 obtained by imaging the workpiece 200 by the visual sensor 14 after the work; a region identification unit 56 for identifying an image region 130 in which the difference δn has been generated in a sensor coordinate system C3 which is set to the visual sensor 14 and which defines the image data 100; and a range delimiting unit 58 for delimiting a first detection range 132 in the sensor coordinate system C3 so as to include the image region 130 identified by the region identification unit 56.

Description

Apparatus, robot system, method, and computer program for setting a detection range for detecting a workpiece

[0001] The present disclosure relates to an apparatus, a robot system, a method, and a computer program for setting a detection range in which a workpiece is to be detected in image data.

[0002] 2. Description of the Related Art A robot system is known that detects a workpiece that appears in image data captured by a visual sensor capable of measuring the distance to the subject, and performs a predetermined task on the detected workpiece (see, for example, Patent Document 1).

[0003] JP 2019-198950 A

[0004] There is a demand to shorten the work cycle time by reducing the calculation processing required for the search process to locate workpieces captured in image data.

[0005] In one aspect of the present disclosure, an apparatus for setting a detection range in which a workpiece should be detected based on image data captured by a visual sensor capable of measuring the distance to a subject includes a difference acquisition unit that calculates the difference between the distance contained in first image data obtained by capturing an image of the workpiece by the visual sensor before work and the distance contained in second image data obtained by capturing an image of the workpiece by the visual sensor after work, an area identification unit that is set on the visual sensor and identifies the image area in which the difference occurs in a sensor coordinate system that defines the image data, and a range definition unit that defines a first detection range in the sensor coordinate system so as to encompass the image area identified by the area identification unit.

[0006] In another aspect of the present disclosure, a method for setting a detection range in which a workpiece should be detected based on image data captured by a visual sensor capable of measuring the distance to a subject includes a processor determining the difference between a first distance contained in first image data of the workpiece captured by the visual sensor before the work and a second distance contained in second image data of the workpiece captured by the visual sensor after the work, identifying the image area where the difference occurs in a sensor coordinate system set on the visual sensor and defining the image data, and defining a first detection range in the sensor coordinate system so as to encompass the identified image area.

[0007] 1 is a schematic diagram of a robot system according to an embodiment. It is a block diagram of the robot system shown in FIG. 1. It is a flowchart showing an example of an operation flow of the robot system shown in FIG. 1. It shows an example of image data of a workpiece captured by the visual sensor shown in FIG. 1 before work. It is a flowchart showing the flow of step S3 in FIG. 3. It shows a state in which a workpiece model is matched to a workpiece captured in image data in a search process. It shows an example of the data structure of a detection result list. It shows an example of image data of a workpiece captured by the visual sensor shown in FIG. 1 after work. It is a flowchart showing the flow of step S3 in FIG. 3. It is a flowchart showing an example of the flow of step S4 in FIG. 3. It is a diagram for explaining the difference in distance contained in image data captured before and after work when the image data is range image data. It is a diagram for explaining the difference in distance contained in image data captured before and after work when the image data is three-dimensional point cloud image data. It shows an example of difference image data. It shows an example of difference image data generated when the image data is range image data. It shows another example of difference image data generated when the image data is range image data. It shows an example of difference image data generated when the image data is three-dimensional point cloud image data. It shows another example of difference image data generated when the image data is three-dimensional point cloud image data. 22. An example of an image area identified in step S24 in FIG. 9 is schematically shown. An example of a first detection range defined to include the image area shown in FIG. 17 is shown. An example of a first detection range defined to include the image area shown in FIG. 18 is shown, which is set in image data captured after work. An example of a state in which a search process is executed within the first detection range shown in FIG. 19, and a workpiece model is matched to the workpiece shown in the image data is shown. An example of a detection result list updated in step S28 in FIG. 9 is shown. A flowchart showing another example of the operation flow of the robot system shown in FIG. 1. A flowchart showing an example of the flow of step S4 in FIG. 22. An example of a detection result list updated after step S7 in FIG. 22. A flowchart showing another example of the flow of step S4 in FIG. 3. An example of a detection range defined in step S25 in FIG. 25 overlaps with a portion of the occupation area of ​​the workpiece model. An example of a state in which a second detection range is further defined is shown.Fig. 4 is a flowchart showing yet another example of the flow of step S4 in Fig. 3. Fig. 5 shows a state in which a third detection range is further defined. Fig. 6 shows another example of the first detection range.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In various embodiments described below, like elements will be designated by like reference numerals, and duplicated descriptions will be omitted. First, a robot system 10 according to one embodiment will be described with reference to FIGS. 1 and 2. The robot system 10 includes a robot 12, a visual sensor 14, and a control device 16.

[0009] In this embodiment, the robot 12 is a vertical articulated robot and includes a robot base 18, a rotating body 20, a lower arm 22, an upper arm 24, a wrist 26, and an end effector 28. The robot base 18 is fixed to the floor of a work cell or on an automated guided vehicle (AGV). The rotating body 20 is mounted on the robot base 18 so as to be rotatable about a vertical axis.

[0010] The base end of the lower arm 22 is rotatably attached to the rotating body 20 around a horizontal axis, and the base end of the upper arm 24 is rotatably attached to the tip of the lower arm 22. The wrist 26 has a wrist base 26a attached to the tip of the upper arm 24 so as to be rotatable around two axes that are perpendicular to each other, and a wrist flange 26b attached to the wrist base 26a so as to be rotatable around a wrist axis A1.

[0011] The end effector 28 is detachably attached to the wrist flange 26 b. The end effector 28 is, for example, a robot hand capable of gripping the workpiece 200, a welding torch for welding the workpiece 200, or a laser processing head for laser processing the workpiece 200, and performs a predetermined operation WK (workpiece handling, welding, laser processing, etc.) on the workpiece 200.

[0012] Each component of the robot 12 (robot base 18, rotating body 20, lower arm 22, upper arm 24, and wrist 26) is provided with a servo motor 30 (FIG. 2). These servo motors 30 rotate each drive shaft of the robot 12 in response to commands from the control device 16. As a result, the robot 12 can move and position the end effector 28 at any desired position.

[0013] 1, a robot coordinate system C1 and a tool coordinate system C2 are set for the robot 12. The robot coordinate system C1 is a coordinate system for automatically controlling the operation of each movable component of the robot 12 (i.e., the rotating body 20, the lower arm 22, the upper arm 24, the wrist base 26a, the wrist flange 26b, and the end effector 28). In this embodiment, the robot coordinate system C1 is fixed relative to the robot base 18 so that its origin is located at the center of the robot base 18 and its z-axis is parallel to (specifically, coincides with) the rotation axis of the rotating body 20.

[0014] On the other hand, the tool coordinate system C2 is a coordinate system that defines the position of the end effector 28 in the robot coordinate system C1. In this embodiment, the tool coordinate system C2 is set with respect to the end effector 28 so that its origin (so-called TCP) is located at the working position of the end effector 28 (i.e., the workpiece gripping position, the welding position, or the laser light emission port) and its z axis is parallel to (specifically, coincides with) the wrist axis A1.

[0015] When moving the end effector 28, the control device 16 sets a tool coordinate system C2 in the robot coordinate system C1 and generates commands to the servo motors 30 of the robot 12 to position the end effector 28 at a position represented by the set tool coordinate system C2. In this manner, the control device 16 can position the end effector 28 at any position in the robot coordinate system C1. In this document, the term "position" may refer to both position and orientation.

[0016] The visual sensor 14 captures an image of the workpiece 200 in the container 202. Specifically, the visual sensor 14 is a three-dimensional visual sensor having an imaging sensor (CMOS, CCD, etc.) and an optical lens (collimator lens, focus lens, etc.) that guides an image of the object to the imaging sensor. The visual sensor 14 is configured to capture an image of the object (e.g., the workpiece 200 and the container 202) along an optical axis A2 and measure the distance d to the object.

[0017] In this embodiment, the visual sensor 14 is fixed to a movable component of the robot 12 (e.g., the end effector 28 or the wrist flange 26b) and is moved by the robot 12. The visual sensor 14 may be fixed at a predetermined imaging position Pi where the workpiece 200 can be accommodated within its field of view. The visual sensor 14 supplies image data of the captured image to the control device 16.

[0018] A sensor coordinate system C3 is set for the visual sensor 14. The sensor coordinate system C3 is a coordinate system that defines the position of the visual sensor 14 in the robot coordinate system C1 (i.e., the position and direction of the optical axis A2). In this embodiment, the sensor coordinate system C3 is set with respect to the visual sensor 14 so that its origin is located at the center of the image sensor of the visual sensor 14 and its z axis is parallel to (specifically, coincides with) the optical axis A2.

[0019] The sensor coordinate system C3 defines the coordinates of each pixel of the image data (or the image sensor) captured by the visual sensor 14. The positional relationship between the sensor coordinate system C3 and the robot coordinate system C1 is known through calibration, and therefore the coordinates of the sensor coordinate system C3 and the coordinates of the robot coordinate system C1 can be mutually converted via a known transformation matrix MX (e.g., a homogeneous transformation matrix).

[0020] The control device 16 controls the operations of the robot 12 and the visual sensor 14. As shown in Fig. 2, the control device 16 is a computer having a processor 32, a memory 34, an I / O interface 36, a display device 38, an input device 40, etc. The processor 32 has a CPU or a GPU, etc., and is communicatively connected to the memory 34, the I / O interface 36, the display device 38, and the input device 40 via a bus 42, and performs calculations to realize various functions described below while communicating with these components.

[0021] The memory 34 includes RAM, ROM, or the like, and temporarily or permanently stores various data. The memory 34 may be configured from a computer-readable non-transitory storage medium, such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium. The I / O interface 36 includes, for example, an Ethernet (registered trademark) port, a USB port, an optical fiber connector, or an HDMI (registered trademark) terminal, and communicates data with external devices via wired or wireless connection under instructions from the processor 32. The servo motors 30 of the robot 12 and the visual sensor 14 are communicatively connected to the I / O interface 36.

[0022] The display device 38 has a liquid crystal display, an organic EL display, or the like, and visibly displays various data under instructions from the processor 32. The input device 40 has a push button, a switch, a keyboard, a mouse, a touch panel, or the like, and receives data input from an operator. The display device 38 and the input device 40 may be integrally incorporated into the housing of the control device 16, or may be connected to the I / O interface 36 as a single computer (PC, etc.) separate from the housing of the control device 16.

[0023] In this embodiment, the robot system 10 uses the visual sensor 14 to capture images of the workpieces 200 piled up in a container 202, and detects the workpieces 200 captured in the image data based on the image data captured by the visual sensor 14. The robot system 10 then performs a predetermined task WK on the detected workpieces 200 using the robot 12. The operation flow of the robot system 10 will be described below with reference to Figure 3. The processor 32 of the control device 16 starts the flow shown in Figure 3 when it receives a task start command from an operator, a higher-level controller, or a computer program PG.

[0024] In step S1, the processor 32 images the workpieces 200 using the visual sensor 14. Specifically, the processor 32 operates the robot 12 to position the visual sensor 14 at an imaging position Pi where the workpieces 200 can be accommodated within its field of view. The processor 32 then transmits an imaging command to the visual sensor 14, and in accordance with the imaging command, the visual sensor 14 images the workpieces 200 piled up in the container 202. The image data 100 captured by the visual sensor 14 is ｎ An example is shown in FIG.

[0025] As an example, the visual sensor 14 may receive image data 100 ｎ The distance image data 100A is captured as follows. The distance image data 100A represents the distance d to the captured object (in this case, the workpiece 200) by the shade of color, and the visual feature of the object captured by each pixel PX of the distance image data 100A is represented by the coordinates (x, y) in the sensor coordinate system C3, and also contains information about the distance d. Therefore, the visual feature of the object captured by each pixel PX of the distance image data 100A is represented by the three-dimensional coordinate Qs(x, y) in the sensor coordinate system C3. ｓ , y ｓ , z ｓ )

[0026] As another example, the visual sensor 14 may receive image data 100 ｎThe three-dimensional point cloud image data 100B is captured as follows. The three-dimensional point cloud image data 100B represents the visual features of the captured subject as a point cloud dispersed in a virtual space defined by the sensor coordinate system C3. Each point O constituting this point cloud has information on the distance d, and is represented by a three-dimensional coordinate Qs(x ｓ , y ｓ , z ｓ Of the above-mentioned coordinates Qs, the coordinate value x ｓ and y ｓ is the distance d from the visual sensor 14 (specifically, the image sensor) to the visual feature of the subject in a direction perpendicular to the optical axis A2. １ while the coordinate value z ｓ is the distance d from the visual sensor 14 to the visual feature of the subject in the direction of the optical axis A2 ２ This shows:

[0027] In this way, the image data 100 captured by the visual sensor 14 ｎ The visual features (edges, surfaces, etc.) of the workpiece 200 shown in the distance image data 100A or the three-dimensional point cloud image data 100B are represented by coordinates Qs (x ｓ , y ｓ , z ｓ ) The image data 100 shown in FIG. ｎ shows only a portion of the workpiece 200 in the container 202. The processor 32 receives the image data 100 captured by the visual sensor 14. ｎ is acquired from the visual sensor 14.

[0028] The processor 32 processes the captured image data 100 ｎ For example, image processing may be performed to remove the visual features of the environmental object SR, such as the container 202 and the floor of the work cell, from the image data captured by the visual sensor 14 located at the imaging position Pi. _ＳＲ The image data 100 captured in step S1 may be stored in advance in the memory 34. Then, the processor 32 ｎ From the coordinate Qs _ＳＲAs a result, as shown in FIG. 4, the image data 100 in which only the workpiece 200 is displayed is removed. ｎ can be generated.

[0029] In step S2, the processor 32 acquires image data 100 of the workpiece 200 captured by the visual sensor 14 before the work WK is performed in step S5 described below. ｎ Here, as will be described later, the processor 32 repeatedly executes the flow of steps S1 to S6 until it determines YES in step S6, and each time it executes step S1, the visual sensor 14 detects a plurality of image data 100. ｎ (n=1, 2, 3, . . . ) are sequentially imaged.

[0030] If step S2 is executed for the first time, step S5 has not yet been executed, so the image data 100 captured before step S5 is used. ０ Therefore, in this case, the processor 32 determines NO in this step S2. On the other hand, when the (n+1)th (n≧1) step S2 is executed, the image data 100 is not included in the nth step S1 executed before the work WK is executed in the nth step S5. ｎ Therefore, in this case, the processor 32 determines YES in step S2. If the processor 32 determines YES or NO, the process proceeds to step S3, and if the processor 32 determines NO, the process proceeds to step S4.

[0031] In step S3, the processor 32 executes a first detection scheme. This step S3 will be described with reference to Fig. 5. In step S11, the processor 32 executes a first detection scheme. ｎ A search process SP is executed to search for the workpiece 200 appearing in (n=1).

[0032] For example, in the most recent step S1, the image data 100 shown in FIG. ｎ In this case, the processor 32 processes the image data 100 ｎThe workpiece model 200M, which is a model of the workpiece 200, is matched to the visual features of the workpiece 200 (i.e., a distance image or a point cloud) captured in the image. Ｃ CAD model 200M Ｃ is a three-dimensional CAD model, which is created in advance by an operator using a CAD device (not shown).

[0033] As another example, the workpiece model 200M may be a CAD model 200M Ｃ A point cloud model 200M that represents the model components (edges, faces, etc.) of the object as a point cloud (or normals). Ｐ In this case, the processor 32 may receive the CAD model 200M from the CAD device. Ｃ and then image-generating the CAD model 200M according to a predetermined image generation algorithm. Ｃ By adding a point cloud to the model component, the point cloud model 200M Ｐ may be generated.

[0034] The processor 32 performs a search process SP by scanning the image data 100 ｎ The processor 32 sequentially matches the workpiece model 200M in various postures with the visual features of the workpiece 200 shown in the image data 100. Then, the processor 32 calculates a score SC as a result of matching each time the processor 32 matches the workpiece model 200M. ｎ The higher (or lower) the score SC, the more similar the two are.

[0035] Then, when the calculated score SC exceeds a predetermined threshold, the processor 32 ｎ It is determined that the visual features of the image data 100M match highly with the workpiece model 200M. ｎ 1 shows a state in which the workpiece model 200M is highly matched to the workpieces 200A, 200B, 200C, 200D, 200E, and 200F among the multiple workpieces 200 shown in the image.

[0036] Here, a workpiece coordinate system C4 is set for each orientation of the workpiece model 200M, which defines the position and orientation of the workpiece model 200M. The processor 32 calculates the coordinates Qs in the sensor coordinate system C3 of the workpiece coordinate system C4 of each workpiece model 200M matched to the workpieces 200A to 200F. _Ａ , Qs _Ｂ , Qs _Ｃ , Qs _Ｄ , Qs _Ｅ and Qs _Ｆ are obtained respectively.

[0037] These coordinates Qs _Ａ ~Qs _Ｆ are expressed by the coordinates Qs (x ｓ , y ｓ , z ｓ , w ｓ , p ｓ , r ｓ ) Coordinate Qs _Ａ ~Qs _Ｆ In this case, the coordinate (x ｓ , y ｓ , z ｓ ) indicates the origin of the workpiece coordinate system C4 in the sensor coordinate system C3 (i.e., the positions of the workpieces 200A to 200F), and the coordinates (w ｓ , p ｓ , r ｓ ) indicate the direction of each axis of the workpiece coordinate system C4 relative to the sensor coordinate system C3 (that is, the attitude of the workpieces 200A to 200F) (so-called yaw, pitch, and roll).

[0038] Then, the processor 32 calculates the coordinate Qs of the acquired sensor coordinate system C3. _Ａ , Qs _Ｂ , Qs _Ｃ , Qs _Ｄ , Qs _Ｅ and Qs _Ｆ is converted into coordinates Qr in the robot coordinate system C1 using the known transformation matrix MX. _Ａ , Qr _Ｂ , Qr _Ｃ , Qr _Ｄ , Qr _Ｅ and Qr _Ｆ These coordinates Qr _Ａ ~Qr _ＦEach of the coordinates Qr(x ｒ , y ｒ , z ｒ , w ｒ , p ｒ , r ｒ )

[0039] In this manner, the processor 32 processes the image data 100 ｎ By matching the workpiece model 200M to the image data 100 ｎ A search process SP is executed to search for the workpiece 200 appearing in the image data 100. ｎ The processor 32 then acquires the position data PD (i.e., coordinates Qs and Qr) of the detected workpieces 200A to 200F. Therefore, the processor 32 functions as a workpiece search unit 44 (FIG. 2) that executes the search process SP.

[0040] In step S12, the processor 32 determines whether or not a workpiece 200 was detected in the immediately preceding step S11. If at least one workpiece 200 was detected, the processor 32 determines YES and proceeds to step S13, whereas if no workpiece 200 was detected, the processor 32 determines NO and proceeds to step S6 in FIG. 3.

[0041] In step S13, the processor 32 creates a detection result list 110. An example of the data structure of the detection result list 110 is shown in FIG. 7. In the example shown in FIG. 7, a column 112 labeled "No" indicates the identification numbers (or work sequence) of the workpieces 200A to 200F detected by the search process SP. Also, a column 114 labeled "Position Data" indicates the position data PD of the workpieces 200A to 200F acquired in the search process SP (specifically, the coordinates Qr in the robot coordinate system C1). _Ａ ~Qr _Ｆ ) is shown. A column 116 labeled "Status" indicates the progress of the operation WK for the workpieces 200A to 200F identified by the identification number "No."

[0042] In this way, in step S13, the processor 32 assigns identification numbers to the workpieces 200A-200F detected in the search process SP, and creates a detection result list 110 that associates the identification numbers with the position data PD of the workpieces 200A-200F and label information indicating the progress of the workpieces WK for the workpieces 200A-200F (e.g., "waiting for work," "work successful," "work failed," "work on hold," etc.). The processor 32 stores the created detection result list 110 in the memory 34. The processor 32 uses this detection result list 110 to manage the position data PD and progress of the workpieces WK for the detected multiple workpieces 200A-200F. After step S13, the processor 32 proceeds to step S5 in FIG. 3.

[0043] The processor 32 may assign identification numbers "No." to the detected workpieces 200A-200F in accordance with a predetermined priority order. For example, the processor 32 may assign identification numbers "No." to the detected workpieces 200A-200F in order of the magnitude of the z coordinate value of the coordinate Qr acquired as the position data PD (i.e., in order of increasing vertical height). Therefore, in this case, the multiple pieces of position data PD are arranged in order of priority order (e.g., z coordinate value) in the detection result list 110. The processor 32 may also display the created detection result list 110 on the display device 38.

[0044] 3 again, in step S5, the processor 32 executes the operation WK on the workpiece 200. Specifically, the processor 32 refers to the detection result list 110 stored in the memory 34 at this time, and acquires the position data PD for which the "status" is "waiting for operation" and the identification number "No." For example, in the example of FIG. 7, the processor 32 acquires the position data PD: coordinates Qr of the workpiece 200A to which the identification number "No. 1" has been assigned. _Ａ Then, the processor 32 obtains the coordinate Qr _Ａ The robot 12 is operated based on the above, and the end effector 28 performs the work WK on the workpiece 200A.

[0045] When the work WK on the workpiece 200A is properly completed, the processor 32 changes the "status" of the position data PD of the identification number "No. 1" to "work successful" in the detection result list 110. The processor 32 may also delete the position data PD of the identification number "No. 1" that has been set to "work successful" from the detection result list 110. In this way, the processor 32 updates the detection result list 110.

[0046] In step S6, the processor 32 determines whether the work WK has been completed for all the workpieces 200 in the container 202. If the processor 32 determines YES, it ends the flow shown in FIG. 3, but if the processor 32 determines NO, it returns to step S1. Then, the processor 32 executes step S1 for the (n+1)th time, and the visual sensor 14 outputs the image data 100. ｎ＋１ This image data 100 ｎ＋１ The (second image data) is image data of the workpiece 200 in the container 202 captured by the visual sensor 14 after the work WK was performed in the most recent step S5.

[0047] Here, when an operation WK is performed on one workpiece 200 in step S5, the arrangement of the workpieces 200 around that one workpiece 200 may change (so-called load collapse). Below, we will explain a case where, in step S5, the robot 12 performs workpiece handling as an operation WK on the workpiece 200A (FIG. 4) and removes the workpiece 200A from the container B.

[0048] In this case, the image data 100 captured in step S1 for the (n+1)th time is ｎ＋１ An example of the image data 100 shown in FIG. ｎ＋１ In this case, the image data 100 shown in FIG. ｎ The workpiece 200A is removed, and the positions of the surrounding workpieces 200B, 200C, 200G, and 200H are changed. On the other hand, the positions of the workpieces 200 (e.g., workpieces 200D, 200E, and 200F) that are located away from the removed workpiece 200A are not changed.

[0049] Referring again to FIG. 3, step S1 is executed for the (n+1)th time to obtain image data 100. ｎ＋１ If the image data 100 is acquired, the processor 32 determines YES in the subsequent step S2, and proceeds to step S4. Step S4 will be described below with reference to FIG. 9. In step S21, the processor 32 acquires the image data 100 ｎ and the distance d included in the image data 100 ｎ＋１ The difference δ from the distance d included in ｎ Here, the image data 100 ｎ (FIG. 4) is image data captured before the n-th operation of step S5, while image data 100 ｎ＋１ FIG. 8 shows image data captured after the nth operation of step S5 has been performed.

[0050] As an example, if the image data 100 is distance image data 100A, the processor 32 calculates the difference δ ｎ As the image data 100 ｎ The z-coordinate value of the sensor coordinate Qs of the visual feature captured by one pixel PX: z ｓ_ｎ and image data 100 corresponding to the one pixel PX. ｎ＋１ The z-coordinate value of the visual feature captured by the pixel PX of the sensor coordinate Qs: z ｓ_ｎ＋１ Difference δ ｎ =z ｓ_ｎ＋１ -z ｓ_ｎ The difference δ ｎ The concept of will be explained with reference to FIG.

[0051] As shown in FIG. 10A, the coordinate Qs (x ｓ , y ｓ As shown in FIG. 10B, the image data 100 before the processing is ｎ In this case, the visual feature PX of the workpiece 200 that this pixel PX represents is ｎ The z coordinate value of the sensor coordinate system C3 is z ｓ_ｎ On the other hand, the image data 100 after the operation ｎ＋１ In this case, the visual feature PX of the workpiece 200 that this pixel PX represents is ｎ＋１ The z coordinate value of the sensor coordinate system C3 is z ｓ_ｎ＋１(>z ｓ_ｎ ) The processor 32 calculates the difference δ ｎ δ ｎ =z ｓ_ｎ＋１ -z ｓ_ｎ In this way, the processor 32 calculates the image data 100 ｎ and 100 ｎ＋１ For each of all pixels PX in ｎ Ask for.

[0052] Next, the difference δ to be calculated when the image data 100 is the three-dimensional point cloud image data 100B ｎ The concept of this will be explained with reference to FIG. 11. First, the processor 32 receives image data 100 ｎ and 100 ｎ＋１ Specifically, as shown in FIG. 11A, the processor 32 quantizes the image data 100 ｎ and 100 ｎ＋１ A unit pixel area PE having a unit area is defined on the xy plane of a sensor coordinate system C3 that defines the pixel PX. This unit pixel area PE is a virtual area that simulates a pixel PX, and a plurality of unit pixel areas PE are defined that are aligned in a grid pattern in the x-axis and y-axis directions of the sensor coordinate system C3, just like the pixel PX.

[0053] Next, the processor 32 generates the image data 100 before processing. ｎ 11A, three points O are present in one unit pixel region PE. Then, for each unit pixel region PE, the processor 32 determines a representative point Or based on the points O present inside the unit pixel region PE. ｎ The following is established.

[0054] More specifically, the processor 32 calculates the z coordinate values ​​of three points O in the unit pixel area PE in the sensor coordinate system C3: z ｓ The average value z ｓAVE (= 1 / 3 Σz ｓ Then, the processor 32 calculates the calculated average value z ｓAVE z coordinate value: z ｓ_ｎ The representative point Or ｎis defined in the sensor coordinate system C3 as shown in FIG. ｎ The coordinate Qs of the sensor coordinate system C3 ｎ Qs ｎ (x ｓ , y ｓ , z ｓ_ｎ ) is expressed as (z ｓ_ｎ =z ｓAVE ). The representative point Or ｎ Among the coordinates Qs, the coordinate (x ｓ , y ｓ ) is determined at an arbitrary position within the unit pixel area PE (for example, the center position of the unit pixel area PE).

[0055] In a similar manner, the processor 32 generates the processed image data 100 ｎ＋１ In this case, based on the point O existing in each unit pixel area PE, the representative point Or ｎ＋１ Define this representative point Or ｎ＋１ The coordinate Qs of the sensor coordinate system C3 ｎ＋１ Qs ｎ＋１ (x ｓ , y ｓ , z ｓ_ｎ＋１ ) The processor 32 calculates the difference δ ｎ δ ｎ =z ｓ_ｎ＋１ -z ｓ_ｎ In this way, the processor 32 calculates the image data 100 ｎ and 100 ｎ＋１ For each of all the unit pixel areas PE defined in ｎ and Or ｎ＋１ The difference δ ｎ Ask for.

[0056] In step S21, the processor 32 uses the above-described method to generate the image data 100. ｎ Distance d included in (specifically, distance d ２ = coordinate value z ｓ_ｎ ) and image data 100 ｎ＋１ Distance d (coordinate value z ｓ_ｎ＋１ ) and the difference δ ｎ Therefore, the processor 32 calculates the difference δ ｎ The difference acquisition unit 52 (FIG. 2) calculates the difference.

[0057] In step S22, the processor 32 determines whether the difference δ is equal to or greater than a predetermined threshold δth as a result of the immediately preceding step S21. ｎ Specifically, the processor 32 determines whether or not the difference δ calculated for each pixel PX or unit pixel area PE in the immediately preceding step S21 has occurred. ｎ The absolute value of (= |z ｓ_ｎ＋１ -z ｓ_ｎ |) with a threshold value δth, and find at least one difference δ ｎ The absolute value of is greater than or equal to the threshold value δth (|δ ｎ |≧δth), the answer is YES.

[0058] On the other hand, the processor 32 detects a difference δ that is equal to or greater than the threshold δth. ｎ If no difference δ occurs, the result is determined to be NO, and the process proceeds to step S29. ｎ Alternatively, if the number N of pixels PX or unit pixel areas PE in which the above error has occurred is equal to or greater than a predetermined threshold value Nth, the result may be YES.

[0059] In step S23, the processor 32 generates the difference image data 120. The difference image data 120 is the difference δ calculated in the most recent step S21. ｎ 12 is a schematic diagram of an example of the differential image data 120. As shown in FIG. 12, the differential image data 120 includes the difference δ calculated for each pixel PX or unit pixel area PE in the most recent step S21. ｎ is displayed as an image in the sensor coordinate system C3 (i.e., an area given brightness or luminance).

[0060] 13 and 14, the data structure of the difference image data 120 when the image data 100 is distance image data 100A will be described. The pixel PX shown in FIG. 13(a) corresponds to the pixel PX shown in FIG. 10(a). In the example shown in FIG. 13, the processor 32 calculates the difference δ in the difference image data 120. ｎ For the pixel PX where the pixel (FIG. 10B) occurs, as shown in FIG. 13B, the coordinate value z ｓA difference pixel 122 having a predetermined brightness (or luminance) is assigned at the position where .gamma.=0.

[0061] On the other hand, in the example shown in FIG. 14, the processor 32 calculates the difference δ in the difference image data 120. ｎ For the pixel PX where the pixel (FIG. 10B) occurs, as shown in FIG. 14B, the coordinate value z ｓ But, z ｓ_ｎ ≦z ｓ ≦z ｓ_ｎ＋１ In other words, the difference pixel 124 is added to the visual feature PX ｎ Visual feature PX ｎ＋１ The difference δ in the z-axis direction of the sensor coordinate system C3 extends to ｎ The range will be equivalent to the above.

[0062] Next, the data structure of the differential image data 120 when the image data 100 is three-dimensional point cloud image data 100B will be described with reference to Figures 15 and 16. The unit pixel area PE shown in Figure 15(a) corresponds to the unit pixel area PE shown in Figure 11(b). In the example shown in Figure 15, the processor 32 calculates the difference δ in the differential image data 120. ｎ 11C) has occurred in the unit pixel area PE, as shown in FIG. 15B, the coordinate value z ｓ A difference pixel 126 having a predetermined brightness is assigned at the position of .times. ...

[0063] On the other hand, in the example shown in FIG. 16, the processor 32 calculates the difference δ in the difference image data 120. ｎ 11C) has occurred in the unit pixel area PE, as shown in FIG. 16B, the coordinate value z ｓ But, z ｓ_ｎ ≦z ｓ ≦z ｓ_ｎ＋１ The difference pixel 128 is given over the range of the representative point Or shown in FIG. ｎ From the representative point Or ｎ＋１ The range will be up to.

[0064] The difference pixels 126 and 128 shown in FIGS. 15 and 16 are the representative points Or shown in FIGS. 11(b) and 11(c). ｎ and Orｎ＋１ and the representative point Or ｎ and Or ｎ＋１ The coordinate Qs (x ｓ , y ｓ As described above, the processor 32 calculates the difference δ calculated in the most recent step S21. ｎ are expressed in the sensor coordinate system C3 as difference pixels 122, 124, 126, or 128. Thus, the processor 32 functions as the image data generator 54 (FIG. 4) that generates the difference image data 120.

[0065] In step S24, the processor 32 calculates the difference δ ｎ Specifically, the processor 32 executes a pixel connection process CP in which a plurality of neighboring difference pixels 122, 124, 126, or 128 are connected to form a group in the difference image data 120 generated in the immediately preceding step S23.

[0066] As an example, when the differential pixels 122 ( FIG. 13 ) or 124 ( FIG. 14 ) of the depth image data 100A are generated in step S23, the processor 32 connects the differential pixels 122 or 124 assigned to adjacent pixels PX to form a group in the pixel connection process CP. Alternatively, the processor 32 may connect the differential pixels 122 or 124 that are within a predetermined distance from each other in the sensor coordinate system C3 to form a group in the pixel connection process CP.

[0067] As another example, when the differential pixels 126 ( FIG. 15 ) or 128 ( FIG. 16 ) of the 3D point cloud image data 100B are generated in step S23, the processor 32 connects the differential pixels 126 or 128 assigned to adjacent unit pixel regions PE to form a cluster in the pixel connection process CP. Alternatively, the processor 32 may connect the differential pixels 126 or 128 that are within a predetermined distance from each other in the sensor coordinate system C3 to form a cluster in the pixel connection process CP.

[0068] By executing the pixel connection process CP as described above, the processor 32 calculates the difference d based on the difference image data 120 generated in step S23. ｎ An image area 130 can be specified in the sensor coordinate system C3 as a collection of difference pixels 122, 124, 126, or 128 that indicate the difference d ｎ The region specifying unit 56 (FIG. 2) functions as the region specifying unit 56 that specifies the image region 130 where the image has occurred.

[0069] In step S25, the processor 32 defines a detection range 132 (first detection range). The detection range 132 is defined by the image data 100 in the immediately following step S26. ｎ＋１ This is the range within which the workpiece 200 should be detected. First, in the sensor coordinate system C3 that defines the differential image data 120, the processor 32 determines a figure 134 that surrounds the image region 130 identified in the immediately preceding step S24 from the outside.

[0070] 18, the processor 32 defines a figure 134 in the sensor coordinate system C3 as a circumscribing figure of a polygon (specifically, a rectangle) circumscribing the image region 130. Specifically, the processor 32 identifies the differential pixels 122, 124, 126, or 128 constituting the image region 130 that are located at both ends of the x-axis direction and the y-axis direction of the sensor coordinate system C3, and calculates the coordinates Qs of these differential pixels in the sensor coordinate system C3. １ (x ｓ１ , y ｓ１ ), Qs ２ (x ｓ２ , y ｓ２ ), Qs ３ (x ｓ３ , y ｓ３ ), and Qs ４ (x ｓ４ , y ｓ４ ) is identified (FIG. 18).

[0071] Then, the processor 32 calculates the x coordinate value of the graphic 134 in the sensor coordinate system C3 as x ｓ１ ≦x ｓ ≦x ｓ２ The range [xｓ１ , x ｓ２ ], the y-coordinate value is y ｓ３ ≦y ｓ ≦y ｓ４ The range [y ｓ３ , y ｓ４ When the figure 134 is defined in this way, a side 134 a of the figure 134 in the negative x-axis direction of the sensor coordinate system C3 contacts the differential pixel at the end of the negative x-axis direction of the sensor coordinate system C3, among the differential pixels 122, 124, 126, or 128 that constitute the image area 130.

[0072] On the other hand, the side 134b of the graphic 134 in the positive x-axis direction of the sensor coordinate system C3 contacts the difference pixels at the ends of the sensor coordinate system C3 in the positive x-axis direction among the difference pixels 122, 124, 126, or 128 that make up the image area 130. Similarly, the sides 134c and 134c of the graphic 134 in the y-axis direction of the sensor coordinate system C3 contact the difference pixels at the ends of the sensor coordinate system C3 in the negative y-axis direction and the positive y-axis direction among the difference pixels 122, 124, 126, or 128 that make up the image area 130. The processor 32 calculates the position data (e.g., the range [x ｓ１ , x ｓ２ ], and the range [y ｓ３ , y ｓ４ ]) is stored in the memory 34.

[0073] Next, the processor 32 defines the detection range 132 based on the defined figure 134. For example, the processor 32 defines the defined figure 134 as the detection range 132. Alternatively, the processor 32 may define the detection range 132 as a range wider than the defined figure 134 by offsetting each of the sides 134 a, 134 b, 134 c, and 134 d of the defined figure 134 outward by a predetermined offset amount.

[0074] In this case, the operator may operate the input device 40 to input the offset amount in advance, and the processor 32 may receive the offset amount through the input device 40. In this way, as shown in Fig. 18, the detection range 132 is defined in the sensor coordinate system C3 so as to include the image area 130 therein. In this way, in this embodiment, the processor 32 functions as the range definition unit 58 (Fig. 2) that defines the detection range 132.

[0075] In step S26, the processor 32 functions as the workpiece search unit 44 and searches the image data 100. ｎ＋１ In step S25, the processor 32 executes the search process SP within the detection range 132 defined in the previous step S25. Specifically, the processor 32 calculates the position data (range [x ｓ１ , x ｓ２ ], range [y ｓ３ , y ｓ４ ]) is read from the memory 34, and the image data 100 captured in the most recent step S1 is ｎ＋１ This state is shown in FIG.

[0076] The processor 32 then functions as a workpiece search unit 44 to search for the image data 100. ｎ＋１ In the inner area of ​​the detection range 132 (range [x ｓ１ , x ｓ２ ], and the range [y ｓ３ , y ｓ４ ] area), the above-described search process SP is executed, and the image data 100 is detected within the detection range 132. ｎ＋１ As a result, the processor 32 newly detects three workpieces 200B, 200C, and 200G within the detection range 132, as shown in FIG.

[0077] The processor 32 calculates the coordinates Qs in the sensor coordinate system C3 of the workpiece coordinate system C4 of the workpiece model 200M that has been matched to the newly detected workpieces 200B to 200G. _Ｂ ', Qs _Ｃ ', and Qs _ＧThen, the processor 32 obtains these coordinates Qs _Ｂ ', Qs _Ｃ ', and Qs _Ｇ is expressed as the coordinate Qr of the robot coordinate system C1. _Ｂ ', Qr _Ｃ ', and Qr _Ｇ In this way, the processor 32 obtains the position data PD (coordinates Qs and Qr) of the newly detected workpieces 200B to 200G.

[0078] On the other hand, in step S26, the processor 32 ｎ＋１ In step S27, the processor 32 determines whether or not the workpiece 200 was detected in the immediately preceding step S26, similar to step S12 described above. If the processor 32 determines YES, the process proceeds to step S28, whereas if the processor 32 determines NO, the process proceeds to step S29.

[0079] In step S28, the processor 32 updates the detection result list 110. Specifically, the processor 32 acquires from the detection result list 110 the position data PD stored in the detection result list 110 stored in the memory 34 at this point in time that is outside the detection range 132 defined in the most recent step S25.

[0080] For example, assume that the detection result list 110 shown in Fig. 7 is stored at this point. In this case, as shown in Figs. 19 and 20, the workpieces 200D, 200E, and 200F are outside the detection range 132, and their position data PD (coordinates Qr _Ｄ , Qr _Ｅ and Qr _Ｆ ) are stored in the detection result list 110 as identification numbers "No. 4," "No. 5," and "No. 6."

[0081] Therefore, in this case, the processor 32 calculates the position data PD (coordinates Qr _Ｄ , Qr _Ｅ and Qr _Ｆ) from the detection result list 110. Then, the processor 32 obtains the position data PD (coordinates Qr _Ｂ ', Qr _Ｃ ', and Qr _Ｇ ), and the position data PD (coordinates Qr _Ｄ , Qr _Ｅ and Qr _Ｆ ) to update the detection result list 110.

[0082] 21 shows an example of the updated detection result list 110. In this detection result list 110, the position data PD (coordinates Qr _G ), position data PD (coordinates Qr _Ｄ ), position data PD (coordinates Qr _Ｂ '), position data PD (coordinates Qr _Ｃ '), position data PD of the workpiece 200E (coordinates Qr _Ｅ ), and the position data PD (coordinates Qr _Ｆ The works 200B, 200C, 200G, 200D, 200E, and 200F are assigned identification numbers "No. 1" to "No. 6" and rearranged in the order of the identification numbers. The processor 32 may assign identification numbers "No." to the works 200B, 200C, 200G, 200D, 200E, and 200F in accordance with the above-mentioned priority order.

[0083] 3, and executes the operation WK on the workpiece 200 by referring to the detection result list 110 stored in the memory 34 at this time. For example, when the detection result list 110 of FIG. 21 is stored, the processor 32 executes the operation WK on the workpiece 200 by referring to the position data PD: coordinates Qr of the workpiece 200G to which the identification number "No. 1" has been assigned and whose "status" is set to "waiting for operation". _Ｇ is acquired and the operation WK is performed on the workpiece 200G.

[0084] 9 returns NO, in step S29, the processor 32 acquires the detection result list 110 stored in the memory 34 at this time. Then, the processor 32 proceeds to step S5 in FIG. 3, refers to the acquired detection result list 110, and performs the work WK based on the position data PD in the detection result list 110 for which the "status" is "waiting for work" and the identification number "No."

[0085] As described above, in this embodiment, the processor 32 functions as the difference acquisition unit 52, the image data generation unit 54, the region specification unit 56, and the range definition unit 58 to generate the image data 100. ｎ＋１ A detection range 132 is set within which the workpiece 200 should be detected. Therefore, the difference acquisition unit 52, the image data generation unit 54, the region identification unit 56, and the range definition unit 58 constitute a device 50 (FIG. 2) that sets the detection range 132.

[0086] In this device 50, the difference acquisition unit 52 acquires first image data 100 of the workpiece 200 captured by the visual sensor 14 before the operation WK. ｎ (Fig. 4) ２ =z ｓ_ｎ ) and second image data 100 obtained by imaging the workpiece 200 by the visual sensor 14 after the operation WK. ｎ＋１ (Fig. 8) ２ =z ｓ_ｎ＋１ ) and the difference δ ｎ (= z ｓ_ｎ＋１ -z ｓ_ｎ ) is calculated (step S21).

[0087] The area specifying unit 56 also calculates the difference δ in the sensor coordinate system C3 that is set in the visual sensor 14 and defines the image data 100. ｎ The area defining unit 58 then defines a first detection area 132 (FIG. 18) in the sensor coordinate system C3 so as to include the image area 130 defined by the area defining unit 56 (step S25).

[0088] Here, when the work WK is performed in step S5, the arrangement of the workpiece 200 may change due to the above-mentioned collapse of the load, etc. Such a change in the arrangement of the workpiece 200 may occur due to the above-mentioned difference δ ｎ According to this embodiment, the image data 100 ｎ＋１ The range in which the workpiece 200 is detected is determined by the image data 100. ｎ＋１ From the whole area, the difference δ ｎ This can reduce the amount of calculation required for the search process SP in step S26, thereby shortening the cycle time of the work WK.

[0089] In the device 50, the image data generating unit 54 calculates the difference δ obtained by the difference obtaining unit 52. ｎ In step S23, the area specifying unit 56 generates differential image data 120 (FIG. 12) that expresses the difference δ in the sensor coordinate system C3. Then, the area specifying unit 56 specifies the image area 130 based on the differential image data 120 generated by the image data generating unit 54. According to this configuration, the difference δ in the sensor coordinate system C3 is calculated based on the differential image data 120. ｎ This effectively identifies the image region 130 where the error occurred.

[0090] The processor 32 can also specify the image region 130 as the region specifying unit 56 without creating the difference image data 120. For example, in step S21 described above, the processor 32 can specify the difference δ ｎ The coordinate Qs of the pixel PX or unit pixel area PE where the difference δ ｎ The processor 32 may then identify the image region 130 in the sensor coordinate system C3 based on the database. In this case, the image data generator 54 can be omitted from the device 50.

[0091] In the device 50, the region specifying unit 56 detects a difference δ that is equal to or greater than a predetermined threshold δth. ｎ The image region 130 where the difference δ occurs is specified in the sensor coordinate system C3. ｎWhile ignoring the difference δ ｎ It should be noted that the region specifying unit 56 can effectively extract the image region 130 representing all the differences δ without setting a threshold value δth. ｎ The image region 130 where the error occurred may be identified.

[0092] In the device 50, the range definition unit 58 defines a figure 134 (FIG. 18) that surrounds the image area 130 identified by the area identification unit 56 from the outside in the sensor coordinate system C3, and defines a first detection range 132 based on the defined figure 134. ｎ＋１ In the sensor coordinate system C3, the difference δ ｎ The detection range 132 can be defined to ensure that it encompasses the image region 130 where the

[0093] The range defining unit 58 may define the detection range 132 without defining the figure 134. For example, the range defining unit 58 may define the image area 130 itself as the detection range 132, or may define the image area 130 itself as the detection range 132 without defining the figure 134. ｎ＋１ may be divided into an area where the image area 130 exists and an area where the image area 130 does not exist, and the former may be defined as the detection range 132.

[0094] In the device 50, the area definition unit 58 defines a circumscribing figure 134 that circumscribes the image region 130. ｎ This allows the size of detection range 132 to be optimized so as to reliably include image region 130 where the defect has occurred, and also allows a relatively simple algorithm to be used to define figure 134 for image region 130. Note that range definition unit 58 is not limited to a circumscribing figure as figure 134, and may define any shape of figure that includes image region 130.

[0095] In this embodiment, the robot system 10 includes the visual sensor 14, the workpiece search unit 44, the robot 12, and the above-described device 50. The workpiece search unit 44 searches the second image data 100. ｎ＋１In step S26, the search process SP is executed within the first detection range 132 defined by the range definition unit 58, but the search process SP is not executed outside the first detection range 132.

[0096] According to this configuration, the image data 100 ｎ＋１ In this case, the difference δ ｎ By executing the search process SP only for the detection range 132 where a workpiece 200 has occurred, it is possible to detect the workpiece 200 within the detection range 132 and obtain its position data PD. On the other hand, for workpieces outside the detection range 132, the search process SP can be omitted and, for example, position data PD of workpieces 200 detected in the past can be used. Therefore, the cycle time of the operation WK can be shortened.

[0097] In the above embodiment, when the image data 100 is three-dimensional point cloud image data 100B, the processor 32 calculates one representative point Or in the unit pixel region PE in step S21. ｎ However, the present invention is not limited to this. The processor 32 may determine the representative point Or ｎ Without determining the coordinates Qs in the sensor coordinate system C3 for each point O (FIG. 11A) in the unit pixel area PE, the image data 100 before the operation is obtained. ｎ The z coordinate value of each point O in the image: ｓ_ｎ and the image data 100 after the operation. ｎ＋１ The z coordinate value of each point O in the image: ｓ_ｎ＋１ Difference δ ｎ You may ask for:

[0098] Next, another example of the operation flow of the robot system 10 will be described with reference to Fig. 22. In this embodiment, the processor 32 executes the flow shown in Fig. 23 in step S4. In step S4 shown in Fig. 23, the processor 32 proceeds to step S7 in Fig. 22 after step S28, and proceeds to step S5 in Fig. 22 after step S29.

[0099] In step S7, the processor 32 performs the work WK on the workpiece 200, similar to step S5 described above. Specifically, the processor 32 references the detection result list 110 stored in the memory 34 at this time, and acquires the position data PD whose "status" is "waiting for work" and whose identification number "No." For example, in the example shown in FIG. 7 , the processor 32 performs the work WK on the workpiece 200A assigned the identification number "No. 1," and changes the "status" of the position data PD of the identification number "No. 1" to "work successful" in the detection result list 110 (or deletes the position data PD of the identification number "No. 1" from the detection result list 110).

[0100] In this embodiment, the processor 32 changes the "status" of the workpieces 200 in the detection result list 110 that are within a predetermined distance from the workpiece 200A with identification number "No. 1" that has been determined to be "work successful" to "work pending." For example, in the example shown in FIG. 6, with respect to the six workpieces 200A to 200F detected in the search process SP of the most recent step S26, it is assumed that workpieces 200B and 200C are within a predetermined distance Δ from the workpiece 200A that has been determined to be "work successful," while workpieces 200D, 200E, and 200F are outside the distance Δ. In this case, the processor 32 changes the "status" of the workpieces 200B and 200C with identification numbers "No. 2" and "No. 3" to "work pending," as shown in FIG. 24. The processor 32 then proceeds to step S8.

[0101] In step S8, processor 32 determines whether or not there is any workpiece 200 that can be worked on. Specifically, processor 32 references detection result list 110 stored in memory 34 at this time point, and determines whether or not there is any workpiece 200 whose "status" is "waiting for work." For example, in the example shown in FIG. 24, the "status" of workpieces 200D to 200F with identification numbers "No. 4" to "No. 6" is "waiting for work." Therefore, in this case, processor 32 determines YES and returns to step S7.

[0102] Then, in step S7, the processor 32 executes the operation WK on the workpiece 200D with the "status" set to "waiting for operation" and the highest identification number "No. 4" in the detection result list 110 stored in the memory 34 at this time. In this manner, in this embodiment, when the processor 32 completes the operation WK on one workpiece 200A, it continuously executes the operation WK on the workpiece 200D that is outside the range of the predetermined distance Δ from the one workpiece 200A.

[0103] Here, when an operation WK is performed on one workpiece 200A, there is a high possibility that the positions of nearby works 200B and 200C will change due to a shift in load, etc., whereas there is a low possibility that the positions of works 200D to 200F located at a distance from the one workpiece 200A will change due to a shift in load, etc. In this embodiment, by continuously performing an operation WK on workpiece 200D located at a distance from workpiece 200A for which the operation WK has been completed, it is possible to shorten the cycle time.

[0104] Next, another example of step S4 in Fig. 3 will be described with reference to Fig. 25. In this embodiment, the processor 32 executes steps S30 and S31 after step S25. In step S30, the processor 32 compares the detection range 132 defined in the immediately preceding step S25 with the image data 100 defined in the search process SP of the most recent step S11 or S26 in the sensor coordinate system C3. ｎ It is determined whether the occupied areas of the matched workpiece models 200M overlap with each other.

[0105] For example, the processor 32 searches the image data 100 in the most recent search process SP as shown in FIG. ｎ In this example, the workpiece model 200M is matched to the sensor coordinate system C3. ｎ A workpiece model 200M is matched to the workpieces 200A, 200B, 200C, and 200I shown in the image, and a portion of the occupied area of ​​the workpiece model 200M matched to the workpiece 200I overlaps with the detection range 132 defined in the immediately preceding step S25.

[0106] In step S30, the processor 32 determines whether the image data 100 was found in the most recent search process SP. ｎ Based on the position data PD of the workpiece model 200M that matches the position data PD and the position data of the detection range 132 defined in the immediately preceding step S25, it is determined whether or not a part of the occupied area of ​​the workpiece model 200M overlaps with the detection range 132.

[0107] For example, the processor 32 determines whether an imaginary plane formed by extending the edge 134c ( FIG. 18 ) that defines the detection range 132 in the z-axis direction of the sensor coordinate system C3 intersects with the workpiece model 200M that is matched to the workpiece 200I. If the imaginary plane intersects with the workpiece model 200M, the processor 32 determines that a portion of the occupation area of ​​the workpiece model 200M that is matched to the workpiece 200I overlaps with the detection range 132. If the processor 32 determines YES, the process proceeds to step S31, and if the processor 32 determines NO, the process proceeds to step S26.

[0108] In step S31, the processor 32 functions as the range definition unit 58 and further defines a detection range 136 (second detection range) in the sensor coordinate system C3. Specifically, as shown in FIG. 27 , the processor 32 further defines the detection range 136 in addition to the detection range 132 so as to include the occupied area of ​​the workpiece model 200M that was determined to overlap with the detection range 132 in the immediately preceding step S30.

[0109] In this way, the detection ranges 132 and 136 are defined in the sensor coordinate system C3. Then, in step S26 in FIG. 25, the processor 32 functions as the workpiece search unit 44 and searches the image data 100 captured in the most recent step S1. ｎ＋１ The detection ranges 132 and 136 are set in the sensor coordinate system C3, and the search process SP is executed within the detection ranges 132 and 136.

[0110] In this manner, in this embodiment, the range defining unit 58 defines the first image data 100 ｎIf a portion of the occupation area of ​​the workpiece model 200M matched to the sensor coordinate system C3 overlaps with the first detection range 132, a second detection range 136 is further defined in the sensor coordinate system C3 so as to encompass the occupation area.

[0111] This configuration increases the number of workpieces 200 that can be detected in the search process SP, thereby improving the efficiency of the search process SP. Furthermore, since it is possible to reliably detect whether or not a workpiece 200 is present within the second detection range 136 and the current posture of the workpiece 200, the possibility of the workpiece WK failing can be reduced, thereby improving the accuracy of the workpiece WK. Note that steps S30 and S31 in FIG. 25 may be applied to the flow in FIG. 23. In this case, the processor 32 may execute steps S30 and S31 after step S25 in FIG. 23.

[0112] Next, another example of step S4 in Fig. 3 will be described with reference to Fig. 28. In this embodiment, the processor 32 executes steps S32 and S33 after step S25. In step S32, the processor 32 executes the image data 100 ｎ In the image data 100, the workpiece model 200M that matches the workpiece 200 detected in the sensor coordinate system C3 is located in an area other than the occupied area. ｎ It is determined whether there are any visual features of the workpiece 200 that appear in the distance image data 100A (i.e., visual features of the workpiece 200 that appear in the distance image data 100A, or a point cloud representing the workpiece 200 that appears in the three-dimensional point cloud image data 100B).

[0113] For example, in the search process SP of the most recent step S11 or S26, the processor 32 searches the image data 100 as shown in FIG. ｎ In this example, the workpiece model 200M is matched to the sensor coordinate system C3. ｎ The workpiece model 200M is matched to the workpieces 200A, 200B, and 200C shown in the image, while the workpiece model 200M is not matched to the workpieces 200J and 200K.

[0114] In step S32, the processor 32 determines YES if the visual features of the workpieces 200J and 200K are present in an area of ​​the workpiece model 200M matched to the workpieces 200A, 200B, and 200C other than the occupied area in the sensor coordinate system C3, and proceeds to step S33. On the other hand, if the processor 32 determines NO, the processor 32 proceeds to step S26.

[0115] In step S33, the processor 32 functions as the range definition unit 58 and further defines a detection range 138 (third detection range) in the sensor coordinate system C3. Specifically, as shown in FIG. 29 , the processor 32 further defines the detection range 138 in addition to the detection range 132 so as to include the visual features of the workpieces 200J and 200K identified in the immediately preceding step S32.

[0116] In this way, the detection ranges 132 and 138 are defined in the sensor coordinate system C3. Then, in step S26 in FIG. 28, the processor 32 functions as the workpiece search unit 44 and searches the image data 100 captured in the most recent step S1. ｎ＋１ 29 shows an example in which the visual features of the workpieces 200J and 200K partially overlap with the detection range 132, but it should be understood that the visual features of the workpieces 200J and 200K may be entirely outside the detection range 132.

[0117] In this manner, in this embodiment, the range defining unit 58 defines the first image data 100 ｎ A third detection range 138 is further defined in the sensor coordinate system C3 so as to encompass the area other than the occupied area in the sensor coordinate system C3 of the workpiece model 200M matched to the workpieces 200A, 200B, and 200C detected in (the visual features of the workpieces 200J and 200K in FIG. 29).

[0118] This configuration increases the number of workpieces 200 that can be detected in the search process SP, thereby improving the efficiency of the search process SP. Furthermore, when step S26 is executed, visual features of the workpiece 200 that could not be detected in the search process SP may exist outside the detection range 132. According to this embodiment, by further defining the third detection range 138, the workpiece 200 that could not be detected in step S26 can be searched for again in the next step S26.

[0119] Note that steps S32 and S33 in Fig. 28 may be applied to the flow in Fig. 23. In this case, the processor 32 may execute steps S32 and S33 after step S25 in Fig. 23. Also, steps S32 and S33 in Fig. 28 may be applied to the flow in Fig. 25. In this case, the processor 32 may execute steps S32 and S33 after step S31 in Fig. 25 (or before step S30).

[0120] In the above-described step S25, the case where a quadrilateral graphic 134 is defined has been described, but the present invention is not limited to this, and the processor 32 may define any m-sided graphic 134 (m=3, 4, 5, 6, 8, ...). In this case, the processor 32 may determine the number "m" defining the m-sided graphic 134 based on the outer shape of the image region 130 identified in step S24.

[0121] 17 is identified in the sensor coordinate system C3, the processor 32 calculates a projected outer edge shape by projecting the outer edge of the image area 130 as viewed from the z-axis direction of the sensor coordinate system C3 onto the x-y plane of the sensor coordinate system C3. The processor 32 then functions as the range definition unit 58 and may determine the "m" of an m-sided polygon based on the projected outer edge shape so as to minimize the margin between the projected outer edge shape. This configuration allows the size of the graphic 134 and the detection range 132 to be optimized.

[0122] When the processor 32 identifies the image region 130, the processor 32 may use the machine learning model LM to determine an optimal graphic 134 (or detection range 132) that encompasses the image region 130. For example, an operator inputs a dataset DS of data on image regions of various shapes and training data on the graphic (or detection range) set for the image region into the machine learning device. The machine learning device uses the dataset DS to learn a machine learning model LM that represents the correlation between the shape of the image region and the graphic (or detection range) to be set for the image region.

[0123] Then, in step S25, the processor 32 inputs the identified image region 130 into the machine learning model LM. The machine learning model LM outputs a graphic 134 (or a detection range 132) that is correlated with the input image region 130 as an optimal solution. The processor 32 may define the graphic 134 (or the detection range 132) output by the machine learning model LM in the sensor coordinate system C3. This configuration makes it possible to define the graphic 134 (or the detection range 132) that is optimal for the identified image region 130. Note that the processor 32 may be configured to execute the functions of a machine learning device.

[0124] Note that, in the above-described embodiment, the graphic 134 is described as being a polygon (quadrilateral), but the graphic 134 may have any shape, such as a circle, an ellipse, or a star. Fig. 30 shows an example in which a circular graphic 134 and a detection range 132 are determined. In the example shown in Fig. 30 , the processor 32 may, for example, determine the center of gravity of the identified image region 130 and determine the center of the graphic 134 at the center of gravity. The processor 32 may then determine the coordinate Qs in the sensor coordinate system C3 of the difference pixel 122, 124, 126, or 128 that is farthest from the center, and determine the circular graphic 134 having a radius equal to the distance from the center to the coordinate Qs.

[0125] In step S25, the processor 32 calculates the x-coordinate range [x ｓ１ , x ｓ２ ] and the range of y coordinate values ​​[y ｓ３ , y ｓ４However, the present invention is not limited to this, and the processor 32 may determine the graphic 134 (detection range 132) by further taking into account the range of z coordinate values.

[0126] For example, assume that the differential image data 120 shown in Fig. 14 (differential pixel 124) or Fig. 16 (differential pixel 128) is generated in step S23, and the image area 130 shown in Fig. 17 is identified based on the differential image data 120 in step S24. In this case, the processor 32 identifies the differential pixel located at the end of the sensor coordinate system C3 in the positive z-axis direction among the differential pixels 124 or 128 that make up the image area 130, and calculates the coordinate Qs of the differential pixel in the sensor coordinate system C3. ５ (x ｓ５ , y ｓ５ , z ｓ５ ) to identify the

[0127] Then, the processor 32 calculates the shape 134 (detection range 132) as a range [x ｓ１ , x ｓ２ ] and the range [y ｓ３ , y ｓ４ ], and the z coordinate value of the sensor coordinate system C3 is 0≦z ｓ ≦z ｓ５ In the range [0, z ｓ５ In this case, the figure 134 (detection range 132) is defined as a three-dimensional rectangular parallelepiped area.

[0128] Alternatively, the processor 32 identifies the difference pixel 124 or 128 that constitutes the image region 130 and that is located at the edge of the sensor coordinate system C3 in the negative z-axis direction, and calculates the coordinate Qs of that difference pixel 124 or 128 in the sensor coordinate system C3. ０ (x ｓ０ , y ｓ０ , z ｓ０ Then, the processor 32 identifies the graphic 134 (detection range 132) as a range [x ｓ１ , x ｓ２ ] and the range [y ｓ３ , y ｓ４ ], and the z coordinate value of the sensor coordinate system C3 is z ｓ０ ≦z ｓ ≦z ｓ５ In the range [0, z ｓ５ ] may be determined as follows.

[0129] 3, 5, 9, 22, 23, 25, or 28 in accordance with a computer program PG pre-stored in the memory 34. The functions of the device 50 (i.e., the difference acquisition unit 52, the image data generation unit 54, the area identification unit 56, and the range definition unit 58) and the workpiece search unit 44 executed by the processor 32 may be functional modules realized by the computer program PG.

[0130] After step S25, the processor 32 may display the defined detection range 132 on the display device 38. Then, the operator may visually check the detection range 132 displayed on the display device 38 and operate the input device 40 to provide input for editing the detection range 132. The processor 32 may change the detection range 132 in accordance with the input received through the input device 40.

[0131] In addition to the above-mentioned detection ranges 132, 136, and 138, the processor 32 may further define a fourth detection range that includes the occupation area of ​​the workpiece model 200M that matches the workpiece 200 for which the workpiece WK has been properly completed. For example, assume that the workpiece WK for the workpiece 200A shown in FIG. 6 has been properly completed, and the position of the surrounding workpiece 200 has not changed due to this workpiece WK. In this case, the image data 100 captured before and after the workpiece WK is used to determine the occupation area of ​​the workpiece 200M. ｎ and 100 ｎ＋１ In this case, the image area where the difference occurs is the workpiece 200A. ｎ The fourth detection range may be defined so as to include the area occupied by the workpiece model 200M that is matched to the workpiece 200A.

[0132] Although the present disclosure has been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0133] The present disclosure describes the following aspects: (Aspect 1) An apparatus 50 for setting detection ranges 132, 136, 138 within which a workpiece 200 should be detected in image data 100 captured by a visual sensor 14 capable of measuring a distance d to a subject, the apparatus 50 including: a first image data 100 of the workpiece 200 captured by the visual sensor 14 before a task; ｎ and the second image data 100 obtained by capturing an image of the workpiece 200 by the visual sensor 14 after the work. ｎ＋１ The difference δ from the distance d included in ｎ and a difference acquisition unit 52 that obtains the difference δ in a sensor coordinate system C3 that is set in the visual sensor 14 and defines the image data 100. ｎ and a range defining unit that defines a first detection range in a sensor coordinate system so as to include the image region identified by the region defining unit. ｎ The device 50 according to aspect 1 further includes an image data generation unit 54 that generates differential image data 120 in which the difference δ is expressed in the sensor coordinate system C3, and the region identification unit 56 identifies the image region 130 based on the differential image data 120 generated by the image data generation unit 54. (Aspect 3) The region identification unit 56 is configured to identify the image region 130 based on the difference δ that is equal to or greater than a predetermined threshold δth. ｎThe device 50 according to aspect 1 or 2, wherein the area defining unit 58 defines, in the sensor coordinate system C3, an image area 130 in which the detection error has occurred. (Aspect 4) The device 50 according to any one of aspects 1 to 3, wherein the range defining unit 58 defines, in the sensor coordinate system C3, a figure 134 that surrounds the image area 130 identified by the area identifying unit 56 from the outside, and defines a first detection area 132 based on the defined figure 134. (Aspect 5) The device 50 according to aspect 4, wherein the range defining unit 58 defines, as the figure 134, a circumscribing figure that circumscribes the image area 130. (Aspect 6) The robot system 10 is configured to search for the workpiece 200 depicted in the image data 100 by matching a workpiece model 200M that is a model of the workpiece 200 with the image data 100, and the range defining unit 58 defines, in the first image data 100, a workpiece model 200M that is a model of the workpiece 200. ｎ The apparatus (50) according to any one of aspects (1) to (5), wherein, when a part of an occupation area in the sensor coordinate system (C3) of the workpiece model (200M) matched to the first image data (100) overlaps with the first detection area (132), the apparatus (50) further defines a second detection area (136) in the sensor coordinate system (C3) so as to encompass the occupation area. (Aspect 7) The robot system (10) is configured to detect the workpiece (200) depicted in the image data (100) by matching the workpiece model (200M) modeled on the workpiece (200) to the image data (100), and the range defining unit (58) ｎ (Aspect 8) A device 50 according to any one of Aspects 1 to 7, further defining a third detection range 138 in the sensor coordinate system C3 so as to include an area other than the occupied area in the sensor coordinate system C3 of the workpiece model 200M matched to the workpieces 200A, 200B, 200C detected in the second image data 100. (Aspect 8) A device 50 according to any one of Aspects 1 to 7, comprising: a visual sensor 14 configured to be able to measure the distance to the subject and capturing an image of the workpiece 200; a workpiece searching unit 44 that executes a searching process SP to search for the workpiece 200 captured in the image data 100 by matching a workpiece model 200M that is a model of the workpiece 200 to the image data 100 captured by the visual sensor 14; a robot 12 that performs a predetermined task on the workpiece 200 detected in the image data 100 by the searching process SP; and the device 50 according to any one of Aspects 1 to 7, wherein the workpiece searching unit 44 searches for the workpiece 200 captured in the image data 100 by the second image data 100. ｎ＋１In the robot system 10, the search process SP is executed within a first detection range 132 defined by the range definition unit 58, but the search process SP is not executed outside the first detection range 132. (Aspect 9) A method for setting detection ranges 132, 136, 138 in which a workpiece 200 is to be detected in image data 100 captured by a visual sensor 14 capable of measuring a distance d to a subject, the method comprising: a processor 32 generating first image data 100 in which the visual sensor 14 captured an image of the workpiece 200 before an operation WK; ｎ and the second distance d included in the second image data obtained by capturing an image of the workpiece 200 by the visual sensor 14 after the operation WK. ｎ is calculated, and the difference δ is calculated in the sensor coordinate system C3 that is set in the visual sensor 14 and defines the image data 100. ｎ and determining a first detection range 132 in the sensor coordinate system C3 so as to encompass the identified image region 130. (Aspect 10) A computer program PG that causes the processor 32 to execute the method according to Aspect 9.

[0134] REFERENCE SIGNS LIST 10 Robot system 12 Robot 14 Visual sensor 16 Control device 32 Processor 44 Workpiece search unit 50 Device 52 Difference acquisition unit 54 Image data generation unit 56 Area identification unit 58 Range definition unit 100 Image data 120 Difference image data

Claims

1. An apparatus for setting a detection range in which a workpiece is to be detected based on image data captured by a visual sensor capable of measuring the distance to a subject, comprising: a difference acquisition unit that obtains the difference between the distance contained in first image data obtained by capturing an image of the workpiece by the visual sensor before an operation is performed on the workpiece, and the distance contained in second image data obtained by capturing an image of the workpiece by the visual sensor after the operation; an area identification unit that is set on the visual sensor and that identifies an image area in which the difference occurs in a sensor coordinate system that defines the image data; and a range definition unit that defines the first detection range in the sensor coordinate system so as to include the image area identified by the area identification unit.

2. The device described in claim 1, further comprising an image data generation unit that generates difference image data that represents the difference obtained by the difference acquisition unit in the sensor coordinate system, and the area identification unit identifies the image area based on the difference image data generated by the image data generation unit.

3. The device according to claim 1, wherein the region identifying section identifies, in the sensor coordinate system, the image region in which the difference exceeds a predetermined threshold value.

4. The device according to claim 1, wherein the range definition unit defines a figure in the sensor coordinate system that surrounds from the outside the image area identified by the area identification unit, and defines the first detection range based on the defined figure.

5. The apparatus according to claim 4, wherein the range definition unit defines, as the figure, a circumscribing figure that circumscribes the image region.

6. The device described in claim 1, wherein the workpiece depicted in the image data is searched for by matching a workpiece model that models the workpiece to the image data, and when a portion of an occupation area in the sensor coordinate system of the workpiece model matched to the first image data overlaps with the first detection range, the range definition unit further defines a second detection range in the sensor coordinate system so as to include the occupation area.

7. The device described in claim 1, wherein the workpiece depicted in the image data is searched for by matching a workpiece model that models the workpiece to the image data, and the range definition unit further defines a third detection range in the sensor coordinate system so as to include an area other than the occupied area in the sensor coordinate system of the workpiece model matched to the workpiece detected in the first image data.

8. A robot system comprising: a visual sensor configured to measure the distance to a subject and capturing an image of a workpiece; a workpiece search unit that executes a search process to search for the workpiece depicted in image data captured by the visual sensor by matching a workpiece model that models the workpiece to the image data; a robot that performs a specified task on the workpiece detected in the image data by the search process; and the device described in claim 1, wherein the workpiece search unit executes the search process within the first detection range defined by the range definition unit in the second image data, but does not execute the search process outside the first detection range.

9. A method for setting a detection range in which a workpiece is to be detected based on image data captured by a visual sensor capable of measuring the distance to a subject, wherein a processor determines the difference between the distance contained in first image data obtained by capturing an image of the workpiece by the visual sensor before an operation on the workpiece is performed and the distance contained in second image data obtained by capturing an image of the workpiece by the visual sensor after the operation is performed, identifies an image area in which the difference occurs in a sensor coordinate system set on the visual sensor and defining the image data, and defines a first detection range in the sensor coordinate system so as to include the identified image area.

10. A computer program product causing said processor to carry out the method of claim 9.

Images