Device for setting detection range of workpiece to be detected, robot system, method, and computer program

Abstract

The objective of the present invention is to reduce the cycle time of a job by reducing the computational processing required for a search process for searching for a workpiece reflected in image data. A device (50) for setting a detection range (132, 136, 138) within image data (100n + 1) in which a workpiece (200) is to be detected comprises: a difference acquisition unit (52) for obtaining the difference ([delta] n) between a distance (d) included in first image data (100n) obtained by imaging the workpiece (200) by a vision sensor (14) before work and a distance (d) included in second image data (100n + 1) obtained by imaging the workpiece (200) by the vision sensor (14) after work; a region specifying unit (56) that specifies an image region (130) in which the difference ([delta] n) has occurred in a sensor coordinate system (C3) that defines the image data (100) and that is set for the vision sensor (14); and a range defining unit (58) that defines a first detection range (132) in the sensor coordinate system (C3) so as to include the image region (130) specified by the region specifying unit (56).

Description

Technical Field

[0001] This disclosure relates to apparatus, robot systems, methods, and computer programs for setting the detection range of a workpiece to be detected in image data. Background Technology

[0002] A robot system is known that detects a workpiece reflected in image data captured by a vision sensor and performs a specified operation on the detected workpiece, wherein the vision sensor is capable of determining the distance to the photographed object (e.g., Patent Document 1).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-198950 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] The aim is to shorten the cycle time of operations by reducing the computational processing required to search for workpieces reflected in search image data.

[0008] Methods for solving problems

[0009] In one aspect of this disclosure, an apparatus is provided that sets a detection range for a workpiece to be detected based on image data captured by a vision sensor capable of measuring the distance to the subject. The apparatus comprises: a difference acquisition unit that calculates the difference between the distance contained in first image data and the distance contained in second image data, wherein the first image data is data obtained by the vision sensor capturing the workpiece before operation, and the second image data is data obtained by the vision sensor capturing the workpiece after operation; a region determination unit that determines the image region in which the difference occurs within a sensor coordinate system defined for the vision sensor and used to define the image data; and a range delineation unit that delineates the first detection range in the sensor coordinate system in a manner that includes the image region determined by the region determination unit.

[0010] In other aspects of this disclosure, a method is used to set a detection range for a workpiece to be detected based on image data captured by a vision sensor capable of measuring the distance to the subject. The processor performs the following actions: calculating the difference between the distance contained in a first set of image data and the distance contained in a second set of image data, wherein the first set of image data is data obtained by the vision sensor capturing the workpiece before the operation, and the second set of image data is data obtained by the vision sensor capturing the workpiece after the operation; determining the image region where the difference occurs in a sensor coordinate system defined for the vision sensor and used to define the image data; and defining the first detection range in the sensor coordinate system in a manner that includes the determined image region. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a robot system according to one embodiment.

[0012] Figure 2 yes Figure 1 The diagram shows a block diagram of the robot system.

[0013] Figure 3 It means Figure 1 The flowchart shows an example of the motion flow of the robot system.

[0014] Figure 4 express Figure 1 The image shown is an example of image data obtained by a vision sensor from photographing a workpiece before operation.

[0015] Figure 5 It means Figure 3 The flowchart of step S3 in the process.

[0016] Figure 6 This indicates the state of matching the workpiece model with the workpiece reflected in the image data during the search process.

[0017] Figure 7 This is an example of data construction representing a list of test results.

[0018] Figure 8 express Figure 1 The image shown is an example of image data obtained by a vision sensor after a task is completed. It represents... Figure 3 The flowchart of step S3 in the process.

[0019] Figure 9 It means Figure 3 The flowchart is an example of step S4 in the process.

[0020] Figure 10It is a graph used to illustrate the difference in distance contained in image data taken before and after an operation, when the image data is distance image data.

[0021] Figure 11 It is a diagram used to illustrate the difference in distance contained in image data taken before and after an operation, when the image data is 3D point cloud image data.

[0022] Figure 12 An example representing poor image data.

[0023] Figure 13 This represents an example of difference image data generated when the image data is distance image data.

[0024] Figure 14 Other examples of difference image data generated when the image data is distance image data.

[0025] Figure 15 This represents an example of poor image data generated when the image data is a 3D point cloud image.

[0026] Figure 16 Other examples of poor image data generated when the image data is 3D point cloud image data.

[0027] Figure 17 Schematic representation in Figure 9 An example of the image region determined in step S24.

[0028] Figure 18 Indicates to include Figure 17 The image region shown is an example of how the first detection range is defined.

[0029] Figure 19 Indicates will Figure 18 The first detection range shown is set to the state of the image data captured after the operation.

[0030] Figure 20 Indicates in Figure 19 The search process is performed within the first detection range shown, matching the workpiece model with the workpiece reflected in the image data.

[0031] Figure 21 Indicates in Figure 9 An example of the updated list of detection results in step S28.

[0032] Figure 22 It means Figure 1 Flowcharts for other examples of the motion flow of the robot system shown.

[0033] Figure 23It means Figure 22 The flowchart is an example of step S4 in the process.

[0034] Figure 24 Indicates in Figure 22 An example of the updated list of detection results after step S7.

[0035] Figure 25 It means Figure 3 The flowchart of another example of the process of step S4 in the process.

[0036] Figure 26 Indicates in Figure 25 The state in step S25 where the defined detection range overlaps with a portion of the occupied area of ​​the workpiece model.

[0037] Figure 27 This indicates that a second testing area has been further defined.

[0038] Figure 28 It means Figure 3 The flowchart of another example of step S4 in the process.

[0039] Figure 29 This indicates that the third testing scope has been further defined.

[0040] Figure 30 Other examples within the first detection range. Detailed Implementation

[0041] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Furthermore, in the various embodiments described below, the same elements are labeled with the same symbols, and repeated descriptions are omitted. First, refer to... Figure 1 as well as Figure 2 The robot system 10 according to one embodiment will be described. The robot system 10 includes: a robot 12, a vision sensor 14, and a control device 16.

[0042] In this embodiment, robot 12 is a vertical articulated robot, comprising: a robot base 18, a rotating body 20, a lower arm 22, an upper arm 24, a wrist 26, and an end effector 28. The robot base 18 is fixed to the ground of the work unit or to an automated guided vehicle (AGV). The rotating body 20 is mounted on the robot base 18 in a manner capable of rotating about a vertical axis.

[0043] The base end of the lower arm portion 22 is rotatably disposed on the rotating body 20, and the base end of the upper arm portion 24 is rotatably disposed on the front end of the lower arm portion 22. The wrist portion 26 has: a wrist base 26a, which is rotatably disposed on the front end of the upper arm portion 24 about two mutually orthogonal axes; and a wrist flange 26b, which is rotatably disposed on the wrist base 26a about the wrist axis A1.

[0044] The end effector 28 is detachably mounted on the wrist flange 26b. The end effector 28 may be, for example, a robotic arm capable of holding the workpiece 200, a welding torch for welding the workpiece 200, or a laser processing head for laser processing the workpiece 200, and performs a specified operation WK (workpiece manipulation, welding, or laser processing, etc.) on the workpiece 200.

[0045] Servo motors 30 are installed in each component of robot 12 (robot base 18, rotating body 20, lower arm 22, upper arm 24, wrist 26). Figure 2 These servo motors 30 rotate the drive axes of the robot 12 according to instructions from the control unit 16. As a result, the robot 12 is able to move the end effector 28 to any position.

[0046] like Figure 1 As shown, a robot coordinate system C1 and a tool coordinate system C2 are established for the robot 12. The robot coordinate system C1 is used for automatically controlling the movements of the various movable components of the robot 12 (i.e., the rotary body 20, the lower arm 22, the upper arm 24, the wrist base 26a, the wrist flange 26b, and the end effector 28). In this embodiment, the robot coordinate system C1 is fixed relative to the robot base 18 such that its origin is located at the center of the robot base 18, and its z-axis is parallel to (specifically, aligned with) the rotation axis of the rotary body 20.

[0047] On the other hand, the tool coordinate system C2 is a coordinate system that defines the position of the end effector 28 in the robot coordinate system C1. In this embodiment, the tool coordinate system C2 is set relative to the end effector 28 with its origin (so-called TCP) configured at the working position of the end effector 28 (i.e., workpiece holding position, welding position, or laser beam exit position), and its z-axis is parallel to (specifically, aligned with) the wrist axis A1.

[0048] When the end effector 28 is moved, the control device 16 sets a tool coordinate system C2 in the robot coordinate system C1 and generates commands for each servo motor 30 of the robot 12, thereby positioning the end effector 28 at the position represented by the set tool coordinate system C2. In this way, the control device 16 can position the end effector 28 at any position in the robot coordinate system C1. Furthermore, in this specification, "position" sometimes refers to both position and orientation.

[0049] The vision sensor 14 captures images of the workpiece 200 inside the container 202. Specifically, the vision sensor 14 is a three-dimensional vision sensor having an image sensor (CMOS, CCD, etc.) and an optical lens (collimating lens, focusing lens, etc.) that guides the image of the subject to the image sensor. The vision sensor 14 is configured to capture images of the subject (e.g., the workpiece 200 and the container 202) along the optical axis A2 and measure the distance d to the subject.

[0050] In this embodiment, the vision sensor 14 is fixed to a movable component of the robot 12 (e.g., an end effector 28 or a wrist flange 26b) and is moved by the robot 12. Alternatively, the vision sensor 14 can be fixed at a predetermined shooting position Pi within its field of view, allowing the workpiece 200 to be received. The vision sensor 14 supplies the captured image data to the control device 16.

[0051] A sensor coordinate system C3 is established for the vision sensor 14. The sensor coordinate system C3 is a coordinate system that defines the position of the vision sensor 14 in the robot coordinate system C1 (i.e., the position and orientation of the optical axis A2). In this embodiment, the sensor coordinate system C3 is set relative to the vision sensor 14 such that its origin is located at the center of the imaging sensor of the vision sensor 14, and its z-axis is parallel to (specifically, aligned with) the optical axis A2.

[0052] The sensor coordinate system C3 specifies the coordinates of each pixel in the image data (or the imaging sensor) captured by the vision sensor 14. The positional relationship between the sensor coordinate system C3 and the robot coordinate system C1 is known through calibration. Therefore, the coordinates of the sensor coordinate system C3 and the robot coordinate system C1 can be converted to each other via a known transformation matrix MX (e.g., a homogeneous transformation matrix).

[0053] The control device 16 controls the movements of the robot 12 and the vision sensor 14. For example... Figure 2As shown, the control device 16 is a computer having a processor 32, a memory 34, an I / O interface 36, a display device 38, and an input device 40. The processor 32 has a CPU or GPU, etc., and is communicatively connected to the memory 34, the I / O interface 36, the display device 38, and the input device 40 via the bus 42, communicates with these components, and performs arithmetic processing to implement various functions described later.

[0054] The memory 34 has RAM or ROM, etc., to temporarily or permanently store various data. The memory 34 can be composed of computer-readable non-temporary storage media such as volatile memory, non-volatile memory, magnetic storage media, or optical storage media. The I / O interface 36 has, for example, an Ethernet port, a USB port, a fiber optic connector, or an HDMI terminal, and communicates with external devices via wired or wireless means under the instruction from the processor 32. The servo motors 30 and vision sensors 14 of the robot 12 are connected to the I / O interface 36 in a communicative manner.

[0055] The display device 38 includes a liquid crystal display or an organic EL display, which displays various data in a visually identifiable manner under instructions from the processor 32. The input device 40 includes buttons, switches, a keyboard, a mouse, or a touch panel, which accept data input from the operator. Furthermore, the display device 38 and the input device 40 can be integrally assembled into the housing of the control device 16, or they can be connected to the I / O interface 36 as a separate computer (PC, etc.) from the housing of the control device 16.

[0056] In this embodiment, the robot system 10 uses a vision sensor 14 to capture images of workpieces 200 loosely packed in the container 202, and detects workpieces 200 reflected in the image data captured by the vision sensor 14. Furthermore, the robot system 10 uses the robot 12 to perform a prescribed operation WK on the detected workpieces 200. Hereinafter, refer to... Figure 3 The operation flow of the robot system 10 is described below. When the processor 32 of the control device 16 receives the work start command from the operator, the host controller, or the computer program PG, it begins... Figure 3 The process is shown below.

[0057] In step S1, the processor 32 takes a picture of the workpiece 200 using the vision sensor 14. Specifically, the processor 32 causes the robot 12 to move and position the vision sensor 14 at a shooting position Pi that can accommodate the workpiece 200 within its field of view. Furthermore, the processor 32 sends a shooting command to the vision sensor 14, and according to the shooting command, the vision sensor 14 takes a picture of the workpiece 200, which is loosely packed in the container 202. Figure 4This indicates that the image data captured by the vision sensor 14 is 100. n An example of.

[0058] As an example, the visual sensor 14 captures distance image data 100A as image data 100. n The distance d from image data 100A to the captured subject (workpiece 200 in this case) is represented by the intensity of color. The visual features of the subject as perceived by each pixel PX of image data 100A, and containing information about distance d, are represented by the coordinates (x, y) of sensor coordinate system C3. Therefore, the visual features of the subject perceived by each pixel PX of image data 100A are represented by the three-dimensional coordinates Qs(x, y) of sensor coordinate system C3. s y s , z s ).

[0059] As another example, the vision sensor 14 captures 100B of 3D point cloud image data as image data 100. n The 3D point cloud image data 100B represents the visual features of the captured subject as a point cloud dispersed within a virtual space defined by the sensor coordinate system C3. Each point O constituting this point cloud has distance information d, represented as the 3D coordinates Qs(x, y, d) in the sensor coordinate system C3. s y s , z s The coordinate value x in the coordinates Qs mentioned above. s and y s The distance d1, orthogonal to the optical axis A2, is the distance from the visual sensor 14 (specifically, the imaging sensor) to the visual features of the subject. On the other hand, the coordinate value z... s d2 represents the distance d2 along the optical axis A2 from the visual sensor 14 to the visual features of the subject.

[0060] Thus, the image data captured by the vision sensor 14 is 100 n The visual features (edges, surfaces, etc.) of the workpiece 200 reflected in the distance image data 100A or the 3D point cloud image data 100B are represented as the coordinates Qs(x) in the sensor coordinate system C3. s y s , z s ).also, Figure 4 The image data shown is 100. n This only represents a portion of the workpiece 200 within container 202. Processor 32 acquires image data 100 captured by vision sensor 14. n .

[0061] In addition, processor 32 can also execute the captured image data 100n Image processing that removes visual features of environmental objects SR, such as the ground of container 202 and work unit. For example, the coordinates Qs in sensor coordinate system C3 can be used to represent the visual features of environmental objects SR as reflected in the image data captured by the visual sensor 14 located at shooting position Pi. _SR The data is pre-stored in memory 34. Furthermore, processor 32 processes the image data 100 captured in step S1. n Remove coordinates Qs _SR The visual characteristics of the environmental objects represented by SR. The results are as follows: Figure 4 As shown, it can generate image data 100 that only displays workpiece 200. n .

[0062] In step S2, the processor 32 determines whether there is image data 100 obtained by the vision sensor 14 from the workpiece 200 before the operation WK is performed in step S5 (described later). n (First image data). Here, as described later, the processor 32 repeatedly executes the process of steps S1 to S6 until it is determined to be "yes" in step S6. Each time step S1 is executed, multiple image data 100 are sequentially captured by the vision sensor 14. n (n = 1, 2, 3, ...).

[0063] Suppose that in the first execution of step S2, since step S5 has not yet been executed, there is no image data 1000 captured before the operation of step S5. Therefore, in this case, the processor 32 determines "no" in step S2. On the other hand, in the (n+1)th execution of step S2 (n≥1), image data 100 is captured in step S1 executed before the operation WK in step S5 of the nth execution. n Therefore, in this case, the processor 32 determines "yes" in step S2. The processor 32 proceeds to step S3 if it determines "yes" or "no", and proceeds to step S4 if it determines "no".

[0064] In step S3, processor 32 executes the first detection scheme. (Refer to...) Figure 5 The following describes step S3. In step S11, processor 32 performs a search for image data 100 acquired in the most recent step S1. n The search process SP for workpiece 200 reflected in (n=1).

[0065] For example, it was achieved in the most recent step S1. Figure 4 The image data shown is 100. n In this case, the processor 32 will combine the workpiece model 200M obtained by modeling the workpiece 200 with the image data 100.n The visual features (i.e., distance images or point clouds) of the workpiece 200 reflected in the image are matched. For example, the workpiece model 200M could be a CAD model 200M of the workpiece 200. C CAD model 200M C It is a 3D CAD model, pre-made by an operator using a CAD device (not shown).

[0066] As another example, the workpiece model 200M can also be represented by a point cloud (or normals) as the CAD model 200M. C Point cloud model of model components (edges, faces, etc.) 200M P In this case, processor 32 can obtain a 200M CAD model from the CAD device. C The 200M CAD model was generated according to a predetermined image generation algorithm. C The model components are assigned to the point cloud, thereby generating a 200M point cloud model. P .

[0067] As part of the search process SP, processor 32 compares the workpiece model 200M in various poses with the image data 100. n The visual features of the workpiece 200 reflected in the image are matched sequentially. Furthermore, each time the processor 32 matches the workpiece model 200M, it calculates a score SC as the matching result. This score SC represents the visual features of the image data 100. n The similarity (or difference) between the visual features reflected in the image and the workpiece model 200M is determined by the score SC. The higher (or lower) the score SC, the more similar the two are.

[0068] Furthermore, when the calculated score SC exceeds a specified threshold, the processor 32 determines the image data as 100. n The visual features are highly matched with the 200M workpiece model. Figure 6 This represents a 200M workpiece model and 100 image data. n The multiple workpieces 200, including workpieces 200A, 200B, 200C, 200D, 200E, and 200F, are highly matched.

[0069] Here, a workpiece coordinate system C4 is set for each workpiece model 200M in various postures to define the position and posture of the workpiece model 200M. The processor 32 obtains the coordinates Qs in the sensor coordinate system C3 of each workpiece model 200M that matches workpieces 200A~200F, respectively. _A Qs _B Qs _C Qs _D Qs _E and Qs _F .

[0070] These coordinates Qs _A ~Qs _F Let Qs(x) be the coordinates of sensor coordinate system C3. s y s , z s w s p s r s ). In coordinates Qs _A ~Qs _F In the middle, coordinates (x s y s , z s ) represents the origin of the workpiece coordinate system C4 in the sensor coordinate system C3 (i.e., the position of workpiece 200A~200F), and the coordinates (w) s p s r s ) represents the orientation of each axis of the workpiece coordinate system C4 relative to the sensor coordinate system C3 (i.e., the posture of the workpiece 200A~200F) (the so-called yaw, pitch, and roll).

[0071] Furthermore, the processor 32 uses the known transformation matrix MX to obtain the coordinates Qs of the sensor coordinate system C3. _A Qs _B Qs _C Qs _D Qs _E and Qs _F Convert to robot coordinate system C1 coordinates Qr _A Qr _B Qr _C Qr _D Qr _E and Qr _F These coordinates Qr _A ~Qr _F Let Qr(x) be the coordinates of robot coordinate system C1. r y r , z r w r p r r r ).

[0072] In this way, processor 32 executes the process of combining workpiece model 200M with image data 100M. n Match to search the image data 100 n The search process SP for workpiece 200 in the image is used to detect image data 100. n The processor 32 displays workpieces 200A to 200F. Furthermore, it acquires the position data PD (i.e., coordinates Q) of the detected workpieces 200A to 200F. sand Q r Therefore, processor 32 serves as the workpiece search unit 44 that executes the search process SP. Figure 2 It can perform its functions.

[0073] In step S12, the processor 32 determines whether workpiece 200 was detected in the previous step S11. If at least one workpiece 200 is detected, the processor 32 determines "yes" and proceeds to step S13; otherwise, if no workpiece 200 is detected, the processor determines "no" and proceeds to step S14. Figure 3 Step S6 in the process.

[0074] In step S13, the processor 32 generates a detection result list 110. Figure 7 This represents an example of data construction for test result list 110. Figure 7 In the example shown, column 112, marked "No," represents the identification number (or job sequence) of workpieces 200A~200F detected by the search process SP. Additionally, column 114, marked "Position Data," represents the position data PD of workpieces 200A~200F obtained through the search process SP (specifically, the coordinates Qr in robot coordinate system C1). _A ~Qr _F Additionally, column 116, displayed as "Status," indicates the progress of operation WK for workpieces 200A~200F identified by identification number "No."

[0075] Thus, in step S13, processor 32 assigns identification numbers to workpieces 200A~200F detected by the search process SP, and associates these identification numbers with a detection result list 110 that stores the position data PD of workpieces 200A~200F and tag information ("job waiting", "job successful", "job failed", "job held", etc.) indicating the progress of the operation WK for workpieces 200A~200F. Processor 32 stores the generated detection result list 110 in memory 34. Based on the detection result list 110, processor 32 manages the progress of the position data PD and operation WK of the multiple detected workpieces 200A~200F. After step S13, processor 32 proceeds to... Figure 3 Step S5 in the process.

[0076] Furthermore, the processor 32 can also label the detected workpieces 200A to 200F with identification numbers "No" according to a predetermined priority order. For example, the processor 32 can label multiple detected workpieces 200A to 200F with identification numbers "No" according to the order of the z-coordinate values ​​of the coordinates Qr obtained as position data PD (i.e., the order of vertical height from high to low). Therefore, in this case, in the detection result list 110, multiple position data PDs are arranged according to priority (e.g., z-coordinate values). In addition, the processor 32 can also display the generated detection result list 110 on the display device 38.

[0077] Refer again Figure 3 In step S5, the processor 32 performs job WK on workpiece 200. Specifically, the processor 32 refers to the detection result list 110 stored in memory 34 at that time point and obtains the position data PD with the "status" "job waiting" and the identification number "No" as the topmost. For example, in Figure 7 In the example case, processor 32 obtains the position data PD, i.e., coordinates Qr, of workpiece 200A, which has been assigned the identification number "No.1". _A Furthermore, processor 32, based on coordinates Qr... _A The robot 12 is made to move, and the end effector 28 performs the operation on the workpiece 200A.

[0078] Upon successful completion of operation WK for workpiece 200A, processor 32 changes the "status" of location data PD with identification number "No.1" to "operation successful" in the detection result list 110. Alternatively, processor 32 can remove location data PD with identification number "No.1" as "operation successful" from the detection result list 110. This updates the detection result list 110.

[0079] In step S6, the processor 32 determines whether operation WK has been completed for all workpieces 200 within the container 202. If the processor 32 determines "yes," it terminates the operation. Figure 3 The process shown, on the other hand, returns to step S1 if the result is "no". Then, the processor 32 executes step S1 for the (n+1)th time, and captures image data 100 again through the vision sensor 14. n+1 The image data is 100. n+1 (Second image data) is image data obtained by the vision sensor 14 taking pictures of the workpiece 200 inside the container 202 after the operation WK was performed in the most recent step S5.

[0080] Here, when operation WK is performed on a workpiece 200 in step S5, there is a situation where the configuration of workpieces 200 surrounding that workpiece 200 changes (so-called collapse). Below, we will discuss the workpiece 200A (…) performed by robot 12 in step S5… Figure 4 The workpiece operation is performed as job WK, and the process of removing workpiece 200A from container B is described.

[0081] In this case, Figure 8 This represents the image data 100 captured in step S1 at the (n+1)th iteration. n+1 One example. In Figure 8 The image data shown is 100. n+1 middle, Figure 4 The image data shown is 100. n Workpiece 200A is removed, and the configuration of workpieces 200B, 200C, 200G, and 200H located around it changes. On the other hand, the configuration of workpieces 200 (e.g., workpieces 200D, 200E, and 200F) located away from the removed workpiece 200A remains unchanged.

[0082] Refer again Figure 3 Image data 100 was obtained by executing step S1 for the (n+1)th time. n+1 In this case, processor 32 determines "yes" in subsequent step S2 and proceeds to step S4. See below for reference. Figure 9 Step S4 will be explained. In step S21, the processor 32 calculates image data 100. n The distance d included is related to the image data 100. n+1 The difference δ of the distance d included n Here, image data 100 n ( Figure 4 The image data is captured before the nth step S5 operation. On the other hand, image data 100... n+1 ( Figure 8 The image data is captured after the nth step S5 operation.

[0083] As an example, when image data 100 is at a distance from image data 100A, processor 32 calculates image data 100. n The z-coordinate value of the sensor coordinates Qs of the visual feature represented by a pixel PX. s_n Image data 100 corresponding to that one pixel PX n+1 The z-coordinate value of the sensor coordinates Qs of the visual features reflected by the pixel PX. s_n+1 The difference δ n =z s_n+1 -zs_n , to be used as difference δ n . Reference Figure 10 Regarding the difference δ n The concept will be explained.

[0084] like Figure 10 As shown in (a), we focus on the coordinates Qs(x) located in the sensor coordinate system C3. s y s A pixel PX at position ) . For example Figure 10 As shown in (b), the image data 100 before the operation. n In the image, pixel PX represents the visual features of workpiece 200. n The z-coordinate value of the sensor coordinate system C3 is z s_n On the other hand, in the image data 100 after the operation n+1 In the image, pixel PX represents the visual features of workpiece 200. n+1 The z-coordinate value of the sensor coordinate system C3 changes as z s_n+1 (>z) s_n Processor 32 will have a difference δ n Find δ n =z s_n+1 -z s_n Thus, processor 32 targets image data 100 n And 100 n+1 For each of all pixels PX, calculate the difference δ. n .

[0085] Next, refer to Figure 11 The difference δ is calculated when image data 100 is 100B of 3D point cloud image data. n The concept will be explained. First, the processor 32 quantizes the image data 100 in the unit pixel region PE. n and 100 n+1 Specifically, such as Figure 11 As shown in (a), processor 32 is used to specify image data 100 n and 100 n+1 In the xy plane of the sensor coordinate system C3, a unit pixel region PE with a unit area is defined. This unit pixel region PE is a virtual region that mimics the pixel PX. Like the pixel PX, it is defined as multiple unit pixel regions PE arranged in a grid pattern along the x-axis and y-axis directions of the sensor coordinate system C3.

[0086] Next, the processor 32 processes the image data 100 before the operation. n The point O located within each unit pixel region PE is determined. Figure 11In the example shown in (a), three points O exist within a unit pixel region PE. Furthermore, for each unit pixel region PE, the processor 32 determines the representative point Or based on the points O existing within it. n .

[0087] More specifically, processor 32 calculates the z-coordinate values ​​of three points O within the unit pixel region PE in the sensor coordinate system C3. s average And, as Figure 11 As shown in (b), the processor 32 defines a sensor coordinate system C3 with the calculated average value z. sAVE z-coordinate value s_n Representative point Or n The representative point Or is defined in this way. n The coordinates Qs of the sensor coordinate system C3 n Represented as Qs n (x) s y s ,zs_n)(z s_n =z sAVE In addition, the representative point Or n The coordinates (x) in the coordinates Qs s y s The location is determined to be any position within the unit pixel region PE (e.g., the center position of the unit pixel region PE).

[0088] Using the same method, processor 32 processes 100 image data after the job. n+1 In the above, based on the points O existing within each unit pixel region PE, a representative point Or is defined. n+1 This represents point Or n+1 The coordinates Qs of the sensor coordinate system C3 n+1 Represented as Qs n+1 (x) s y s , z s_n+1 Processor 32 will have a difference δ n Find δ n =z s_n+1 -z s_n Thus, processor 32 targets image data 100 n And 100 n+1 For each unit pixel region PE defined in the code, find the representative point Or. n With Or n+1 The difference δ n .

[0089] Using the above method, the processor 32 obtains the image data 100 in step S21. nThe included distance d (specifically, distance d2 = coordinate value z) s_n ) and image data 100 n+1 The included distance d (coordinate value z) s_n+1 The difference δ n Therefore, processor 32 is used to calculate the difference δ. n The difference obtained is part 52 ( Figure 2 It can perform its functions.

[0090] In step S22, the processor 32 determines whether the result of the previous step S21 has produced a difference δth greater than a predetermined threshold. n Specifically, the processor 32 will calculate the difference δ for each pixel PX or unit pixel region PE in the previous step S21. n The absolute value (=|z s_n+1 -z s_n |) is compared with the threshold δth, and at least one difference δ n The absolute value is above the threshold δth (|δ n In the case of |≥δth), it is determined as "yes".

[0091] On the other hand, processor 32 does not generate a difference δ above the threshold δth. n If the condition is "no", proceed to step S29. Alternatively, in step S22, the processor 32 can also determine "no" if a difference δth or higher than the threshold δth is generated. n If the number N of pixels PX or unit pixel regions PE is above the specified threshold Nth, it is judged as "yes".

[0092] In step S23, processor 32 generates difference image data 120. Difference image data 120 is the difference δ calculated in the most recent step S21. n Image data represented in sensor coordinate system C3. Figure 12 This schematically illustrates an example of difference image data 120. For example... Figure 12 As shown, in the difference image data 120, the difference δ is calculated for each pixel PX or unit pixel region PE in the most recent step S21. n The image is displayed as a sensor coordinate system C3 (i.e., the area given brightness or luminance).

[0093] The following is for reference Figure 13 and Figure 14 The data construction of the difference image data 120 when the image data 100 is at a distance of 100A from the image data 100A is explained. Figure 13 The pixel PX shown in (a) corresponds to Figure 10 Pixel PX is shown in (a). In Figure 13In the example shown, processor 32, in the difference image data 120, generates a difference δ n ( Figure 10 (b)) pixel PX, such as Figure 13 As shown in (b), at coordinate value z s The position = 0 is assigned a difference pixel 122 with a specified brightness (or luminance).

[0094] On the other hand, Figure 14 In the example shown, processor 32, in the difference image data 120, generates a difference δ n ( Figure 10 (b)) pixel PX, such as Figure 14 As shown in (b), at coordinate value z s For z s_n ≤z s ≤z s_n+1 The difference pixel 124 is assigned within a range. That is, the difference pixel 124 has a range from Figure 10 The visual feature PX shown in (b) n Extending to visual features PX n+1 The difference δ in the z-axis direction of the sensor coordinate system C3 is equivalent to n The range.

[0095] Next, refer to Figure 15 as well as Figure 16 The data construction of the difference image data 120 when image data 100 is three-dimensional point cloud image data 100B is explained. Figure 15 The unit pixel region PE shown in (a) corresponds to Figure 11 (b) shows the unit pixel region PE. Figure 15 In the example shown, processor 32, in the difference image data 120, generates a difference δ n ( Figure 11 The unit pixel region PE of (c) is as follows: Figure 15 As shown in (b), at coordinate value z s The position = 0 is assigned a difference pixel of 126 with a specified brightness.

[0096] On the other hand, Figure 16 In the example shown, processor 32, in the difference image data 120, generates a difference δ n ( Figure 11 The unit pixel region PE of (c) is as follows: Figure 16 As shown in (b), at coordinate value z s For z s_n ≤z s ≤z s_n+1 The difference pixel 128 is assigned within a range. The difference pixel 128 has a range from... Figure 11The representative point Or shown in (c) n To the representative point Or n+1 The range.

[0097] about Figure 15 as well as Figure 16 The difference pixels 126 and 128 shown are used to... Figure 11 The representative point Or shown in (b) and (c) n And Or n+1 The corresponding method is assigned a value, and has a relationship with the representative point Or. n And Or n+1 The coordinates Qs(x) in the same sensor coordinate system C3 s y s As described above, processor 32 generates the difference δ that will be calculated in the most recent step S21. n The difference image data 120 is represented in the sensor coordinate system C3 as difference pixels 122, 124, 126, or 128. Therefore, the processor 32 serves as the image data generation unit 54 that generates the difference image data 120. Figure 4 It can perform its functions.

[0098] In step S24, processor 32 determines that a difference δ has been generated. n Image region 130. Specifically, in the difference image data 120 generated in the previous step S23, the processor 32 performs a pixel linking process CP, which connects multiple difference pixels 122, 124, 126 or 128 located nearby to form a collection.

[0099] As an example, in step S23, difference pixels 122 (100A) are generated at a distance of 100A from the image data. Figure 13 ) or 124 ( Figure 14 In the case of a pixel joining process CP, the processor 32 connects the difference pixels 122 or 124 assigned to multiple adjacent pixels PX to form a collection. Alternatively, the processor 32 may also connect the difference pixels 122 or 124 located within a specified distance from each other in the sensor coordinate system C3 to form a collection during the pixel joining process CP.

[0100] As another example, in step S23, the difference pixels 126 of the three-dimensional point cloud image data 100B are generated. Figure 15 ) or 128 ( Figure 16 In the case of pixel joining process CP, processor 32 connects the difference pixels 126 or 128 assigned to multiple adjacent unit pixel regions PE to form a collection. Alternatively, processor 32 may also connect the difference pixels 126 or 128 located within a specified distance from each other in sensor coordinate system C3 to form a collection during pixel joining process CP.

[0101] By executing the pixel linking process CP above, the processor 32 can determine the difference d generated in the sensor coordinate system C3 based on the difference image data 120 generated in step S23. n Image region 130 consisting of a collection of 122, 124, 126, or 128 pixels. Figure 17 An example of such a defined image region 130 is schematically shown. Thus, in this embodiment, the processor 32 determines the difference d generated in the sensor coordinate system C3. n Image region 130 region determination unit 56 ( Figure 2 It can perform its functions.

[0102] In step S25, the processor 32 defines a detection range 132 (first detection range). The detection range 132 is defined in the subsequent step S26 within the image data 100. n+1 The internal detection range of the workpiece 200. The processor 32 first determines the pattern 134 that surrounds the image region 130 determined in the previous step S24 in the sensor coordinate system C3 of the specified difference image data 120.

[0103] As an example, such as Figure 18 As shown, the processor 32 determines the graphic 134 in the sensor coordinate system C3 as the circumscribed shape of a polygon (specifically, a quadrilateral) that is circumscribed to the image region 130. Specifically, the processor 32 determines the difference pixels 122, 124, 126, or 128 constituting the image region 130 that are located at both ends of the x-axis and y-axis directions of the sensor coordinate system C3, and determines the coordinates Qs1 (x, y, y) of these difference pixels in the sensor coordinate system C3. s1 y s1 ), Qs2(x s2 y s2 ), Qs3 (x s3 y s3 ) and Qs4(x s4 y s4 () Figure 18 ).

[0104] Furthermore, the processor 32 can determine the graph 134 as the x-coordinate value of the sensor coordinate system C3. s1 ≤x s ≤x s2 range [x s1 x s2 The y-coordinate value is y. s3 ≤y s ≤y s4 range [y s3 y s4With the pattern 134 thus determined, the edge 134a of the negative x-axis of the sensor coordinate system C3 of the pattern 134 is connected to the difference pixel of the difference pixels 122, 124, 126 or 128 constituting the image area 130, which is located at the end of the negative x-axis of the sensor coordinate system C3.

[0105] On the other hand, the edge 134b of the sensor coordinate system C3 of pattern 134 in the positive x-axis direction is connected to the difference pixels of the difference pixels 122, 124, 126 or 128 constituting the image region 130 located at the ends of the difference pixels in the positive x-axis direction of the sensor coordinate system C3. Similarly, the edges 134c and 134c of the sensor coordinate system C3 of pattern 134 in the y-axis direction are connected to the difference pixels of the difference pixels 122, 124, 126 or 128 constituting the image region 130 located at the ends of the difference pixels in the negative y-axis direction and the positive y-axis direction of the sensor coordinate system C3. The processor 32 calculates the determined position data of the sensor coordinate system C3 of pattern 134 (e.g., range [x]). s1 x s2 ] and range [y s3 y s4 The coordinate data of ] is stored in memory 34.

[0106] Next, the processor 32 defines a detection range 132 based on the determined pattern 134. For example, the processor 32 defines the determined pattern 134 as the detection range 132. Alternatively, the processor 32 may define the detection range 132 as a range larger than the pattern 134 by shifting each side 134a, 134b, 134c, and 134d of the determined pattern 134 outward by a predetermined offset.

[0107] Alternatively, the operator can operate the input device 40 to pre-input an offset, and the processor 32 can process this offset through the input device 40. Thus, as... Figure 18 As shown, in the sensor coordinate system C3, the detection range 132 is defined such that the image region 130 is included inside. Thus, in this embodiment, the processor 32 serves as the range defining unit 58 for defining the detection range 132. Figure 2 It can perform its functions.

[0108] In step S26, the processor 32 functions as the workpiece search unit 44, searching the image data 100. n+1 In this process, a search process SP is performed within the detection range 132 defined in the previous step S25. Specifically, the processor 32 reads the position data (range [x] of the sensor coordinate system C3 within the detection range 132 defined in the previous step S25) from the memory 34. s1 x s2 ], range [ys3 y s4 ]), and set the image data 100 captured in the most recent step S1. n+1 The sensor coordinate system is C3. Figure 19 This indicates the state.

[0109] Furthermore, the processor 32 functions as a workpiece search unit 44, targeting image data 100. n+1 The inner region of the detection range 132 (range [x]) s1 x s2 ] and range [y s3 y s4 The above search process SP is performed in the area 132, and the workpiece model 200M and image data 100 are compared within the detection range 132. n+1 The 200 workpieces shown in the image are matched sequentially. The result is as follows: Figure 20 As shown, processor 32 newly detects three workpieces 200B, 200C and 200G within the detection range 132.

[0110] Processor 32 acquires the coordinates Qs in the workpiece coordinate system C4 and the sensor coordinate system C3 of the workpiece model 200M that matches the newly detected workpieces 200B~200G. _B '、Qs _C 'and Qs _G Furthermore, processor 32 will use these coordinates Qs _B '、Qs _C 'and Qs _G Convert to robot coordinate system C1 coordinates Qr _B '、Qr _C 'and Qr _G In this way, the processor 32 obtains the position data PD (coordinates Qs and Qr) of the newly detected workpieces 200B~200G.

[0111] On the other hand, in step S26, processor 32 processes image data 100. n+1 The search process SP is not performed on the outer region of the detection range 132. In step S27, the processor 32, as in step S12 above, determines whether the workpiece 200 was detected in the previous step S26. If the determination is "yes", the processor 32 proceeds to step S28; otherwise, if the determination is "no", it proceeds to step S29.

[0112] In step S28, the processor 32 updates the detection result list 110. Specifically, the processor 32 retrieves the position data PD stored in the detection result list 110 stored in the memory 34 at that time point from the detection result list 110, which is located outside the detection range 132 defined in the most recent step S25.

[0113] For example, storing at this point in time Figure 7 The test results list 110 is shown below. In this case, as... Figure 19 as well as Figure 20 As shown, workpieces 200D, 200E, and 200F are located outside the detection range 132, and the position data PD (coordinates Qr) of these workpieces are... _D Qr _E and Qr _F The results are stored in the detection result list 110 as identification numbers “No.4”, “No.5”, and “No.6”.

[0114] Therefore, at this time, the processor 32 obtains the location data PD (coordinates Qr) of the identification numbers "No.4", "No.5" and "No.6" from the detection result list 110. _D Qr _E and Qr _F Furthermore, processor 32 uses the newly acquired location data PD (coordinates Qr) from the most recent step S26. _B '、Qr _C 'and Qr _G ) and location data PD (coordinates Qr) obtained from the detection result list 110 _D Qr _E and Qr _F ), Update the test result list 110.

[0115] Figure 21 This represents an example of the updated detection result list 110. In this detection result list 110, for... Figure 20 The position data PD (coordinates Qr) of workpiece 200G in the middle _G ), the position data PD (coordinates Qr) of workpiece 200D _D ), Position data PD (coordinates Qr) of workpiece 200B _B '), Position data PD (coordinates Qr) of workpiece 200C _C '), Position data PD (coordinates Qr) of workpiece 200E _E ) and the position data PD (coordinates Qr) of workpiece 200F. _FThe identification numbers “No.1” to “No.6” are marked on the workpieces, and they are sorted according to the order of these identification numbers. In addition, the processor 32 can also mark the workpieces 200B, 200C, 200G, 200D, 200E and 200F with the identification number “No” according to the above priority.

[0116] After step S28, processor 32 enters... Figure 3 In step S5, referring to the detection result list 110 stored in memory 34 at that time point, operation WK is performed on workpiece 200. For example, in the stored... Figure 21 In the case of the detection result list 110, the processor 32 obtains the position data PD, i.e., coordinates Qr, of the workpiece 200G that has been assigned the identification number "No.1" and whose "status" is set to "job waiting". _G Workpiece 200G is operated by WK.

[0117] On the other hand, Figure 9 If the result in step S22 or S27 is "no", in step S29, the processor 32 retrieves the detection result list 110 stored in the memory 34 at that time point. Then, the processor 32 proceeds to... Figure 3 In step S5, referring to the obtained detection result list 110, and based on the position data PD in the detection result list 110 where the "status" is "job waiting" and the identification number "No" is the topmost position, job WK is executed.

[0118] As described above, in this embodiment, the processor 32 functions as a difference acquisition unit 52, an image data generation unit 54, a region determination unit 56, and a range delineation unit 58, in the image data 100 n+1 The detection range 132 of the workpiece 200 to be detected is set internally. Therefore, the difference acquisition unit 52, the image data generation unit 54, the area determination unit 56, and the range delineation unit 58 constitute the device 50 for setting the detection range 132. Figure 2 ).

[0119] In this device 50, the difference acquisition unit 52 calculates the first image data 100. n ( Figure 4 The distance d (d2 = z) included in ) s_n ) and the second image data 100 n+1 ( Figure 8 The distance d (d2 = z) included in ) s_n+1 The difference δ n (=z) s_n+1 -z s_n (Step S21), wherein the first image data 100 nThe data is obtained by the vision sensor 14 capturing images of workpiece 200 before operation WK, and the second image data is 100. n+1 The data is obtained by the vision sensor 14 taking pictures of workpiece 200 after operation WK.

[0120] Furthermore, the region determination unit 56 determines the generated difference δ in the sensor coordinate system C3 set for the vision sensor 14 to define the image data 100. n Image region 130 ( Figure 17 (Step S24). Then, the range delineation unit 58 delineates the first detection range 132 in the sensor coordinate system C3 in a manner that includes the image region 130 determined by the region determination unit 56. Figure 18 (Step S25).

[0121] Here, when operation WK is performed in step S5, the configuration of workpiece 200 may sometimes change due to reasons such as collapse mentioned above. Such a change in the configuration of workpiece 200 manifests as the aforementioned difference δ. n According to this embodiment, image data 100 n+1 The detection range of workpiece 200 extends from the image data 100. n+1 The entire region is defined as generating the difference δ n The range (i.e., the detection range 132) is reduced. This reduces the computational processing required for the search process SP in step S26, thus shortening the cycle time of job WK.

[0122] Furthermore, in the device 50, the image data generation unit 54 generates the difference δ obtained by the difference acquisition unit 52. n The difference image data 120 is represented in the sensor coordinate system C3. Figure 12 (Step S23). Then, the region determination unit 56 determines the image region 130 based on the difference image data 120 generated by the image data generation unit 54. According to this structure, the difference δ generated in the sensor coordinate system C3 can be effectively determined based on the difference image data 120. n Image region 130.

[0123] Furthermore, the processor 32 may not generate the difference image data 120, but instead function as a region determination unit 56 to determine the image region 130. For example, in step S21 described above, the processor 32 may generate the difference δ... n The coordinates Qs of the pixel PX or unit pixel region PE in the sensor coordinate system C3 and the difference δ n The database is linked together and stored. Furthermore, the processor 32 can determine the image region 130 in the sensor coordinate system C3 based on this database. That is, in this case, the image data generation unit 54 can be omitted from the device 50.

[0124] Furthermore, in device 50, the region determination unit 56 determines a difference δ that has been generated above a predetermined threshold δth in the sensor coordinate system C3. n Image region 130. Based on this structure, the extremely small difference δ, considered as an error, can be ignored. n On the other hand, it can effectively extract the difference δ caused by the aforementioned collapse, etc. n Image region 130. Alternatively, instead of setting a threshold δth, the region determination unit 56 can determine all regions that have generated differences δth. n Image region 130.

[0125] Furthermore, in the device 50, the range delineation unit 58 determines the pattern 134 of the image region 130 defined by the outer surrounding region determination unit 56 in the sensor coordinate system C3. Figure 18 The first detection range 132 is defined based on the determined pattern 134. According to this structure, in image data 100... n+1 In the sensor coordinate system C3, the difference δ can be reliably contained. n The detection range is defined by using an image region of 130.

[0126] Furthermore, the range delineation unit 58 may not determine the image 134, but instead delineate the detection range 132. For example, the range delineation unit 58 may delineate the image region 130 itself as the detection range 132, or it may delineate the image data 100. n+1 The region is divided into a region where the image region 130 exists and a region where the image region 130 does not exist, and the former is designated as the detection range 132.

[0127] Furthermore, in the device 50, the range delineation unit 58 determines the outer graphic 134 that is circumscribed outside the image region 130. According to this structure, it is possible to reliably include the graphic generated by the difference δ. n The size of the detection range 132 of the image region 130 is optimized, and a pattern 134 can be determined for the image region 130 using a relatively simple algorithm. Furthermore, the range delineation unit 58, as the pattern 134, is not limited to an external pattern, but can also determine a pattern of any shape that includes the image region 130.

[0128] Furthermore, in this embodiment, the robot system 10 includes: a vision sensor 14, a workpiece search unit 44, a robot 12, and the aforementioned device 50. The workpiece search unit 44 is located in the second image data 100. n+1 In this process, a search process SP is performed within the first detection range 132 defined by the range definition unit 58, while the search process SP is not performed outside the first detection range 132 (step S26).

[0129] According to this structure, in image data 100 n+1 In the middle, only the difference δ was generated n The search process SP is executed within the detection range 132, which can detect the workpiece 200 within the detection range 132 and obtain its position data PD. On the other hand, the search process SP can be omitted for workpieces outside the detection range 132, for example, by using the position data PD of previously detected workpieces 200. Therefore, the cycle time of operation WK can be shortened.

[0130] Furthermore, in the above-described embodiment, when the image data 100 is three-dimensional point cloud image data 100B, the processor 32 determines a representative point Or in the unit pixel region PE in step S21. n The situation has been described. However, it is not limited to this; processor 32 may also not determine the representative point Or. n Instead, it targets each point O within a unit pixel region PE. Figure 11 (a) Obtain the coordinates Qs of the sensor coordinate system C3, and calculate the image data 100 before the operation. n The z-coordinate values ​​of each point O reflected in the image. s_n Image data after the operation 100 n+1 The z-coordinate values ​​of each point O reflected in the image. s_n+1 The difference δ n .

[0131] Next, refer to Figure 22 Other examples of the motion flow of the robot system 10 will be described below. In this embodiment, the processor 32 executes step S4. Figure 23 The process is shown below. Figure 23 In step S4 shown, processor 32 enters the following state after step S28: Figure 22 In step S7, on the other hand, after step S29, proceeding to... Figure 22 Step S5 in the process.

[0132] In step S7, the processor 32 performs the job WK on the workpiece 200, similar to step S5 described above. Specifically, the processor 32 refers to the detection result list 110 stored in the memory 34 at that time point and obtains the position data PD with the "status" set to "job waiting" and the identification number "No" being the topmost. For example, in Figure 7 In the example shown, the processor 32 performs operation WK on the workpiece 200A which has been assigned the identification number "No.1". In the detection result list 110, the "status" of the location data PD of the identification number "No.1" is changed to "operation successful" (or, the location data PD of the identification number "No.1" is deleted from the detection result list 110).

[0133] In this embodiment, the processor 32 changes the "status" of workpiece 200 that is within a specified distance range relative to workpiece 200A, which is identified as "Job Successful," to "Job Reservation" in the detection result list 110. For example, in Figure 6 In the example shown, regarding the six workpieces 200A~200F detected in the search process SP of the most recent step S26, let's assume that workpieces 200B and 200C exist within a specified distance Δ relative to workpiece 200A, which has been "successfully completed," while workpieces 200D, 200E, and 200F exist outside the distance Δ. In this case, as... Figure 24 As shown, processor 32 changes the "status" of workpieces 200B and 200C with identification numbers "No.2" and "No.3" to "job held". Then, processor 32 proceeds to step S8.

[0134] In step S8, the processor 32 determines whether there is a workpiece 200 that can be operated. Specifically, the processor 32 refers to the detection result list 110 stored in the memory 34 at that time point to determine whether there is a workpiece 200 whose "status" is "job waiting". For example, in Figure 24 In the example shown, the "status" of workpieces 200D to 200F with identification numbers "No.4" to "No.6" is "job waiting". Therefore, in this case, the processor 32 determines "yes" and returns to step S7.

[0135] Then, in step S7, the processor 32 executes job WK on the workpiece 200D whose "status" is "job waiting" and whose identification number "No" is the topmost "No.4" in the detection result list 110 stored in the memory 34 at that time. Thus, in this embodiment, when the processor 32 completes the job WK for a workpiece 200A, it continuously executes job WK for workpieces 200D that are located outside the range of a specified distance Δ relative to that workpiece 200A.

[0136] Here, when performing operation WK on a workpiece 200A, the nearby workpieces 200B and 200C are highly likely to change position due to collapse or other reasons. On the other hand, the workpieces 200D to 200F, located far from workpiece 200A, are less likely to change position due to collapse or other reasons. In this embodiment, by continuously performing operation WK on workpiece 200D, which is located far from workpiece 200A after operation WK has been completed, the cycle time can be shortened.

[0137] Next, refer to Figure 25 ,right Figure 3Other examples of step S4 will be described below. In this embodiment, processor 32 executes steps S30 and S31 after step S25. In step S30, processor 32 determines the detection range 132 defined in the sensor coordinate system C3 in the previous step S25 and the image data 100 in the search process SP of the most recent step S11 or S26. n Does the occupied area of ​​the matched workpiece model 200M overlap?

[0138] For example, suppose processor 32, in the most recent search process SP, ... Figure 26 As shown, the workpiece model 200M and image data 100 are combined. n The sensor coordinate system C3 is used for matching. In this example, in sensor coordinate system C3, the workpiece model 200M is matched with the image data 100. n The workpieces 200A, 200B, 200C and 200I shown in the image are matched. A portion of the area occupied by the workpiece model 200M that matches workpiece 200I overlaps with the detection range 132 defined in the previous step S25.

[0139] In step S30, processor 32, based on the image data 100 from the most recent search process SP, n The position data PD of the matched workpiece model 200M and the position data of the detection range 132 defined in the previous step S25 are used to determine whether a part of the occupied area of ​​the workpiece model 200M overlaps with the detection range 132.

[0140] For example, processor 32 determines that edge 134c of the detection range 132 will be defined. Figure 18 The processor 32 determines whether the virtual plane extended along the z-axis of the sensor coordinate system C3 intersects with the workpiece model 200M that matches the workpiece 200I. If the virtual plane intersects with the workpiece model 200M, the processor 32 determines that a portion of the area occupied by the workpiece model 200M that matches the workpiece 200I overlaps with the detection range 132. If the determination is "yes", the processor 32 proceeds to step S31; otherwise, if the determination is "no", the processor proceeds to step S26.

[0141] In step S31, the processor 32 functions as a range delineation unit 58, further delineating the detection range 136 (second detection range) in the sensor coordinate system C3. Specifically, as follows... Figure 27 As shown, the processor 32 further defines the detection range 136 based on the detection range 132, in a manner that includes the occupied area of ​​the workpiece model 200M that was determined to overlap with the detection range 132 in the previous step S30.

[0142] Thus, detection ranges 132 and 136 are defined in the sensor coordinate system C3. Then, in... Figure 25 In step S26, the processor 32 functions as the workpiece search unit 44, and retrieves the image data 100 captured in the most recent step S1. n+1 In the sensor coordinate system C3, the detection ranges 132 and 136 are set, and the search process SP is performed within the detection ranges 132 and 136.

[0143] Thus, in this embodiment, in conjunction with the first image data 100 n If a portion of the occupied area of ​​the matched workpiece model 200M in the sensor coordinate system C3 overlaps with the first detection range 132, the range delineation unit 58 also delineates the second detection range 136 in the sensor coordinate system C3 in a manner that includes the occupied area.

[0144] According to this structure, the number of workpieces 200 that can be detected in the search process SP can be increased, thus improving the efficiency of the search process SP. Furthermore, it can reliably detect whether the workpiece 200 is within the second detection range 136 and the current posture of the workpiece 200, thus reducing the possibility of operation WK failure and improving the accuracy of operation WK. In addition, it can also... Figure 25 Steps S30 and S31 are applied to Figure 23 The process. In this case, processor 32 can... Figure 23 Steps S25 are followed by steps S30 and S31.

[0145] Next, refer to Figure 28 ,right Figure 3 Another example of step S4 will be described. In this embodiment, the processor 32 executes steps S32 and S33 after step S25. In step S32, the processor 32 determines whether the image data 100 n Does image data 100 exist in the area outside the occupied area in the sensor coordinate system C3 of the workpiece 200 matched by the workpiece model 200M detected in the sensor? n The visual features of the workpiece 200 reflected in the image data 100A (i.e., the visual features of the workpiece 200 reflected in the image data 100A, or the point cloud representing the workpiece 200 reflected in the three-dimensional point cloud image data 100B).

[0146] For example, in the search process SP of the most recent step S11 or S26, processor 32, such as Figure 29 As shown, in image data 100 n In the sensor coordinate system C3, the workpiece model 200M is matched. In this example, in the sensor coordinate system C3, the workpiece model 200M is matched with the image data 100.n The model matches workpieces 200A, 200B, and 200C, but does not match workpiece model 200M with workpieces 200J and 200K.

[0147] In step S32, if the processor 32 determines that visual features of workpieces 200J and 200K exist in an area outside the occupied area in the sensor coordinate system C3 of the workpiece model 200M that matches workpieces 200A, 200B, and 200C, it determines "yes" and proceeds to step S33. On the other hand, if the processor 32 determines "no", it proceeds to step S26.

[0148] In step S33, the processor 32 functions as a range delineation unit 58, further delineating the detection range 138 (third detection range) in the sensor coordinate system C3. Specifically, as follows... Figure 29 As shown, the processor 32 further defines the detection range 138 based on the detection range 132, in a manner that includes the visual features of the workpieces 200J and 200K determined in the previous step S32.

[0149] Thus, detection ranges 132 and 138 are defined in the sensor coordinate system C3. Then, in... Figure 28 In step S26, the processor 32 functions as the workpiece search unit 44, and retrieves the image data 100 captured in the most recent step S1. n+1 In the sensor coordinate system C3, detection ranges 132 and 138 are set, and a search process SP is performed within these detection ranges 132 and 136. Furthermore, Figure 29 This example illustrates a portion of the visual features of workpieces 200J and 200K overlapping with the detection range 132. However, it should be understood that this is not a limitation; the visual features of workpieces 200J and 200K may also be entirely outside the detection range 132.

[0150] Thus, in this embodiment, the range delineation unit 58 includes the first image data 100. n The areas outside the occupied area in the sensor coordinate system C3 of the workpieces 200A, 200B, and 200C that were detected in the test. Figure 29 Using the visual features of workpieces 200J and 200K, a third detection range 138 is further defined in the sensor coordinate system C3.

[0151] According to this structure, the number of workpieces 200 that can be detected in the search process SP can be increased, thus improving the efficiency of the search process SP. Furthermore, when step S26 is executed, sometimes visual features of workpieces 200 that cannot be detected in the search process SP exist outside the detection range 132. According to this embodiment, by further defining a third detection range 138, workpieces 200 that could not be detected in step S26 can be re-searched in the subsequent step S26.

[0152] In addition, it is also possible to Figure 28 Steps S32 and S33 are applied to Figure 23 The process. In this case, processor 32 can... Figure 23 After step S25, steps S32 and S33 are executed. Alternatively, steps S32 and S33 can also be performed. Figure 28 Steps S32 and S33 are applied to Figure 25 The process. In this case, processor 32 can... Figure 25 Steps S32 and S33 are executed after step S31 (or before step S30).

[0153] Furthermore, while step S25 described above, the case of determining the quadrilateral shape 134 was not limited thereto. The processor 32 can also determine any m-sided shape 134 (m = 3, 4, 5, 6, 8, ...). In this case, the processor 32 can determine the number "m" defining the m-sided shape based on the outer edge shape of the image region 130 determined in step S24.

[0154] For example, the sensor coordinate system C3 is determined as follows: Figure 17 In the case of the image region 130 shown, the processor 32 calculates the shape of the projected outer edge of the image region 130 when viewed from the z-axis direction of the sensor coordinate system C3 onto the xy-plane of the sensor coordinate system C3. Furthermore, the processor 32 can function as a range delineation unit 58, determining the "m" of the m-sided polygon based on the projected outer edge shape to minimize the blank space between the m-sided polygon and the projected outer edge shape. With this structure, the size of the pattern 134 and the detection range 132 can be optimized.

[0155] Furthermore, once image region 130 is determined, processor 32 can use a machine learning model LM to determine the optimal shape 134 (or detection range 132) encompassing image region 130. For example, an operator inputs a dataset DS of data on image regions of various shapes and training data on the shape (or detection range) set for that image region into the machine learning device. Using the dataset DS, the machine learning device learns a machine learning model LM that represents the correlation between the shape of an image region and the shape (or detection range) determined for that image region.

[0156] Furthermore, in step S25, the processor 32 inputs the determined image region 130 into the machine learning model LM. The machine learning model LM outputs a pattern 134 (or detection range 132) that is correlated with the input image region 130 as the optimal solution. The processor 32 can determine the pattern 134 (or detection range 132) output by the machine learning model LM in the sensor coordinate system C3. According to this structure, the optimal pattern 134 (or detection range 132) can be determined for the determined image region 130. In addition, the processor 32 can also be configured to perform the functions of a machine learning device.

[0157] Furthermore, in the above embodiment, the case where the graphic 134 is a polygon (quadrilateral) has been described, but the graphic 134 may also have any shape such as a circle, an ellipse, or a star. Figure 30 This represents an example that determines the circular shape 134 and the detection range 132. In Figure 30 In the example shown, processor 32 can, for example, determine the centroid of a defined image region 130, which determines the center of pattern 134. Furthermore, processor 32 can determine the coordinates Qs of the farthest difference pixels 122, 124, 126, or 128 relative to the center in sensor coordinate system C3, and determine a circular pattern 134 with a radius equal to the distance from the center to the coordinates Qs.

[0158] Furthermore, in step S25 above, the range of x-coordinate values ​​of the processor 32 through the sensor coordinate system C3 is [x s1 x s2 The range of y coordinate values ​​[y] s3 y s4 The case of determining the pattern 134 (i.e., the detection range 132) has been described. However, it is not limited to this; the processor 32 may also further consider the range of z-coordinate values ​​to determine the pattern 134 (detection range 132).

[0159] For example, generating in step S23 Figure 14 (Difference pixels 124) or Figure 16 In step S24, based on the difference image data 120 (difference pixels 128), a determination is made. Figure 17 The image region 130 is shown. In this case, the processor 32 determines the difference pixel at the end of the difference pixel in the positive z-axis direction of the sensor coordinate system C3 among the difference pixels 124 or 128 constituting the image region 130, and determines the coordinates Qs5 (x) of the sensor coordinate system C3. s5 y s5 , z s5 ).

[0160] Furthermore, processor 32 can also access the range [x]s1 x s2 ] and range [y s3 y s4 The z-coordinate value of the sensor coordinate system C3 is 0 ≤ z s ≤z s5 The range is [0, z] s5 The shape 134 (detection range 132) is determined by the method used to determine the shape 134. That is, in this case, the shape 134 (detection range 132) is determined to be a cuboid-shaped three-dimensional region.

[0161] Alternatively, the processor 32 determines the difference pixels in the difference pixels 124 or 128 constituting the image region 130 that are located at the end of the difference pixels in the negative z-axis direction of the sensor coordinate system C3, and determines the coordinates Qs0(x) of the sensor coordinate system C3. s0 y s0 , z s0 Furthermore, processor 32 can also access the range [x] s1 x s2 ] and range [y s3 y s4 The z-coordinate value of the sensor coordinate system C3 is z. s0 ≤z s ≤z s5 The range is [0, z] s5 To determine the graphic 134 (detection range 132).

[0162] Furthermore, the processor 32 can execute according to the computer program PG pre-stored in the memory 34. Figure 3 , Figure 5 , Figure 9 , Figure 22 , Figure 23 , Figure 25 or Figure 28 The process is shown. In addition, the functions of the device 50 (i.e., the difference acquisition unit 52, the image data generation unit 54, the region determination unit 56, and the range delineation unit 58) and the workpiece search unit 44 executed by the processor 32 can also be implemented by a computer program PG as a functional module.

[0163] Furthermore, after step S25, the processor 32 can also display the defined detection range 132 on the display device 38. The operator can also visually confirm the detection range 132 displayed on the display device 38 and operate the input device 40 to provide input for editing the detection range 132. The processor 32 can change the detection range 132 according to the input received through the input device 40.

[0164] Furthermore, the processor 32 can further define a fourth detection range, based on the aforementioned detection ranges 132, 136, and 138, including the occupied area of ​​the workpiece model 200M that matches the workpiece 200 for which the operation WK has been appropriately completed. For example, assuming that the operation WK has been appropriately completed... Figure 6 The workpiece 200A shown is subjected to operation WK, and the positions of the surrounding workpieces 200 do not change due to operation WK. In this case, image data 100 is generated before and after operation WK. n And 100 n+1 The difference in image area is only for workpiece 200A. In this case, processor 32 can include the image data 100 n The fourth detection range is defined by the area occupied by the workpiece model 200M that matches the workpiece 200A.

[0165] The present disclosure has been described in detail above, but it is not limited to the various embodiments described above. Various additions, substitutions, modifications, and partial deletions can be made to these embodiments without departing from the core essence of the present disclosure, or without departing from the core essence of the present disclosure derived from the claims and their equivalents. Furthermore, these embodiments can also be implemented in combination. For example, in the above embodiments, the order of each action or the order of each process has been shown as an example, but it is not a limitation. The same applies to the use of numerical values ​​or mathematical formulas in the description of the above embodiments.

[0166] This disclosure describes the following methods.

[0167] (Method 1) An apparatus 50, which sets the detection range 132, 136, 138 of the workpiece 200 to be detected in the image data 100 captured by the vision sensor 14, wherein the vision sensor 14 is capable of measuring the distance d of the subject, and the apparatus 50 includes: a difference acquisition unit 52, which calculates the first image data 100 n The distance d contained in the second image data 100 n+1 The difference δ of the distance d included n Among them, the first image data 100 n The data is obtained by the vision sensor 14 capturing images of workpiece 200 before the operation; the second image data is 100. n+1 The data is obtained by the vision sensor 14 after the operation, which captures images of the workpiece 200; the area determination unit 56 determines the difference δ generated in the sensor coordinate system C3 set for the vision sensor 14 to define the image data 100. n The image region 130; the range delineation unit 58 delineates a first detection range 132 in the sensor coordinate system C3 in such a way that it includes the image region 130 determined by the region determination unit 56.

[0168] (Method 2) The apparatus 50 according to Method 1, wherein the apparatus 50 further comprises: an image data generation unit 54, which generates the difference δ obtained by the difference acquisition unit 52. n The region determination unit 56 determines the image region 130 based on the difference image data 120 generated by the image data generation unit 54, which is represented in the sensor coordinate system C3.

[0169] (Method 3) According to the apparatus 50 of Method 1 or 2, wherein the region determination unit 56 determines, in the sensor coordinate system C3, a difference δth greater than or equal to a predetermined threshold δth. n Image region 130.

[0170] (Method 4) The apparatus 50 according to any one of Methods 1 to 3, wherein the range delineation unit 58 determines the pattern 134 of the image region 130 determined by the outer surrounding region determination unit 56 in the sensor coordinate system C3, and the range delineation unit 58 delineates the first detection range 132 according to the determined pattern 134.

[0171] (Method 5) According to the apparatus 50 of Method 4, the range delineation unit 58 determines the external graphic 134 that is circumscribed to the image region 130.

[0172] (Method 6) The apparatus 50 according to any one of methods 1 to 5, wherein the robot system 10 is configured to: search for the workpiece 200 reflected in the image data 100 by matching the workpiece model 200M obtained by modeling the workpiece 200 with the image data 100, and in accordance with the first image data 100 n If a portion of the occupied area of ​​the matched workpiece model 200M in the sensor coordinate system C3 overlaps with the first detection range 132, the range delineation unit 58 further delineates the second detection range 136 in the sensor coordinate system C3 in a manner that includes the occupied area.

[0173] (Method 7) The apparatus 50 according to any one of Methods 1 to 6, wherein the robot system 10 is configured to detect the workpiece 200 reflected in the image data 100 by matching the workpiece model 200M obtained by modeling the workpiece 200 with the image data 100, and the range delineation unit 58 further delineates a third detection range 138 in the sensor coordinate system C3 in such a way that the range includes the area in the first image data 100. n The area outside the occupied area in the sensor coordinate system C3 of the workpiece model 200M that matches the workpieces 200A, 200B, and 200C detected in the test.

[0174] (Method 8) A robot system 10 includes: a vision sensor 14 configured to determine the distance to a subject and capture an image of a workpiece 200; a workpiece search unit 44 that performs a search process SP in which a workpiece model 200M obtained by modeling the workpiece 200 is matched with image data 100 captured by the vision sensor 14 to search for the workpiece 200 reflected in the image data 100; a robot 12 that performs a prescribed operation on the workpiece 200 detected in the image data 100 by the search process SP; and a device 50 according to any one of methods 1 to 7, wherein the workpiece search unit 44 is in the second image data 100 n+1 In the process, the search process SP is performed within the first detection range 132 defined by the range definition unit 58, while the search process SP is not performed outside the first detection range 132.

[0175] (Method 9) A method for setting detection ranges 132, 136, and 138 of the workpiece 200 to be detected in image data 100 captured by vision sensor 14, where vision sensor 14 is capable of measuring the distance d of the subject, and processor 32 performs the following actions: derives the first image data 100 n The difference δ between the distance d contained in the first image data and the second distance d contained in the second image data. n Among them, the first image data 100 n The first image data is the data obtained by the vision sensor 14 capturing images of workpiece 200 before operation WK, and the second image data is the data obtained by the vision sensor 14 capturing images of workpiece 200 after operation WK. In the sensor coordinate system C3 set for the vision sensor 14 to define the image data 100, a difference δ is determined. n Image region 130; a first detection range 132 is defined in sensor coordinate system C3 in a manner that includes the defined image region 130.

[0176] (Method 10) A computer program PG that causes processor 32 to execute the method described in Method 9.

[0177] Symbol Explanation

[0178] 10 Robotic Systems

[0179] 12 robots

[0180] 14. Visual Sensors

[0181] 16. Control device

[0182] 32 processors

[0183] 44. Workpiece Search Department

[0184] 50 devices

[0185] 52 Difference Acquisition Department

[0186] 54 Image Data Generation Unit

[0187] 56. Area Determination Department

[0188] 58. Scope Delineation Department

[0189] 100 image data

[0190] 120 different image data.

Claims

1. An apparatus for setting a detection range for a workpiece to be detected based on image data captured by a vision sensor, wherein the vision sensor is capable of determining the distance to the photographed object, characterized in that, The device has: The difference acquisition unit calculates the difference between the distance contained in the first image data and the distance contained in the second image data, wherein the first image data is data obtained by the vision sensor from the workpiece before the operation, and the second image data is data obtained by the vision sensor from the workpiece after the operation; A region determination unit determines, within a sensor coordinate system defined for the visual sensor and used to define the image data, the image region where the difference occurred; and The range delineation unit delineates the first detection range in the sensor coordinate system in a manner that includes the image region determined by the region determination unit.

2. The apparatus according to claim 1, characterized in that, The device further includes an image data generation unit that generates difference image data by representing the difference obtained by the difference acquisition unit in the sensor coordinate system. The region determination unit determines the image region based on the difference image data generated by the image data generation unit.

3. The apparatus according to claim 1, characterized in that, The region determination unit determines, in the sensor coordinate system, the image region where the difference exceeds a predetermined threshold.

4. The apparatus according to claim 1, characterized in that, The range delineation unit determines, in the sensor coordinate system, a shape that surrounds the image region determined by the region determination unit from the outside. The range delineation unit delineates the first detection range based on the determined pattern.

5. The apparatus according to claim 4, characterized in that, The range delineation unit determines the outer graphic that is circumscribed to the image region as the graphic.

6. The apparatus according to claim 1, characterized in that, The workpiece model obtained by modeling the workpiece is matched with the image data to search for the workpiece reflected in the image data. If a portion of the occupied area of ​​the workpiece model matching the first image data in the sensor coordinate system overlaps with the first detection range, the range delineation portion further delineates the second detection range in the sensor coordinate system in a manner that includes the occupied area.

7. The apparatus according to claim 1, characterized in that, The workpiece model obtained by modeling the workpiece is matched with the image data to search for the workpiece reflected in the image data. The range delineation unit further delineates the third detection range in the sensor coordinate system in a manner that includes a region other than the occupied region in the sensor coordinate system of the workpiece model that matches the workpiece detected in the first image data.

8. A robot system, characterized in that, have: A vision sensor, configured to determine the distance to a subject and photograph the workpiece; The workpiece search unit performs the following search process: by matching the workpiece model obtained by modeling the workpiece with the image data captured by the vision sensor, the workpiece reflected in the image data is searched. A robot that performs a prescribed operation on the workpiece detected in the image data through the search process; as well as The apparatus of claim 1, The workpiece search unit performs the search process within the first detection range defined by the range definition unit in the second image data, but does not perform the search process outside the first detection range.

9. A method for setting a detection range for a workpiece to be detected based on image data captured by a vision sensor, said vision sensor being capable of determining the distance to the photographed object, characterized in that, The processor performs the following actions: The difference between the distance contained in the first image data and the distance contained in the second image data is calculated, wherein the first image data is data obtained by the vision sensor taking pictures of the workpiece before the operation, and the second image data is data obtained by the vision sensor taking pictures of the workpiece after the operation; In the sensor coordinate system set for the vision sensor and used to define the image data, the image region that produced the difference is determined; The first detection range is defined in the sensor coordinate system in a manner that includes the determined image region.

10. A computer program, characterized in that, The processor is made to execute the method of claim 9.

Images