Parameter adjustment device, robot system, method, and computer program

Abstract

Conventionally, an operator with expertise was needed to adjust a parameter for collating a workpiece feature of a workpiece imaged by a visual sensor and a workpiece model of the workpiece.　A device 50 comprises: an image generation unit 52 that generates image data displaying a workpiece feature of a workpiece imaged by a visual sensor; a position detection unit 54 that uses a parameter for collating a workpiece model with a workpiece feature to obtain, as a detected position, a position of the workpiece in the image data; a matching position acquisition unit 58 that acquires, as a matching position, a position of the workpiece model in the image data when the workpiece model is arranged so as to match the workpiece feature in the image data; and a parameter adjustment unit 60 that adjusts the parameter on the basis of data indicating the difference between the detected position and the matching position.

Description

[Technical field]

[0001] The present invention relates to an apparatus, a robot system, a method, and a computer program for adjusting parameters for matching workpiece features of a workpiece in image data with a workpiece model. [Background technology]

[0002] There is known a technique for acquiring parameters for detecting the position of a workpiece shown in image data captured by a visual sensor (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2-210584 Summary of the Invention [Problem to be solved by the invention]

[0004] In some cases, the position of a workpiece shown in image data captured by a visual sensor is determined by comparing the workpiece features of the workpiece shown in the image data with a workpiece model that is a model of the workpiece using parameters. Conventionally, an operator with specialized knowledge was required to adjust the parameters for such comparison. [Means for solving the problem]

[0005] In one aspect of the present disclosure, the device includes a position detection unit that determines the position of the work in image data as a detection position using parameters for matching a work model that models the work in image data in which the work features of the work are displayed and captured by a visual sensor with the work features, a matching position acquisition unit that acquires the position of the work model in the image data as a matching position when the work model is positioned in the image data to match the work features, and a parameter adjustment unit that adjusts the parameters based on data representing the difference between the detection position and the matching position so as to enable the position detection unit to determine the detection position as a position corresponding to the matching position.

[0006] In another aspect of the present disclosure, a method includes a processor using parameters for matching a work model that models the work in image data in which the work features of the work are displayed and captured by a visual sensor with the work features to determine the position of the work in the image data as the detection position, obtain the position of the work model in the image data when the work model is positioned in the image data to match the work features as a matching position, and adjust the parameters based on data representing the difference between the detection position and the matching position so as to enable the processor to determine the detection position as a position corresponding to the matching position. Effect of the Invention

[0007] According to the present disclosure, the parameters are adjusted using the matching position obtained when matching the workpiece model to the workpiece features in the image data. Therefore, even an operator who does not have specialized knowledge about parameter adjustment can obtain the matching position, thereby enabling the adjustment of the parameters. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of a robot system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram of the robot system shown in FIG. [Diagram 3] 1 is a flowchart illustrating a parameter adjustment method according to an embodiment. [Figure 4] 4 shows an example of image data generated in step S3 in FIG. [Diagram 5] An example of the flow of step S4 in FIG. 3 is shown. [Figure 6] 6 shows an example of image data generated in step S11 in FIG. 5. [Figure 7] This shows the state in which the workpiece model matches the workpiece features in the image data. [Figure 8] An example of the flow of step S5 in FIG. 3 is shown. [Figure 9] 8 shows another example of the flow of step S4 in FIG. [Figure 10] 10 shows an example of image data generated in step S11 in FIG. [Figure 11] 10 shows another example of image data generated in step S11 in FIG. [Figure 12] 10 shows an example of image data generated in step S32 in FIG. [Figure 13] This shows a state in which work models are displayed randomly in the image data generated in step S11 in FIG. [Figure 14] This shows a state in which the workpiece model is displayed according to a predetermined rule in the image data generated in step S11 in FIG. [Figure 15] 4 shows yet another example of the flow of step S4 in FIG. [Figure 16] An example of image data generated in step S42 in FIG. 15 is shown. [Figure 17] 13 shows an example of a flow for acquiring image data by a visual sensor. [Figure 18] 13 is a flowchart illustrating a parameter adjustment method according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In various embodiments described below, similar elements are given the same reference numerals, and duplicated explanations will be omitted. First, a robot system 10 according to one embodiment will be described with reference to Figs. 1 and 2. The robot system 10 includes a robot 12, a visual sensor 14, and a control device 16.

[0010] In this embodiment, the robot 12 is a vertical articulated robot and includes a robot base 18, a rotating body 20, a lower arm 22, an upper arm 24, a wrist 26, and an end effector 28. The robot base 18 is fixed onto the floor of a work cell. The rotating body 20 is mounted on the robot base 18 so as to be rotatable around a vertical axis.

[0011] The lower arm 22 is rotatably attached to the rotating body 20 around a horizontal axis, and the upper arm 24 is rotatably attached to the tip of the lower arm 22. The wrist 26 has a wrist base 26a rotatably attached to the tip of the upper arm 24, and a wrist flange 26b attached to the wrist base 26a so as to be rotatable around the wrist axis A1.

[0012] The end effector 28 is detachably attached to the wrist flange 26b, and performs a predetermined operation on the workpiece W. In this embodiment, the end effector 28 is a robot hand capable of gripping the workpiece W, and has, for example, a plurality of openable and closable fingers or an adsorption part (a negative pressure generator, a suction cup, an electromagnet, or the like).

[0013] Each component of the robot 12 (robot base 18, rotating body 20, lower arm 22, upper arm 24, wrist 26) is provided with a servo motor 29 (FIG. 2). These servo motors 29 rotate each movable element of the robot 12 (rotating body 20, lower arm 22, upper arm 24, wrist 26, wrist flange 26b) around a drive shaft in response to a command from the control device 16. As a result, the robot 12 can move the end effector 28 and place it in any position and posture.

[0014] The visual sensor 14 is fixed to the end effector 28 (or the wrist flange 26b). For example, the visual sensor 14 is a three-dimensional visual sensor having an image sensor (CMOS, CCD, etc.) and an optical lens (collimator lens, focus lens, etc.) that guides a subject image to the image sensor, and is configured to capture a subject image along an optical axis A2 and measure a distance d to the subject image.

[0015] 1, a robot coordinate system C1 and a tool coordinate system C2 are set for the robot 12. The robot coordinate system C1 is a control coordinate system for controlling the operation of each movable element of the robot 12. In this embodiment, the robot coordinate system C1 is fixed to the robot base 18 such that its origin is located at the center of the robot base 18 and its z axis is parallel to the vertical direction.

[0016] On the other hand, the tool coordinate system C2 is a control coordinate system for controlling the position of the end effector 28 in the robot coordinate system C1. In this embodiment, the tool coordinate system C2 is set with respect to the end effector 28 so that its origin (so-called TCP) is located at the working position (workpiece gripping position) of the end effector 28 and its z axis is parallel to (specifically, coincides with) the wrist axis A1.

[0017] When moving the end effector 28, the control device 16 sets a tool coordinate system C2 in the robot coordinate system C1, and generates a command to each servo motor 29 of the robot 12 to place the end effector 28 at a position represented by the set tool coordinate system C2. In this manner, the control device 16 can position the end effector 28 at any position in the robot coordinate system C1. In this document, "position" may mean both position and orientation.

[0018] Meanwhile, a sensor coordinate system C3 is set for the visual sensor 14. The sensor coordinate system C3 defines the coordinates of each pixel of image data (or the image sensor) captured by the visual sensor 14. In this embodiment, the sensor coordinate system C3 is set for the visual sensor 14 so that its origin is located at the center of the image sensor and its z axis is parallel to (specifically, coincides with) the optical axis A2.

[0019] The positional relationship between the sensor coordinate system C3 and the tool coordinate system C2 is known by calibration, and therefore the coordinates of the sensor coordinate system C3 and the coordinates of the tool coordinate system C2 can be mutually converted via a known conversion matrix (e.g., a homogeneous conversion matrix). In addition, the positional relationship between the tool coordinate system C2 and the robot coordinate system C1 is known, and therefore the coordinates of the sensor coordinate system C3 and the coordinates of the robot coordinate system C1 can be mutually converted via the tool coordinate system C2.

[0020] The control device 16 controls the operation of the robot 12. Specifically, the control device 16 is a computer having a processor 30, a memory 32, and an I / O interface 34. The processor 30 is communicatively connected to the memory 32 and the I / O interface 34 via a bus 36, and performs calculations to realize various functions described below while communicating with these components.

[0021] The memory 32 has a RAM or a ROM, etc., and temporarily or permanently stores various data. The I / O interface 34 has, for example, an Ethernet (registered trademark) port, a USB port, an optical fiber connector, or an HDMI (registered trademark) terminal, and communicates data with an external device via wired or wireless communication under instructions from the processor 30. The servo motors 29 and the visual sensor 14 of the robot 12 are connected to the I / O interface 34 so as to be able to communicate with each other.

[0022] The control device 16 is also provided with a display device 38 and an input device 40. The display device 38 and the input device 40 are communicatively connected to the I / O interface 34. The display device 38 has a liquid crystal display, an organic EL display, or the like, and visibly displays various data under instructions from the processor 30.

[0023] The input device 40 has a keyboard, a mouse, a touch panel, or the like, and receives input data from an operator. The display device 38 and the input device 40 may be integrated into the housing of the control device 16, or may be separate from the housing of the control device 16 and externally attached to the housing.

[0024] In this embodiment, the processor 30 operates the robot 12 to perform a work handling operation in which the end effector 28 grasps and picks up the workpieces W randomly piled up in the container B. To perform this work handling operation, the processor 30 first captures an image of the workpieces W in the container B with the visual sensor 14.

[0025] The image data ID1 captured by the visual sensor 14 at this time includes work features WP that depict the visual features (edges, contours, faces, sides, corners, holes, protrusions, etc.) of each captured work W, and information on the distance d from the visual sensor 14 (specifically, the origin of the sensor coordinate system C3) to the points on the work W represented by each pixel of the work feature WP.

[0026] Next, the processor 30 acquires parameters PM for matching the workpiece model WM, which is a model of the workpiece W, with the workpiece characteristics WP of the workpiece W imaged by the visual sensor 14. The processor 30 then applies the parameters PM to a predetermined algorithm AL (software) and matches the workpiece model WM with the workpiece characteristics WP according to the algorithm AL to acquire data (specifically, coordinates) of the position (specifically, position and orientation) of the workpiece W captured in the image data ID1 in the sensor coordinate system C3. The processor 30 then converts the position of the acquired sensor coordinate system C3 into the robot coordinate system C1 to acquire position data of the imaged workpiece W in the robot coordinate system C1.

[0027] Here, in order to acquire the position of the workpiece W captured in the image data ID1 with high accuracy, it is necessary to optimize the parameters PM. In this embodiment, the processor 30 adjusts the parameters PM to be optimized by using the workpiece characteristics WP of the workpiece W captured by the visual sensor 14.

[0028] A method for adjusting the parameter PM will be described below with reference to Fig. 3. The flow shown in Fig. 3 is started, for example, when the control device 16 is started. At the start of the flow in Fig. 3, the above-mentioned algorithm AL and the parameter PM1 prepared in advance are stored in the memory 32.

[0029] In step S1, the processor 30 determines whether or not a parameter adjustment command has been received. For example, an operator operates the input device 40 to manually input a parameter adjustment command. If the processor 30 receives a parameter adjustment command from the input device 40 through the I / O interface 34, the result is YES, and the process proceeds to step S2. On the other hand, if the processor 30 has not received a parameter adjustment command, the result is NO, and the process proceeds to step S6.

[0030] In step S2, the processor 30 captures an image of the workpiece W using the visual sensor 14. Specifically, the processor 30 operates the robot 12 to position the visual sensor 14 at an imaging position where at least one workpiece W falls within the field of view of the visual sensor 14, as shown in FIG.

[0031] Next, the processor 30 sends an imaging command to the visual sensor 14, and in response to the imaging command, the visual sensor 14 captures an image of the workpiece W to obtain image data ID1. As described above, the image data ID1 includes information on the workpiece characteristics WP of each captured workpiece W and the above-mentioned distance d. The processor 30 obtains the image data ID1 from the visual sensor 14. Each pixel of the image data ID1 is represented as a coordinate in the sensor coordinate system C3.

[0032] In step S3, the processor 30 generates image data ID2 in which the workpiece characteristics WP are displayed. Specifically, the processor 30 generates the image data ID2 as a graphical user interface (GUI) that allows the operator to visually recognize the workpiece characteristics WP, based on the image data ID1 acquired from the visual sensor 14. An example of this image data ID2 is shown in FIG.

[0033] 4, the workpiece feature WP is displayed as a three-dimensional point cloud in the image data ID2. A sensor coordinate system C3 is set in the image data ID2, and each pixel of the image data ID2 is expressed as a coordinate in the sensor coordinate system C3, similar to the image data ID1 captured by the visual sensor 14.

[0034] Each of the points constituting the workpiece feature WP has information on the distance d described above, and can therefore be expressed as three-dimensional coordinates (x, y, z) in the sensor coordinate system C3. That is, in this embodiment, the image data ID2 is three-dimensional image data. Note that, for ease of understanding, FIG. 4 shows an example in which a total of three workpiece features WP are displayed in the image data ID2, but it should be understood that in reality, more workpiece features WP (i.e., workpieces W) may be displayed.

[0035] The processor 30 may generate the image data ID2 as image data different from the image data ID1, as a GUI with better visibility than the image data ID1. For example, the processor 30 may generate the image data ID2 by making the areas other than the workpiece characteristics WP shown in the image data ID1 colorless, while coloring the workpiece characteristics WP (black, blue, red, etc.) so that the operator can easily identify the workpiece characteristics WP.

[0036] The processor 30 displays the generated image data ID2 on the display device 38. In this way, the operator can visually recognize the image data ID2 as shown in Fig. 4. In this way, in this embodiment, the processor 30 functions as an image generating unit 52 (Fig. 2) that generates the image data ID2 in which the workpiece characteristics WP are displayed.

[0037] The processor 30 may update the image data ID2 displayed on the display device 38 in response to an operation of the input device 40 by the operator so as to change the direction in which the workpiece W shown in the image data ID2 is viewed (for example, as in the case of 3D CAD data). In this case, the operator may of By operating this, it becomes possible to view the workpiece W shown in the image data ID2 from the desired direction.

[0038] 3 again, in step S4, the processor 30 executes a process of acquiring a matching position. This step S4 will be described with reference to FIG. 5. In step S11, the processor 30 further displays a work model WM on the image data ID2 generated in the above-mentioned step S3. In this embodiment, the work model WM is 3D CAD data.

[0039] 6 shows an example of image data ID2 generated in step S11. In step S11, the processor 30 places the work model WM in a virtual space defined by the sensor coordinate system C3, and generates image data ID2 of the virtual space in which the work model WM is placed together with the work feature WP of the work W. The processor 30 also sets a work coordinate system C4 together with the work model WM in the sensor coordinate system C3. This work coordinate system C4 is a coordinate system that defines the position (specifically, the position and posture) of the work model WM.

[0040] In this embodiment, in step S11, the processor 30 determines the position of the workpiece W in the image data ID2 as a detection position DP1 using the parameter PM1 stored in the memory 32 at the start of step S11. When determining the detection position DP1, the processor 30 applies the parameter PM1 to an algorithm AL, and matches the workpiece model WM with the workpiece feature WP depicted in the image data ID2 in accordance with the algorithm AL.

[0041] More specifically, the processor 30 gradually changes the position of the work model WM by a predetermined displacement amount E in the virtual space defined by the sensor coordinate system C3 in accordance with the algorithm AL to which the parameter PM1 is applied, and searches for a position of the work model WM where the feature points (edges, contours, faces, sides, corners, holes, protrusions, etc.) of the work model WM coincide with the feature points of the work feature WP corresponding to the feature points.

[0042] Then, when a feature point of the workpiece model WM coincides with a feature point of the corresponding workpiece feature WP, the processor 30 detects the coordinates (x, y, z, W, P, R) in the sensor coordinate system C3 of the workpiece coordinate system C4 set in the workpiece model WM as the detection position DP1. Here, the coordinates (x, y, z) indicate the origin position of the workpiece coordinate system C4 in the sensor coordinate system C3, and the coordinates (W, P, R) indicate the attitude (so-called yaw, pitch, roll) of the workpiece coordinate system C4 with respect to the sensor coordinate system C3.

[0043] The above-mentioned parameters PM1 are for matching the work model WM with the work feature WP, and include, for example, the above-mentioned displacement amount E, the window size SZ that defines the range in which to search for feature points to be matched with each other in the image data ID2, the image roughness (or resolution) σ at the time of matching, and data that specifies which feature points of the work model WM and the work feature WP are to be matched with each other (for example, data that specifies the "contours" of the work model WM and the work feature WP to be matched with each other).

[0044] In this way, the processor 30 obtains the detected position DP1 (x, y, z, W, P, R) by comparing the workpiece model WM with the workpiece feature WP using the parameter PM1. Therefore, in this embodiment, the processor 30 functions as a position detection unit 54 (FIG. 2) that obtains the detected position DP1 using the parameter PM1.

[0045] Next, the processor 30 functions as an image generating unit 52 to display the workpiece model WM at the acquired detection position DP1 in the image data ID2. Specifically, the processor 30 displays the workpiece model WM at a position represented by the workpiece coordinate system C4 arranged at the coordinates (x, y, z, W, P, R) of the sensor coordinate system C3 detected as the detection position DP1. Model Show WM.

[0046] In this way, the three workpiece models WM are displayed at positions corresponding to the three workpiece features WP in the image data ID2, as shown in Fig. 6. Here, if the parameter PM1 is not optimized, the acquired detection position DP1 (i.e., the position of the workpiece model WM displayed in Fig. 6) may deviate from the workpiece feature WP, as shown in Fig. 6.

[0047] 5 again, in step S12, the processor 30 determines whether or not input data IP1 (first input data) for displacing the position of the workpiece model WM in the image data ID2 has been received. Specifically, while visually checking the image data ID2 shown in FIG. 6 displayed on the display device 38, the operator operates the input device 40 to input the input data IP1 in order to move the workpiece model WM displayed in the image data ID2 to a position that coincides with the corresponding workpiece feature WP on the image.

[0048] If the processor 30 receives the input data IP1 from the input device 40 through the I / O interface 34, the result is YES, and the process proceeds to step S13. On the other hand, if the input data IP1 has not been received from the input device 40, the result is NO, and the process proceeds to step S14. Thus, in this embodiment, the processor 30 functions as an input receiving unit 56 (FIG. 2) that receives the input data IP1 for displacing the position of the work model WM in the image data ID2.

[0049] In step S13, the processor 30 displaces the position of the workpiece model WM displayed in the image data ID2 in accordance with the input data IP1. Specifically, the processor 30 functions as the image generating unit 52 and updates the image data ID2 so as to displace the position of the workpiece model WM in accordance with the input data IP1 within the virtual space defined by the sensor coordinate system C3. In this way, the operator can displace the workpiece model WM in the image data ID2 so as to approach the corresponding workpiece feature WP by operating the input device 40 while visually checking the image data ID2 displayed on the display device 38.

[0050] In step S14, the processor 30 determines whether or not the input data IP2 for acquiring the matching position MP has been received. Specifically, when the position of the work model WM coincides with the position of the work feature WP in the image data ID2 as a result of displacing the work model WM in step S13, the operator operates the input device 40 to input the input data IP2 for acquiring the matching position MP.

[0051] 7 shows a state in which the position of the workpiece model WM coincides with the workpiece feature WP in the image data ID2. When the processor 30 receives input data IP2 from the input device 40 through the I / O interface 34, it determines YES and proceeds to step S15, whereas when the processor 30 has not received input data IP2 from the input device 40, it determines NO and returns to step S12. In this way, the processor 30 loops steps S12 to S14 until it determines YES in step S14.

[0052] In step S15, the processor 30 acquires the position of the work model WM in the image data ID2 when the input data IP2 is received as the matching position MP. As described above, when the processor 30 receives the input data IP2, the work models WM match the corresponding work features WP in the image data ID2, as shown in FIG.

[0053] The processor 30 acquires, as the matching position MP, coordinates (x, y, z, W, P, R) in the sensor coordinate system C3 of the work coordinate system C4 set for each of the work models WM shown in Fig. 7, and stores them in the memory 32. Thus, in this embodiment, the processor 30 functions as a matching position acquisition unit 58 (Fig. 2) that acquires the matching position MP.

[0054] 3, in step S5, the processor 30 executes a process for adjusting the parameter PM. This step S5 will be described with reference to FIG. 8. In step S21, the processor 30 executes a process for adjusting the parameter PM nSet the number "n" that specifies the number of updates to "1".

[0055] In step S22, the processor 30 functions as the position detector 54 and detects the detected position DP n Specifically, the processor 30 acquires the parameter PM stored in the memory 32 at the start of step S22. n Using the detection position DP n If n=1 is set at the start of step S22 (i.e., when step S22 is executed for the first time), processor 30 determines detection position DP1 shown in FIG. 6 by using parameter PM1, similarly to step S11 described above.

[0056] In step S23, the processor 30 calculates the detected position DP n and the matching position MP obtained in step S4. n This data Δ n For example, the detection position DP n The objective function is, for example, a value of the objective function representing the difference between a pair of corresponding detection positions DP n and the matching position MP, may be a sum, a sum of squares, an average, or a function indicating the average square of the differences. n is stored in the memory 32.

[0057] If n=1 is set at the start of step S23 (i.e., when step S23 is executed for the first time), the processor 30 acquires data Δ1 representing the difference between the detection position DP1 and the matching position MP. This data Δ1 represents the difference between the position of the sensor coordinate system C3 of the work model WM shown in Fig. 6 (i.e., the detection position DP1) and the position of the sensor coordinate system C3 of the work model WM shown in Fig. 7 (i.e., the matching position MP).

[0058] In step S24, the processor 30 calculates the data Δn The value of is a predetermined threshold value Δ th Below (Δ n ≦Δ th ) is determined. th is determined by an operator and prestored in memory 32. The processor 30 calculates Δ n ≦Δ th If so, the determination is YES, step S5 is ended, and the process proceeds to step S6 in FIG.

[0059] If the result of the determination in step S24 is YES, the detection position DP n substantially coincides with the matching position MP, and therefore the parameter PM n can be considered to be optimized. On the other hand, the processor 30 n >Δ th If so, the result is NO, and the process proceeds to step S25.

[0060] In step S25, the processor 30 calculates the data Δ n Based on this, the detection position DP n The parameter PM that can reduce the difference between the matching position MP and n The change in α n Specifically, the processor 30 determines the data Δ n 8. Then, the data Δ n The parameter PM can be used to converge the value of n (For example, the amount of change in displacement E, size SZ, or image roughness σ) n The processor 30 determines the amount of change α n Let data Δ n and a predetermined algorithm.

[0061] In step S26, the processor 30 calculates the parameter PM nSpecifically, the processor 30 updates the change amount α n Only the parameter PM n By varying the parameter PM (e.g., the displacement E, the size SZ, or the image roughness σ), n and a new parameter PM n+1 The processor 30 updates the parameters PM n+1 is stored in the memory 32. If n=1 is set at the start of step S26, the processor 30 changes the prepared parameter PM1 by an amount of change α1, and updates it to the parameter PM2.

[0062] In step S27, the processor 30 calculates the parameter PM n In step S28, the processor 30 increments the number "n" that specifies the number of updates of the parameter PM n The number "n" that specifies the number of updates is the maximum value n MAX Exceeded (n>n MAX ) or the change amount α determined in the most recent step S25 n is a predetermined threshold value α th The result is as follows (α n ≦α th ) is checked. MAX and threshold α th is predetermined by an operator and stored in memory 32.

[0063] 8, the processor 30 repeatedly executes the loop of steps S22 to S28 until it determines YES in step S24 or S28. In the above-mentioned step S25, the processor 30 n and the matching position MP (i.e., the data Δ n The amount of change α n Since the data Δ n and the change amount α determined in step S25 nBut it gets smaller.

[0064] Therefore, the change amount α n is the threshold α th If the following occurs, parameter PM n On the other hand, even if the loop of steps S22 to S28 is repeated many times, the change amount α n For some value (>α th ) and the threshold α th In this case, the PM n can be considered to be sufficiently optimized.

[0065] Therefore, in step S28, the processor 30 MAX , or α n ≦α th Determine whether n>n MAX , or α n ≦α th If n≦n is satisfied, the result is YES and step S5 is terminated. MAX , and α n >α th If so, the result is NO, and the process returns to step S22. n+1 The loop of steps S22 to S28 is executed using the above.

[0066] In this manner, the processor 30 repeatedly executes the series of operations from steps S22 to S28 until it determines YES in step S24 or S28, thereby obtaining the data Δ n Based on the parameters PM n Therefore, in this embodiment, the processor 30 updates and adjusts the data Δ n Based on the parameters PM n The parameter adjustment unit 60 (FIG. 2) adjusts the

[0067] The processor 30 functions as a position detector 54 and detects the detected position DP from the image data ID2. nWhen calculating the parameter PM n By using this, the detection position DP n can be obtained as a position that corresponds to (eg, substantially coincides with) the matching position MP.

[0068] 3 again, in step S6, processor 30 determines whether or not an operation end command has been received. For example, an operator operates input device 40 to manually input an operation end command. When processor 30 receives an operation end command from input device 40 through I / O interface 34, processor 30 determines YES, and ends the operation of control device 16. On the other hand, when processor 30 has not received an operation end command, processor 30 determines NO, and returns to step S1.

[0069] For example, after step S5 is completed, the operator changes the arrangement of the workpiece W in the container B shown in FIG. 1 without inputting an operation end command. Next, the operator operates the input device 40 to input the above-mentioned parameter adjustment command. In response to this, the processor 30 determines YES in step S1, executes steps S2 to S5 for the workpiece W after the arrangement in the container B has been changed, and adjusts the parameter PM n In this way, by executing steps S2 to S5 every time the arrangement of the workpiece W in the container B is changed, the parameter PM n can be optimized.

[0070] As described above, in this embodiment, the processor 30 adjusts the parameters PM by functioning as the image generation unit 52, the position detection unit 54, the input reception unit 56, the matching position acquisition unit 58, and the parameter adjustment unit 60. Therefore, the image generation unit 52, the position detection unit 54, the input reception unit 56, the matching position acquisition unit 58, and the parameter adjustment unit 60 constitute the device 50 (FIG. 2) for adjusting the parameters PM.

[0071] According to this device 50, the parameters PM are adjusted using the matching position MP obtained when matching the workpiece model WM to the workpiece features WP in the image data ID2. Therefore, even an operator who does not have specialized knowledge about adjusting the parameters PM can obtain the matching position MP, thereby making it possible to adjust the parameters PM.

[0072] In the present embodiment, the processor 30 repeatedly executes a series of operations from steps S22 to S28 in FIG. n According to this configuration, the parameter PM n can be automatically adjusted, and the parameter PM n can be quickly optimized.

[0073] In this embodiment, the processor 30 functions as an image generating unit 52 to further display the workpiece model WM in the image data ID2 (step S11) and displaces the position of the workpiece model WM displayed in the image data ID2 in accordance with the input data IP1 (step S13). Then, when the workpiece model WM is positioned in the image data ID2 so as to match the workpiece feature WP, the processor 30 functions as a matching position acquiring unit 58 to acquire the matching position MP (step S15).

[0074] According to this configuration, an operator can easily match the workpiece model WM to the workpiece characteristics WP in the image data ID2 by operating the input device 40 while visually checking the image data ID2 displayed on the display device 38, thereby obtaining the matching position MP. Therefore, even an operator who does not have specialized knowledge regarding adjustment of the parameters PM can easily obtain the matching position MP by simply aligning the workpiece model WM to the workpiece characteristics WP on the image.

[0075] As mentioned above, the parameter PM n After adjusting the adjusted parameter PM nUsing this, the robot 12 is caused to perform an operation on the workpiece W (specifically, a workpiece handling operation). The operation on the workpiece W performed by the robot 12 is described below. When the processor 30 receives an operation start command from an operator, a higher-level controller, or a computer program, it operates the robot 12 to position the visual sensor 14 at an imaging position where the workpiece W in the container B can be imaged, and operates the visual sensor 14 to image the workpiece W.

[0076] At this time, image data ID3 captured by the visual sensor 14 indicates at least one workpiece feature WP of the workpiece W. The processor 30 acquires the image data ID3 from the visual sensor 14 through the I / O interface 34, and generates a motion command CM for operating the robot 12 based on the image data ID3.

[0077] More specifically, the processor 30 functions as a position detector 54 to detect the adjusted parameter PM n The processor 30 applies the algorithm AL to the workpiece model WM and matches the workpiece feature WP shown in the image data ID3 according to the algorithm AL. As a result, the processor 30 obtains the detection position DP ID3 are obtained as coordinates in the sensor coordinate system C3.

[0078] Next, the processor 30 calculates the obtained detection position DP ID3 into the robot coordinate system C1 (or the tool coordinate system C2), thereby acquiring position data PD of the imaged workpiece W in the robot coordinate system C1 (or the tool coordinate system C2). Next, the processor 30 generates an operation command CM for controlling the robot 12 based on the acquired position data PD, and controls each servo motor 29 in accordance with the operation command CM, thereby causing the robot 12 to execute a workpiece handling operation of gripping and picking up the workpiece W for which the position data PD has been acquired with the end effector 28.

[0079] Thus, in this embodiment, the processor 30 functions as a command generator 62 that generates the motion command CM. n By using ID3 Since the processor 30 can detect the position data PD (i.e., the position data PD), the processor 30 can cause the robot 12 to perform a workpiece handling operation with high accuracy.

[0080] Next, another example of the above-mentioned step S4 (i.e., the process of acquiring the matching position) will be described with reference to Fig. 9. In the flow shown in Fig. 9, the same step numbers are given to processes similar to those in the flow shown in Fig. 5, and duplicated explanations will be omitted. In step S4 shown in Fig. 9, the processor 30 executes steps S31 and S32 after step S11.

[0081] Specifically, in step S31, the processor 30 determines whether or not it has received input data IP3 (second input data) for deleting the work model WM from the image data ID2, or input data IP4 (second input data) for adding a further work model WM to the image data ID2.

[0082] Here, in the above-mentioned step S11, the processor 30 may erroneously display the workpiece model WM in an inappropriate position. Fig. 10 shows an example of image data ID2 in which the workpiece model WM is displayed in an inappropriate position. In the image data ID2 shown in Fig. 10, a feature F of a member other than the workpiece W is captured.

[0083] When the processor 30 determines the detection position DP1 using the parameter PM1, the processor 30 may mistakenly recognize the feature F as a work feature WP of the workpiece W and determine the detection position DP1 corresponding to the feature F. In such a case, the operator needs to delete the workpiece model WM displayed at the position corresponding to the feature F from the image data ID2.

[0084] On the other hand, in the above-mentioned step S11, there are cases where the processor 30 is unable to recognize the workpiece feature WP shown in the image data ID2 and fails to display the workpiece model WM. Such an example is shown in Fig. 11. In the image data ID2 shown in Fig. 11, of the total of three workpiece features WP, the workpiece model WM corresponding to the upper right workpiece feature WP is not displayed. In such a case, the operator needs to add the workpiece model WM to the image data ID2.

[0085] Therefore, in this embodiment, the processor 30 is configured to receive input data IP3 for deleting the workpiece model WM from the image data ID2, and input data IP4 for adding a further workpiece model WM to the image data ID2. Specifically, when the image data ID2 shown in Fig. 10 is displayed in step S11, the operator operates the input device 40 while visually checking the image data ID2 to input input data IP3 that specifies the workpiece model WM to be deleted.

[0086] Furthermore, when the image data ID2 shown in FIG. 11 is displayed in step S11, the operator operates the input device 40 while visually viewing the image data ID2 to input input data IP4 that specifies the position (e.g., coordinates) in the image data ID2 (sensor coordinate system C3) of the work model WM to be added.

[0087] In step S31, if the processor 30 receives the input data IP3 or IP4 from the input device 40 through the I / O interface 34, the result is YES, and the process proceeds to step S32. On the other hand, if the processor 30 has not received the input data IP3 or IP4 from the input device 40, the result is NO, and the process proceeds to step S12.

[0088] In step S32, the processor 30 functions as the image generation unit 52, and deletes the displayed work model WM from the image data ID2, or additionally displays a further work model WM in the image data ID2, according to the received input data IP3 or IP4. For example, when the input data IP3 is received, the processor 30 deletes the work model WM displayed at the position corresponding to the feature F from the image data ID2 shown in Fig. 10. As a result, the image data ID2 is updated as shown in Fig. 12.

[0089] On the other hand, when the input data IP4 is received, the processor 30 additionally displays a work model WM at the position specified by the input data IP4 in the image data ID2 shown in Fig. 11. As a result, a total of three work models WM corresponding to all of the work features WP are displayed in the image data ID2, as shown in Fig. 6.

[0090] After step S32, the processor 30 executes steps S12 to S15 in the same manner as in the flow of Fig. 5. In the flow shown in Fig. 9, if the processor 30 determines NO in step S14, the process returns to step S31. As described above, according to this embodiment, the operator can delete or add a work model WM as necessary in the image data ID2 displayed in step S11.

[0091] 9, the processor 30 may display the work model WM at a randomly determined position in the image data ID2. An example in which the processor 30 randomly displays the work model WM in the image data ID2 is shown in Fig. 13. In this case, the processor 30 may randomly determine the number of work models WM to be placed in the image data ID2, or the operator may determine the number in advance.

[0092] Alternatively, in step S11 in Fig. 9, the processor 30 may display the workpiece model WM in the image data ID2 at a position determined according to a predetermined rule. For example, this rule may be defined as a rule that the workpiece model WM is arranged in a grid pattern at equal intervals in the image data ID2. Fig. 14 shows an example in which the processor 30 displays the workpiece model WM in the image data ID2 according to the rule that the workpiece model WM is arranged in a grid pattern at equal intervals.

[0093] After the processor 30 arranges the work models WM randomly or according to a predetermined rule in step S11, the operator can delete or add the work models WM displayed in the image data ID2 as necessary by inputting input data IP3 or IP4 into the input device 40 in step S31.

[0094] Next, another example of the above-mentioned step S4 (process of acquiring a matching position) will be described with reference to Fig. 15. In step S4 shown in Fig. 15, the processor 30 executes steps S41 and S42 after step S11. In step S41, the processor 30 determines whether or not there is a work model WM in the image data ID2 that satisfies the condition G1 to be deleted.

[0095] Specifically, for each of the workpiece models WM depicted in the image data ID2, the processor 30 calculates the number N of points (or pixels depicting the workpiece features WP) of the three-dimensional point cloud constituting the workpiece features WP that are present within the occupation area of ​​the workpiece model WM. Then, in this step S41, the processor 30 checks whether the calculated number N for each of the workpiece models WM is equal to or smaller than a predetermined threshold value N th (N≦N th ) and determine whether N≦N th If there is a work model WM that is determined to be, the work model WM is specified as a deletion target, and the result is determined as YES. th It is stipulated that the following is true:

[0096] For example, suppose that the processor 30 generates the image data ID2 shown in Fig. 14 in step S11. In this case, among the work models WM appearing in the image data ID2, the second and fourth work models WM from the left in the top row and the first, third and fourth work models WM from the left in the bottom row have fewer points (pixels) of the work feature WP existing within their exclusive areas. Therefore, in this case, the processor 30 identifies these five work models WM in total as deletion targets in step S11 and determines YES.

[0097] In step S42, the processor 30 functions as the image generating unit 52, and automatically deletes the work model WM identified in step S41 as the workpiece model to be deleted from the image data ID2. In the example shown in FIG. 14, the processor 30 automatically deletes the total of five workpiece models WM identified as the workpiece models to be deleted from the image data ID2. FIG. 16 shows an example of the image data ID2 from which these five workpiece models WM have been deleted. Thus, in this embodiment, the processor 30 automatically deletes the displayed workpiece models WM from the image data ID2 in accordance with the predetermined condition G1.

[0098] Alternatively, the processor 30 may determine in step S41 whether or not the condition G2 for adding the workpiece model WM is satisfied in the image data ID2. For example, assume that the processor 30 generates the image data ID2 shown in Fig. 11 in step S11. The processor 30 determines whether or not each workpiece feature WP has a point (or pixel) included in the occupied area of ​​the workpiece model WM.

[0099] Then, when there is a work feature WP that does not have a point (pixel) included in the exclusive area of ​​the work model WM, the processor 30 identifies the work feature WP as a model addition target and determines YES. That is, in this embodiment, the condition G2 is defined as the existence of a work feature WP that does not have a point (pixel) included in the exclusive area of ​​the work model WM. For example, in the case of the example shown in FIG. 11, the processor 30 identifies the work feature WP shown in the upper right of the image data ID2 as a model addition target in this step S41 and determines YES.

[0100] Then, in step S42, the processor 30 functions as the image generating unit 52, and automatically adds a workpiece model WM to the image data ID2 at a position corresponding to the workpiece feature WP identified as the model addition target in step S41. As a result, the workpiece model WM is added as shown in, for example, FIG.

[0101] Thus, in this embodiment, the processor 30 additionally displays the workpiece model WM in the image data ID2 in accordance with the predetermined condition G2. According to the flow shown in Fig. 15, the processor 30 can automatically delete or add the workpiece model WM in accordance with the condition G1 or G2, thereby reducing the workload of the operator.

[0102] Next, other functions of the robot system 10 will be described with reference to Fig. 17 and Fig. 18. In this embodiment, the processor 30 first executes an image acquisition process shown in Fig. 17. In step S51, the processor 30 acquires image data ID1 captured by the visual sensor 14. _i Set the identifying number "i" to "1".

[0103] In step S52, the processor 30 determines whether or not an imaging start command has been received. For example, the operator operates the input device 40 to input an imaging start command. If the processor 30 receives an imaging start command from the input device 40 through the I / O interface 34, the result is YES, and the process proceeds to step S53. On the other hand, if the processor 30 has not received an imaging start command, the result is NO, and the process proceeds to step S56.

[0104] In step S53, the processor 30 captures an image of the workpiece W by the visual sensor 14, similar to step S2 described above. As a result, the visual sensor 14 captures the i-th image data ID1 _i In step S54, the processor 30 captures the image data ID1 _i is stored together with the identification number "i" in the memory 32. In step S55, the processor 30 increments the identification number "i" by "1" (i=i+1).

[0105] In step S56, the processor 30 determines whether or not an imaging end command has been received. For example, the operator operates the input device 40 to input an imaging end command. If the imaging end command has been received, the processor 30 determines YES, and ends the flow shown in Fig. 17. On the other hand, if the imaging end command has not been received, the processor 30 determines NO, and returns to step S52.

[0106] For example, after step S55 is completed, the operator changes the arrangement of the workpiece W in the container B shown in FIG. 1 without inputting an image capture end command. Next, the operator operates the input device 40 to input an image capture start command. Then, the processor 30 determines YES in step S52, executes steps S53 to S55 for the workpiece W after the position in the container B has been changed, and generates the (i+1)th image data ID1 _i+1 Get the.

[0107] After the flow shown in Fig. 17 ends, the processor 30 executes the flow shown in Fig. 18. In the flow shown in Fig. 18, the same processes as those in the flows shown in Fig. 3 and Fig. 17 are given the same step numbers, and duplicated explanations are omitted. If the processor 30 judges YES in step S1, it proceeds to step S51, whereas if the processor 30 judges NO, it proceeds to step S6.

[0108] If the result of the determination in step S6 is YES, the processor 30 ends the flow shown in FIG. 18, whereas if the result of the determination is NO, the processor 30 returns to step S1. _i Set the identification number “i” to “1”.

[0109] In step S62, the processor 30 selects image data ID2 in which the workpiece feature WP is displayed. _i Specifically, the processor 30 generates the i-th image data ID1 identified by the identification number “i”. _i The processor 30 then reads out the i-th image data ID1 _i Based on this, the i-th image data ID1 _i The workpiece feature WP shown in the image is displayed as a GUI that the operator can visually recognize. For example, the i-th image data ID2 _i Generate.

[0110] After step S62, the processor 30 receives the i-th image data ID2 _i The above steps S4 and S5 are executed in sequence using the above, and the i-th image data ID2 _i The parameter PM is optimized with respect to n Next, the processor 30 executes step S55 and increments the identification number "i" by "1" (i=i+1).

[0111] In step S64, the processor 30 checks whether the identification number “i” is equal to or smaller than the maximum value i MAX Exceeded (i>i MAX ) is checked.MAX Image data ID1 acquired by the processor 30 in the flow of FIG. _i In step S64, the processor 30 determines whether i>i MAX If i≦i, the flow shown in FIG. 18 is terminated. MAX If so, the processor 30 judges the result as NO and returns to step S62. In this manner, the processor 30 repeatedly executes the loop of steps S62, S4, S5, S55 and S64 until the processor 30 judges the result as YES in step S64, and stores all the image data ID2 _i (i=1,2,3,...i MAX ) with respect to the parameter PM n Adjust.

[0112] As described above, in this embodiment, the image data ID1 of the workpiece W placed at various positions is obtained by the flow shown in FIG. _i Then, in the flow shown in FIG. 18, the stored image data ID1 _i The parameters PM are adjusted using the above formula. According to this configuration, the parameters PM can be optimized for the workpieces W placed in various positions.

[0113] In the flow shown in FIG. 17, the processor 30 omits step S52, and executes the above-mentioned step S64 instead of step S56, and the identification number "i" is set to the maximum value i MAX Exceeded (i>i MAX ) or not. The threshold value i MAX is predetermined by an operator as an integer greater than or equal to 2.

[0114] If the processor 30 judges YES in step S64, it ends the flow of FIG. 17, and if the processor 30 judges NO, it returns to step S53. Here, in such a modified example of the flow of FIG. 17, the processor 30 may change the position (specifically, the position and the posture) of the visual sensor 14 each time step S53 is executed, and image the workpiece W from a different position and line of sight direction A2. According to this modified example, image data ID1 of the workpiece W in various positions can be stored without the operator having to manually change the position of the workpiece W. _i can be automatically acquired and stored.

[0115] The processor 30 may execute the flows shown in Figures 3, 17, and 18 in accordance with a computer program stored in advance in the memory 32. This computer program includes instructions for causing the processor 30 to execute the flows shown in Figures 3, 17, and 18, and is stored in the memory 32 in advance.

[0116] In the above embodiment, a case has been described in which the processor 30 acquires image data ID1 of the actual workpiece W using the actual robot 12 and visual sensor 14. However, the processor 30 may acquire image data ID1 by virtually capturing an image of the workpiece model WM using a visual sensor model 14M that is a model of the visual sensor 14.

[0117] In this case, the processor 30 may place, in a virtual space, a robot model 12M that is a model of the robot 12 and a visual sensor model 14M fixed to an end effector model 28M of the robot model 12M, and simulate the operation of the robot model 12M and the visual sensor model 14M in the virtual space to execute the flows shown in Figures 3, 17, and 18 (i.e., simulation). According to this configuration, the parameters PM can be adjusted by a so-called offline operation without using the actual robot 12 and visual sensor 14.

[0118] It is also possible to omit the input receiving unit 56 from the above-mentioned device 50. In this case, the processor 30 may omit steps S12 to S14 in Fig. 5 and automatically acquire the matching position MP from the image data ID2 generated in step S11. For example, a learning model LM indicating the correlation between the work feature WP shown in the image data ID and the matching position MP may be stored in advance in the memory 32.

[0119] This learning model LM can be constructed, for example, by repeatedly providing a learning data set of image data ID containing at least one work feature WP and data on a matching position MP in the image data ID to a machine learning device (for example, supervised learning). The processor 30 inputs the image data ID2 generated in step S11 to the learning model LM. Then, the learning model LM outputs a matching position MP corresponding to the work feature WP appearing in the input image data ID2. In this way, the processor 30 can automatically obtain the matching position MP from the image data ID2. The processor 30 may be configured to execute the functions of this machine learning device.

[0120] In the above embodiment, the workpiece feature WP is configured from a three-dimensional point cloud in the image data ID2. However, this is not limiting, and the processor 30 may generate the image data ID2 in step S3 as a distance image in which the color or tone (shade) of each pixel representing the workpiece feature WP changes according to the above-mentioned distance d.

[0121] Also, in step S3, the processor 30 may generate the image data ID1 acquired from the visual sensor 14 as image data to be displayed on the display device 38, without newly generating the image data ID2, and display the image data on the display device 38. Then, the processor 30 may execute steps S3 and S4 using the image data ID1.

[0122] Also, the image generating unit 52 may be omitted from the above-described device 50, and the function of the image generating unit 52 may be provided in an external device (for example, the visual sensor 14 or a PC). For example, the processor 30 may adjust the parameters PM by using the image data ID1 captured by the visual sensor 14 as the original data format without making any changes. In this case, the visual sensor 14 will assume the function of the image generating unit 52.

[0123] The visual sensor 14 is not limited to a three-dimensional visual sensor, and may be a two-dimensional camera. In this case, the processor 30 may generate two-dimensional image data ID2 based on the two-dimensional image data ID1, and execute steps S3 and S4. In this case, the sensor coordinate system C3 is a two-dimensional coordinate system (x, y).

[0124] In the above-described embodiment, the device 50 is implemented in the control device 16. However, the present invention is not limited to this, and the device 50 may be implemented in a computer (for example, a desktop PC, a mobile electronic device such as a tablet terminal device or a smartphone, or a teaching device for teaching the robot 12) other than the control device 16. In this case, the other computer may have a processor that functions as the device 50 and be communicatively connected to the I / O interface 34 of the control device 16.

[0125] The end effector 28 is not limited to a robot hand, but may be any device that performs an operation on a workpiece (such as a laser processing head, a welding torch, or a paint applicator). Although the present disclosure has been described above through the embodiments, the invention according to the claims is not limited to the above-described embodiments. [Explanation of symbols]

[0126] 10 Robot System 12. Robot 14 Visual Sensor 16 Control device 30 processors 50 equipment 52 Image Generation Unit 54 Position detection unit 56 Input reception section 58 Matching position acquisition unit 60 Parameter Adjustment Section 62 Command generation section

Claims

1. a position detection unit that determines a position of the workpiece in the image data as a detection position by using parameters for matching a workpiece model obtained by modeling the workpiece in image data showing workpiece features of the workpiece captured by the visual sensor with the workpiece features; an image generating unit that generates the image data that displays the workpiece model together with the workpiece features; an input receiving unit that receives first input data for displacing a position of the work model in the image data; a matching position acquisition unit that acquires, as a matching position, a position of the work model in the image data when the image generation unit displaces a position of the work model displayed in the image data in accordance with the first input data and arranges the work model to match the work feature; An apparatus comprising: a parameter adjustment unit that determines whether or not adjustment is necessary based on a comparison of data representing a difference between the detection position and the matching position with a predetermined threshold, and adjusts the parameter if it is determined that adjustment is necessary.

2. The image generating unit includes: Displaying the workpiece model at the detection position acquired by the position detection unit; Displaying the workpiece model at a randomly determined location in the image data; or The apparatus of claim 1 , wherein the workpiece model is displayed in the image data at a position determined according to a predetermined rule.

3. The input receiving unit further receives second input data for deleting the workpiece model from the image data or adding a second workpiece model to the image data; The device according to claim 1 or 2, wherein the image generation unit deletes the displayed work model from the image data or additionally displays the second work model in the image data in accordance with the second input data.

4. The device according to claim 1 or 2, wherein the image generation unit deletes the displayed work model from the image data or additionally displays a second work model in the image data according to a predetermined condition.

5. The apparatus according to any one of claims 1 to 4, wherein the workpiece features are acquired by virtually imaging the workpiece model with a visual sensor model that models the visual sensor.

6. A visual sensor that captures an image of a workpiece; A robot that performs a task on the workpiece; a command generating unit that generates an operation command for operating the robot based on image data captured by the visual sensor; and a device according to any one of claims 1 to 5, The position detection unit acquires, as a detection position, a position of the workpiece in the image data captured by the visual sensor, using the parameters adjusted by the parameter adjustment unit; The command generating unit is acquiring position data of the workpiece in a control coordinate system for controlling the robot based on the detected position acquired by the position detection unit using the adjusted parameters; The robot system generates the motion command based on the position data.

7. The processor: A position of the workpiece in the image data is determined as a detection position by using a parameter for matching a workpiece model obtained by modeling the workpiece in image data showing workpiece features of the workpiece captured by the visual sensor with the workpiece features; generating the image data representative of the workpiece model along with the workpiece features; accepting first input data for displacing a position of the workpiece model in the image data; acquiring a position of the work model in the image data as a matching position when the position of the work model displayed in the image data is displaced in accordance with the first input data so that the work model matches the work feature; A method for determining whether or not adjustment is necessary based on a comparison between data representing a difference between the detected position and the matching position and a predetermined threshold, and adjusting the parameters if it is determined that adjustment is necessary.

8. A computer program product causing the processor to carry out the method of claim 7.

Images