Position and orientation measurement system

Abstract

This invention provides a system that can accurately measure the three-dimensional position and posture of a measurement object with simple teaching and a short measurement time. A position and posture measurement system includes: a calibrated vision sensor; a position and posture movement unit capable of moving the measurement object or the vision sensor to measure its three-dimensional position and posture; a calibration data memory unit; an image processing unit for processing captured images of the measurement object; a position and posture measurement unit for measuring the three-dimensional position and posture of the measurement object using the image processing results; a camera position correction unit for performing at least one correction process, wherein the correction process involves using the measured three-dimensional position and posture, moving the vision sensor or the measurement object by the position and posture movement unit, and then sequentially performing image processing and three-dimensional position and posture measurement; and a camera position correction termination unit that, if it is determined that the correction process should not be terminated, performs the correction process again; and, if it is determined that the correction process should be terminated, uses any three-dimensional position and posture measured at the end or in the middle as the three-dimensional position and posture for robot control.

Description

Technical Field

[0001] This disclosure relates to a position and attitude measurement system. Prior Technology

[0002] Background Technology

[0003] Previously, workpiece processing was performed using robots mounted on conveying devices such as trolleys. After moving to the vicinity of the workpiece using the conveying device, the robot performed various tasks such as loading / unloading the workpiece, changing tools, and processing the workpiece. However, when the robot's stopping position or posture changes, its position or posture relative to the workspace also changes. Therefore, the robot cannot simply perform the same actions repeatedly to ensure proper operation. Thus, it is necessary to measure the deviation of the robot's position or posture relative to the workspace and correct the robot's movements accordingly.

[0004] For example, some have proposed the following technology: using a vision sensor installed on the robot's fingers, etc., to detect the three-dimensional position of a plurality of reference points set in the work space, and calculating the deviation from the robot's reference position or reference posture relative to the work space as a correction amount, and applying correction to the robot's movement (for example, see Patent Documents 1 and 2). Prior technology documents Patent documents

[0005] [Patent Document 1] Japanese Patent Application Publication No. 11-156764 [Patent Document 2] Japanese Patent Application Publication No. 2019-093481 Summary of the Invention

[0006] Invention Summary The problem the invention aims to solve

[0007] However, the technologies in Patent Documents 1 and 2 have the following drawbacks: the deviation in the three-dimensional position and posture of the object to be measured obtained from the captured images is relatively large; or, in order to capture multiple objects to be measured positioned at multiple locations, the number of teaching points at the camera position increases, and the time required for teaching and measurement also increases. Therefore, the prior art cannot be used for applications requiring precise actions such as installing workpieces with short teaching and measurement times.

[0008] The purpose of this disclosure is to provide a position and pose measurement system that can accurately measure the three-dimensional position and pose of the object being measured with simpler instruction and shorter measurement time than before. The means to solve the problem

[0009] This disclosure discloses a position and pose measurement system for measuring the three-dimensional position and pose of a target object used in robot control. The system includes: a calibrated vision sensor; a position and pose movement unit capable of moving the target object or the vision sensor's three-dimensional position and pose; a calibration data memory unit that pre-memorizes calibration data obtained during the calibration of the vision sensor; an image processing unit that processes images captured by the vision sensor of the target object; a position and pose measurement unit that uses the image processing results to measure the three-dimensional position and pose of the target object; and a camera position correction unit that performs... The correction process is performed once less. The aforementioned correction process uses the three-dimensional position and posture of the object being measured as measured by the aforementioned position and posture measurement unit. After the aforementioned vision sensor or the object being measured is moved by the aforementioned position and posture movement unit, the image processing unit and the measurement unit of the aforementioned position and posture measurement unit are performed sequentially. The camera position correction termination unit determines whether to end the aforementioned correction process of the aforementioned camera position correction unit. If it is determined that it is not over, the aforementioned correction process is executed again. If it is determined that it is about to end, the three-dimensional position and posture of the object being measured is adopted by the aforementioned position and posture measurement unit at the end or in the middle as the three-dimensional position and posture of the object being measured for use in the control of the aforementioned robot. Invention Effects

[0010] According to one aspect of this disclosure, a position and pose measurement system can be provided that can accurately measure the three-dimensional position and pose of the object being measured with simpler instruction and shorter measurement time than before. Simple Explanation of the Diagram

[0011] Figure 1 is a diagram showing the configuration of the robot system in the first embodiment. Figure 2 is a block diagram showing the configuration of the position and posture measurement system in the first embodiment. Figure 3 is a flowchart showing the processing sequence of the position and attitude measurement system in the first embodiment. Figure 4 shows the changes in the input image as the camera position is moved. Figure 5 is a diagram showing the position of the object being measured before the camera position was moved and the position of the object being measured when the reference position was set. Figure 6 is a diagram showing the position of the object being measured after the camera position has been moved, and the position of the object being measured when the reference position is set. Figure 7 is a flowchart showing the processing sequence of the position and attitude measurement system of the modified example of the first embodiment. Figure 8 is a block diagram showing the configuration of the position and posture measurement system in the second embodiment. Figure 9 shows the changes in the input image when the camera position is moved and the brightness is adjusted. Figure 10 is a block diagram showing the configuration of the position and posture measurement system in the third embodiment. Figure 11 is a diagram showing the display screen of the interface device when the brightness difference of the measured object before and after the camera position is determined to be greater than the second threshold. Figure 12 is a block diagram showing the configuration of the position and posture measurement system in the fourth embodiment. Figure 13 is a diagram showing the interface device display screen when the undetectable range of the input image after the camera position is determined to have moved is greater than the undetectable threshold. Implementation

[0012] Forms used to implement inventions

[0013] The embodiments disclosed herein will now be described with reference to the diagrams. Furthermore, in the description of each embodiment, the same symbols are used for components common to the first embodiment, and their descriptions are omitted as appropriate.

[0014] [First Implementation Form] The position and pose measurement system of the first embodiment disclosed herein is a system that uses a calibrated vision sensor to capture images of the object being measured and the calibration data to measure the three-dimensional position and pose (X, Y, Z, W, P, R) of the object being measured for robot motion control. More specifically, the position and pose measurement system of the first embodiment is a system that, by performing correction to make the relative position between the current camera position and the object being measured close to the relative position between the camera position and the object being measured when the robot's reference position is set, measurement deviations can be suppressed compared to the case without correction, and the three-dimensional position and pose of the object being measured can be measured more accurately than before.

[0015] Here, the robot's reference position refers to the position used as a reference for correction. Correction is achieved by calculating the difference between the position of the measured object when the reference position is set (reference position) and the position of the measured object during correction. This difference is applied to the action to be corrected. The difference is represented by a homogeneous transformation matrix, which is applied to the three-dimensional position (vector) to obtain the corrected position. Alternatively, the three-dimensional position of the measured object when the reference position is set can be pre-memorized, and this position can be used as a reference coordinate system to teach the robot's actions. The coordinate system can then be rewritten using the object's position during measurement to perform the same correction.

[0016] Figure 1 is a diagram showing the configuration of the robot system 100 according to the first embodiment. As shown in Figure 1, the robot system 100 is configured including the position and posture measurement system 1 of the first embodiment. The robot system 100 includes: a robot 2, a vision sensor 4, a vision sensor control device 20, a robot control device 10, and a measurement object 7. For example, based on an image of the workpiece captured by the vision sensor 4, the robot system 100 identifies the three-dimensional position and posture of the workpiece and performs predetermined operations such as workpiece handling or processing.

[0017] A tool 6 with fingers is mounted on the front end of the robot arm 3 of robot 2 via a flange 5. Robot 2 performs predetermined tasks such as workpiece handling or processing under the control of robot control device 10. Furthermore, a vision sensor 4 is mounted on the front end of the robot arm 3 of robot 2 via a flange 5. In this way, the vision sensor 4 can move to a three-dimensional position and posture.

[0018] The vision sensor 4 is controlled by the vision sensor control device 20 to capture images of the measurement object 7, such as the workpiece, calibration fixture, or markings. The vision sensor 4 uses a calibrated vision sensor. The vision sensor 4 can be a general 2D camera or a 3D camera such as a stereo camera.

[0019] The vision sensor control device 20 controls the vision sensor 4 and performs image processing on the images captured by the vision sensor 4. Furthermore, the vision sensor control device 20 detects the three-dimensional position and pose of the object to be measured 7 from the images captured by the vision sensor 4.

[0020] The robot control device 10 executes the motion program of the robot 2 and controls the actions of the robot 2. At this time, the robot control device 10 corrects the actions of the robot 2 based on the three-dimensional position and posture of the measured object 7 detected by the vision sensor control device 20, so that the robot 2 can perform the predetermined task.

[0021] Furthermore, when the vision sensor 4 is taking pictures, the robot control device 10 controls the position and posture of the robot 2, thereby controlling the position and posture of the vision sensor 4. In this way, the robot control device 10 controls the relative position of the object being measured 7 and the vision sensor 4 by setting the position and posture of the object being measured 7 to be fixed and controlling the position and posture of the vision sensor 4.

[0022] Figure 2 is a block diagram showing the configuration of the position and posture measurement system 1 according to the first embodiment. As shown in Figure 2, the position and posture measurement system 1 according to the first embodiment includes: a vision sensor control device 20 and a robot control device 10. The vision sensor control device 20 includes: an image processing unit 21, a correction data memory unit 22, and an image acquisition unit 23. The robot control device 10 includes: a position and posture measurement unit 11, a camera position correction unit 12, a camera position correction termination unit 13, and a position and posture movement unit 14.

[0023] The image acquisition unit 23 acquires the captured image using the vision sensor 4 as the input image. The image acquisition unit 23 then transmits the acquired input image to the image processing unit 21.

[0024] The image processing unit 21 performs image processing on the input image transmitted from the image acquisition unit 23. More specifically, the image processing unit 21 detects the object to be measured 7 from the input image, for example, using a pre-memorized model pattern, when the detection score based on the consistency with the model pattern is above a predetermined threshold. Furthermore, the image processing unit 21 calculates the three-dimensional position and pose of the object to be measured 7 based on the vision sensor 4 from the input image and the correction data stored in the correction data memory unit 22 (described later).

[0025] Furthermore, regarding the principle of determining the three-dimensional position and pose of the object 7 based on the vision sensor 4 from the position information of the object 7, such as the calibration fixture, in the input image captured using the calibrated vision sensor 4, well-known sources include, for example, "Computer Vision, Technology Review, and Future Prospects" by Takashi Matsuyama. That is, the geometric transformation characteristics within the vision sensor 4 and the geometric relationship between the three-dimensional space where the object exists and the two-dimensional image plane are determined, thereby obtaining a plurality of pairs of information. These pairs of information are the positions on the two-dimensional image and in three-dimensional space of the features such as the plurality of dots displayed by the calibration fixture, etc.; thereby, the relative position of the vision sensor 4 and the object 7 can be determined without ambiguity. Therefore, the three-dimensional position and pose of the object 7 based on the vision sensor 4 can be determined from the position information of the object 7 in the input image captured using the calibrated vision sensor 4.

[0026] The image processing unit 21 and the robot control unit 10 in the vision sensor control device 20 are composed of computing processors such as DSP (Digital Signal Processor) and FPGA (Field-Programmable Gate Array). The various functions of the image processing unit 21 and the robot control unit 10 in the vision sensor control device 20 are implemented, for example, by executing predetermined software (programs, applications). The various functions of the image processing unit 21 and the robot control unit 10 in the vision sensor control device 20 can also be implemented through the cooperation of hardware and software, or solely through hardware (electronic circuitry).

[0027] The calibration data memory unit 22 stores the calibration data obtained during the calibration of the vision sensor 4. The calibration data memory unit 22 in the vision sensor control device 20 is constructed, for example, a rewritable memory such as EEPROM (Electrically Erasable Programmable Read-Only Memory).

[0028] The calibration data of vision sensor 4 contains both internal and external parameters. Examples of internal parameters of vision sensor 4 include lens distortion error and focal distance. Examples of external parameters of vision sensor 4 include the three-dimensional position and pose of vision sensor 4 based on the reference position of robot 2 when the robot 2 is set to its reference position, or the three-dimensional position and pose of vision sensor 4 based on the position of flange 5.

[0029] The position and pose measurement unit 11 uses the image processing results of the image processing unit 21 to measure the three-dimensional position and pose of the object to be measured 7. Specifically, the position and pose measurement unit 11 calculates the three-dimensional position and pose of the visual sensor 4 (i.e., the current position and pose of the visual sensor 4) based on the reference position of the robot 2 at the time of shooting, and the three-dimensional position and pose of the object to be measured 7 based on the visual sensor 4 obtained by the image processing unit 21, to determine the three-dimensional position and pose of the object to be measured based on the reference position of the robot 2 at the time of shooting.

[0030] The three-dimensional position and pose of the vision sensor 4, which is based on the reference position of robot 2 during shooting, are obtained as follows. Since the vision sensor 4 in this embodiment is a handheld camera, its position changes accordingly when robot 2 moves. Therefore, after installing the vision sensor 4, the three-dimensional position and pose of the vision sensor 4, which is based on the flange 5 (which does not change with the movement of robot 2), is obtained from the calibration data stored in the calibration data memory unit 22. Furthermore, the three-dimensional position and pose of the flange 5, which is based on the reference position of robot 2 during shooting, is obtained from the robot control device 10, which generates movement commands to robot 2. Then, based on the obtained three-dimensional position and pose of the vision sensor 4 with the flange 5 as the reference, and the obtained three-dimensional position and pose of the flange 5 with the reference position of robot 2 during shooting, the three-dimensional position and pose of the vision sensor 4 with the reference position of robot 2 during shooting are calculated and obtained.

[0031] The camera position correction unit 12 uses the three-dimensional position and posture of the object 7 measured by the position and posture measurement unit 11, and moves the vision sensor 4 using the position and posture movement unit 14 (described later). Then, it sequentially executes the image processing of the image processing unit 21 and the measurement of the position and posture measurement unit 11. The camera position correction unit 12 performs such correction processing at least once. The aforementioned correction processing consists of the movement of the vision sensor 4, image processing, and three-dimensional position and posture measurement.

[0032] Furthermore, the camera position correction unit 12 preferably performs the above correction process after the robot 2 moves to the robot position. The aforementioned robot position is a position that allows the vision sensor 4 to approach the position of the vision sensor 4 as seen from the measurement object 7 when the reference position is set.

[0033] By means of the correction processing of the camera position correction unit 12, the relative position between the current camera position and the measured object 7 will be close to the relative position between the camera position and the measured object 7 when the robot 2's reference position is set. This suppresses the deviation of the position and posture measurement unit 11 in measuring the three-dimensional position and posture of the measured object 7. That is, previously, if the deviation of the three-dimensional position and posture of the measured object obtained from the captured image was large, or if the distance between the measured object and the position to be corrected was large, there was a problem of increased correction error; however, this embodiment solves this problem. Therefore, this embodiment can also be applied to applications requiring precise movements such as mounting workpieces.

[0034] The camera position correction termination unit 13 determines whether to end the correction processing of the camera position correction unit 12. In this embodiment, the camera position correction termination unit 13 determines whether to end the correction processing based on whether the correction processing of the camera position correction unit 12 has been performed a predetermined number of times. The predetermined number of times is, for example, set to three times.

[0035] Specifically, if the camera position correction termination unit 13 determines that the correction process of the camera position correction unit 12 has not been executed a predetermined number of times and does not end the correction process, it will execute the correction process of the camera position correction unit 12 again.

[0036] Furthermore, when the camera position correction termination unit 13 determines that the correction processing of the camera position correction unit 12 has been executed a predetermined number of times and is about to end the correction processing, it adopts the three-dimensional position and posture of the measurement object 7 measured by the position and posture measurement unit 11 at the end or in the middle as the three-dimensional position and posture of the measurement object 7 to be used for the control of the robot 2.

[0037] During the correction process, the three-dimensional position and posture of the object being measured, which is based on the reference position of the robot 2, is finally measured by the position and posture measurement unit 11. This is the correct three-dimensional position and posture that suppresses measurement deviations. Therefore, by using the three-dimensional position and posture of the object being measured, which is based on the reference position of the robot 2 and adopted by the camera position correction termination unit 13, the robot 2's movements can be controlled correctly.

[0038] Furthermore, when the movement of the vision sensor 4 is very small during the correction process (e.g., less than a predetermined movement threshold), the movement accuracy of the robot 2 itself has a greater impact on the correction error. Therefore, in this case, using any of the three-dimensional position and pose of the measured object 7 measured during the correction process, such as the three-dimensional position and pose measured before the last one, can sometimes better suppress the correction error.

[0039] For example, in addition to using the three-dimensional position and posture of the object to be measured 7, which is based on the reference position of the robot 2, adopted by the camera position correction end unit 13, to control the movement of the robot 2 relative to the object on which the object to be measured 7 is mounted, a coordinate system for controlling the robot 2 can also be set. In this case, the amount of movement of the coordinate system obtained by moving and rotating the coordinate system itself by overlapping the set coordinate system with the reference coordinate system located at the reference position can be used as the deviation amount, i.e., the correction amount, to correct the movement of the robot 2. Furthermore, it can also be used for a coordinate system used to set the teaching position of the robot 2. In the example shown in Figure 1, the object to be measured 7 is placed on a platform T, and the coordinate system used for controlling the robot 2 is set on the object to be measured 7 on the platform T.

[0040] The position and pose movement unit 14 moves the three-dimensional position and pose of the vision sensor 4. Specifically, the position and pose movement unit 14 moves the three-dimensional position and pose of the vision sensor 4 by controlling the movement of the robot 2. The camera position correction unit 12 described above performs correction processing by moving the vision sensor 4 using the position and pose movement unit 14.

[0041] The processing of the position and attitude measurement system 1 of the first embodiment, which has the above-described configuration, will be described in detail with reference to FIG3. FIG3 is a flowchart showing the processing sequence of the position and attitude measurement system 1 of the first embodiment.

[0042] First, an input image is acquired in step S11. Specifically, the object to be measured, 7, is photographed by the vision sensor 4, and the input image is acquired by the image acquisition unit 23. Then, the process proceeds to step S12.

[0043] Next, in step S12, the three-dimensional position and pose of the object being measured 7 are measured. Specifically, in the position and pose measurement unit 11, the three-dimensional position and pose of the object being measured 7 are measured using the image processing results of the input image from the image processing unit 21. More specifically, in the position and pose measurement unit 11, the three-dimensional position and pose of the vision sensor 4, which is based on the reference position of the robot 2 at the time of shooting, are calculated from the three-dimensional position and pose of the object being measured, which is based on the reference position of the vision sensor 4 at the time of shooting, and the three-dimensional position and pose of the object being measured, which is based on the vision sensor 4, obtained by the image processing unit 21. Then, proceed to step S13.

[0044] Next, in step S13, the vision sensor 4 is moved to the position set when the robot 2's reference position is set. Specifically, in the camera position correction unit 12, the robot 2 is moved using the three-dimensional position and posture of the object 7 measured by the position and posture measurement unit 11, so that the vision sensor 4 is moved to the position set when the robot 2's reference position is set. Then, proceed to step S14.

[0045] Next, the input image is acquired in step S14. Specifically, the visual sensor 4, which has moved since step S13, takes another picture of the object 7 being measured, thereby acquiring the input image in the image acquisition unit 23. Then, the process proceeds to step S15.

[0046] Next, in step S15, the three-dimensional position and pose of the object being measured 7 are measured. Specifically, the image processing result obtained by the image processing unit 21 using the input image of the object being measured 7 captured by the vision sensor 4 after movement in step S13 in step S14 is used to measure the three-dimensional position and pose of the object being measured 7 by the position and pose measurement unit 11. Furthermore, the measurement sequence of the three-dimensional position and pose of the object being measured 7 in the position and pose measurement unit 11 is the same as in step S12. Then, proceed to step S16.

[0047] Next, in step S16, it is determined whether the camera position correction has been performed a predetermined number of times, for example, three times. If the determination is negative, the process returns to step S13, and the correction processing of the camera position correction unit 12, which consists of the movement of the vision sensor 4, image processing, and three-dimensional position and pose measurement, is performed again. If the determination is positive, the process proceeds to step S17.

[0048] Next, in step S17, the three-dimensional position and posture of the measurement object 7, which was last measured by the position and posture measurement unit 11, is used as the three-dimensional position and posture of the measurement object 7 to control the movement of the robot 2 relative to the object on which the measurement object 7 is mounted. After that, this process ends.

[0049] Furthermore, in the example of the flowchart in Figure 3, although the position and pose of the measured object is used in step S17, as mentioned above, when the movement of the vision sensor 4 is very small during the correction process, from the perspective of the influence of the robot 2's own movement accuracy on the correction error, the three-dimensional position and pose measured one before the last one can also be used.

[0050] Figure 4 is a diagram showing the changes in the input image when the camera position is moved. Figure 4 shows the changes in the input image when the correction processing of the camera position correction unit 12 is repeated three times. Also, Figure 4 shows an example of using a calibration fixture as the measurement object 7.

[0051] Here, the calibration fixture is explained in detail. Regarding the calibration fixture, previously known calibration fixtures that can be used to calibrate the vision sensor 4 can be used. The calibration fixture is a fixture used to obtain the information required for calibrating the vision sensor 4 by photographing a dot pattern arranged on a plane with the vision sensor 4. The dot pattern must meet the following three necessary conditions: (1) the grid spacing of the dot pattern is known; (2) there are a certain number of grid points; and (3) each grid point can be unambiguously identified as corresponding to a specific grid point. The calibration fixture is not limited to features such as the predetermined dot pattern shown in Figure 4 arranged on a two-dimensional plane; it can also be a feature arranged in a three-dimensional shape, as long as it can obtain three-dimensional position information, including position information in the height direction (Z direction) in addition to two-dimensional position information (X direction, Y direction). Furthermore, the calibration fixture can be the same as or different from the one used to perform the calibration of the vision sensor 4. Furthermore, in order to calculate the three-dimensional position and pose of the dot pattern with the visual sensor 4 as a reference from the dot pattern captured by the visual sensor 4, the internal parameters of the aforementioned correction data are used.

[0052] As shown in Figure 4, since the relative position of the camera and the measured object 7 will change when the camera position correction unit 12 performs correction processing, it can be seen that the three-dimensional position and pose of the measured object 7 in the input image gradually changes. Then, it can be seen that by repeating the correction processing a certain number of times, the movement of the vision sensor 4 will approach 0, and the position of the measured object 7 in the input image will converge to the center of the input image.

[0053] Furthermore, Figure 5 shows the position of the object being measured 7 before the camera position is moved and the position S of the object being measured 7 when the reference position is set. Figure 6 shows the position of the object being measured 7 after the camera position is moved and the position S of the object being measured 7 when the reference position is set. From Figures 5 and 6, it can be seen that by performing correction processing by the camera position correction unit 12, the position of the object being measured 7 in the input image converges to the position S of the object being measured 7 when the reference position is set in the center of the input image. That is, it can be seen that the current relative position between the camera position and the object being measured 7 is close to the relative position between the camera position and the object being measured 7 when the reference position of the robot 2 is set.

[0054] The position and posture measurement system 1 according to the first embodiment can achieve the following effects.

[0055] The position and posture measurement system 1 of this embodiment is provided with a camera position correction unit 12. The camera position correction unit 12 performs at least one correction process. The correction process uses the three-dimensional position and posture of the measurement object 7 measured by the position and posture measurement unit 11. After the robot 2 is moved to the position of the robot 2 by the position and posture movement unit 14, the image processing unit 21 and the measurement unit 11 are performed in sequence. The position of the robot 2 is such that the vision sensor 4 can approach the position of the vision sensor 4 as seen from the measurement object 7 when the reference position is set. Furthermore, the position and posture measurement system 1 of this embodiment is provided with a camera position correction termination unit 13. When the aforementioned camera position correction termination unit 13 determines that the correction processing of the camera position correction unit 12 has not been executed a predetermined number of times and does not end the correction processing, it executes the correction processing again. When it determines that the correction processing has been executed a predetermined number of times and it is time to end the correction processing, it adopts the three-dimensional position and posture of the measurement object 7 calculated by the position and posture measurement unit 11 at the end or in the middle as the three-dimensional position and posture of the measurement object 7 used for the control of the robot 2.

[0056] Therefore, through the correction processing of the camera position correction unit 12, the current relative position of the camera and the measured object 7 will be close to the relative position of the camera and the measured object 7 when the reference position of the robot 2 is set. Thus, according to this embodiment, since the deviation in the measurement of the three-dimensional position and posture of the measured object 7 by the position and posture measurement unit 11 can be suppressed, the three-dimensional position and posture of the measured object 7 can be accurately measured with simpler teaching and shorter measurement time than before. Furthermore, according to this embodiment, the measured three-dimensional position and posture of the measured object 7 can be used in operations requiring precise movements of the robot 2.

[0057] [A variation of the first implementation] The position and posture measurement system of the first embodiment is identical to the position and posture measurement system 1 of the first embodiment, except for the configuration of the camera position correction termination unit 13. Specifically, the camera position correction termination unit of the first embodiment determines whether to terminate the correction process based on the amount of movement of the visual sensor 4 during the correction process of the camera position correction unit 12. That is, the camera position correction termination unit of the first embodiment determines that the correction process should not be terminated when the amount of movement of the visual sensor 4 of the camera position correction unit 12 is greater than a predetermined threshold, and determines that the correction process should be terminated when the amount of movement of the visual sensor 4 is less than the predetermined threshold.

[0058] Figure 7 is a flowchart showing the processing sequence of the position and attitude measurement system of the modified example of the first embodiment. Steps S21 to S25 in Figure 7 are the same as steps S11 to S15 in the first embodiment.

[0059] In step S26, it is determined whether the movement of the vision sensor 4 is less than a predetermined threshold. Specifically, it is determined whether the movement of the vision sensor 4 in step S23 is less than the predetermined threshold. The movement of the vision sensor 4 can be calculated from the movement of the robot 2. If the determination is negative, return to step S23 and perform the correction processing of the camera position correction unit 12, which consists of the movement of the vision sensor 4, image processing, and three-dimensional position and pose measurement, again. If the determination is positive, proceed to step S27.

[0060] Next, in step S27, the same as step S17 in the first embodiment, the three-dimensional position and posture of the object 7 measured last by the position and posture measuring unit 11 is used as the three-dimensional position and posture of the object 7 for control of the robot 2. After that, this process ends.

[0061] Furthermore, in the example of the flowchart in Figure 7, although the position and pose of the measured object is used in step S27, it is the same as the first embodiment. When the movement of the vision sensor 4 is very small in the correction process, from the perspective of the influence of the robot 2's own movement accuracy on the correction error, for example, the three-dimensional position and pose measured one before the last one can also be used.

[0062] Alternatively, it can be configured to issue an alarm to notify the user when the movement of the visual sensor 4 is not less than a predetermined threshold. In this case, it is best to pre-set an upper limit for the number of correction processes.

[0063] The position and posture measurement system according to the modified example of the first embodiment can achieve the same effect as the position and posture measurement system 1 of the first embodiment. In addition, according to this modified example, when the amount of movement of the visual sensor 4 is very small from the beginning of the first correction process, the correction process can be performed in a shorter time than in the first embodiment.

[0064] [Second Implementation] Figure 8 is a block diagram showing the configuration of the position and posture measurement system 1A in the second embodiment. As shown in Figure 8, the position and posture measurement system 1A in the second embodiment is identical to the position and posture measurement system 1A in the first embodiment, except for the configuration of the vision sensor control device 20A. Specifically, in the position and posture measurement system 1A of the second embodiment, the vision sensor control device 20A further includes a measurement object brightness determination unit 24 and a measurement object brightness adjustment unit 25.

[0065] The object brightness determination unit 24 measures the brightness of the object 7 in the captured image processed by the image processing unit 21. Furthermore, the object brightness determination unit 24 determines whether the difference between the measured brightness and the brightness of the object 7 in the captured image when the robot 2's reference position is set is greater than a predetermined first threshold.

[0066] When the object brightness determination unit 24 determines that the brightness difference is greater than a predetermined first threshold, the object brightness adjustment unit 25 adjusts the brightness of the object 7 to be close to the brightness when the robot 2 is set to its reference position. Specifically, the object brightness adjustment unit 25 changes the exposure conditions of the vision sensor 4 or combines multiple images captured by the vision sensor 4 with different exposure conditions to make the brightness of the object 7 close to the brightness when the robot 2 is set to its reference position.

[0067] Furthermore, the processing of the object brightness determination unit 24 and the object brightness adjustment unit 25 is performed together with the correction processing of the camera position correction unit 12. That is, the processing of the object brightness determination unit 24 and the object brightness adjustment unit 25 is performed in parallel with the movement of the vision sensor 4, the image processing, and the three-dimensional position and pose measurement.

[0068] Figure 9 shows the changes in the input image when the camera position is moved and the brightness is adjusted. In the example shown in Figure 9, the brightness of the measured object 7 in the input image after the first camera position movement is lower than the brightness of the measured object 7 in the input image when the reference position is set, and the difference in brightness exceeds the first threshold. Therefore, the brightness is adjusted by moving the camera position a second time and changing the exposure conditions through the measured object brightness adjustment unit 25. As a result, it can be seen that the brightness of the measured object 7 in the input image after the second camera position movement is equal to the brightness of the measured object 7 in the input image when the reference position is set.

[0069] According to the position and pose measurement system 1A of the second embodiment, when taking a picture at the moved camera position, the exposure conditions of the vision sensor 4 are automatically adjusted so that the brightness of the measured object 7 in the captured image is close to the brightness at the reference position setting. Alternatively, an image can be synthesized from multiple captured images with different brightness. Image synthesis is performed using previously known HDR synthesis, etc. In this way, the brightness of the measured object 7 in the captured image can be made close to that at the reference position setting and when the vision sensor 4 is moved, which can suppress the increase in the measurement error of the three-dimensional position and pose of the measured object 7 caused by the difference between the brightness of the measured object 7 and the brightness at the reference position setting.

[0070] [Third Implementation Form] Figure 10 is a block diagram showing the configuration of the position and posture measurement system 1B in the third embodiment. As shown in Figure 10, the position and posture measurement system 1B in the third embodiment differs from the position and posture measurement system 1 in the first embodiment in that the configuration of the vision sensor control device 20B is different and it further includes an interface device 30. The rest of the configuration is the same as that in the first embodiment.

[0071] Specifically, in the position and posture measurement system 1B of the third embodiment, the vision sensor control device 20B further includes a measurement object brightness determination unit 24. This measurement object brightness determination unit 24 has a configuration that is basically the same as that of the second embodiment.

[0072] The object brightness determination unit 24 measures the brightness of the object 7 in the captured image processed by the image processing unit 21. Furthermore, the object brightness determination unit 24 determines whether the difference between the brightness of the object 7 in the captured image when the robot 2's reference position is set and that in the image is greater than a predetermined second threshold. Here, the predetermined second threshold can be set to the same value as the first threshold mentioned above, or it can be set to a value different from the first threshold.

[0073] The interface device 30 includes a measurement object information prompting unit 31. When the measurement object brightness determination unit 24 determines that the brightness difference is greater than a predetermined second threshold, the measurement object information prompting unit 31 prompts the user that the brightness of the measurement object 7 is significantly different from the brightness when the robot 2's reference position is set. For example, the measurement object information prompting unit 31 can display a pop-up message on the display screen of the interface device 30 stating, "The brightness of the measurement object (marker) is significantly different from the brightness when the reference position is set. It is recommended to change the lighting environment."

[0074] Furthermore, the measurement object information prompting unit 31 is not limited to being installed on the interface device 30. For example, instead of the interface device 30, the measurement object information prompting unit can be installed on the display unit of the vision sensor control device 20B.

[0075] Furthermore, the processing of the object brightness determination unit 24 and the object information prompting unit 31 is performed together with the correction processing of the camera position correction unit 12. That is, the processing of the object brightness determination unit 24 and the object information prompting unit 31 is performed in parallel with the movement of the vision sensor 4, the image processing, and the three-dimensional position and pose measurement.

[0076] Figure 11 shows the display screen of the interface device 30 when the difference in brightness of the measured object before and after the camera position is moved is greater than the second threshold. In the example shown in Figure 11, the brightness of the measured object 7 in the input image after the camera position is moved is lower than the brightness of the measured object 7 in the input image when the reference position is set, and the difference in brightness exceeds the second threshold. Therefore, it can be seen that the measured object information prompt unit 31 displays "The brightness of the mark is significantly different from the brightness when the reference position is set" in a pop-up form on the display screen of the interface device 30. It is recommended to change the lighting environment.

[0077] According to the position and posture measurement system 1B in the third embodiment, when the brightness of the measured object 7 differs significantly from the brightness at the reference position setting, the system can alert the user. This not only prompts the user to change the lighting environment but also allows the user to manually change the exposure conditions of the vision sensor 4, thus suppressing the increase in measurement error of the three-dimensional position and posture of the measured object 7 caused by changes in lighting conditions due to external factors.

[0078] [Fourth Implementation Form] Figure 12 is a block diagram showing the configuration of the position and posture measurement system 1C in the fourth embodiment. As shown in Figure 12, the position and posture measurement system 1C in the fourth embodiment differs from the position and posture measurement system 1 in the first embodiment in that the configuration of the vision sensor control device 20C is different and it further includes an interface device 30. The rest of the configuration is the same as that in the first embodiment.

[0079] Specifically, in the position and posture measurement system 1C of the fourth embodiment, the vision sensor control device 20C further includes a measurement object undetectable range determination unit 26. The measurement object undetectable range determination unit 26 measures the undetectable range of the measurement object 7 in the captured image processed by the image processing unit 21, and determines whether the measured value is greater than a predetermined undetectable threshold.

[0080] Here, the undetectable area of ​​the object 7 in the captured image refers to the area that cannot be detected due to reflections or halos caused by dust or any obstruction, or by the illumination installed on the vision sensor 4. The undetectable area of ​​the object 7 can be determined, for example, by the ratio of the area of ​​the undetectable area of ​​the object 7 to the area of ​​the entire captured image.

[0081] The interface device 30 includes a measurement object information prompt unit 31. When the measurement object undetectable range determination unit 26 determines that the measured value of the undetectable range exceeds a predetermined undetectable threshold, the measurement object information prompt unit 31 prompts the user that the undetectable range is very large. For example, the measurement object information prompt unit 31 can display a pop-up message on the display screen of the interface device 30 stating, "The undetectable range of the measurement object (marker) is very large. Please change the lighting environment or check whether the measurement object is obstructed by any object."

[0082] Furthermore, the measurement object information prompting unit 31 is not limited to being installed on the interface device 30. For example, instead of the interface device 30, the measurement object information prompting unit can be installed on the display unit of the vision sensor control device 20C.

[0083] Furthermore, the processing of the object detection range determination unit 26 and the object information prompting unit 31 is performed together with the correction processing of the camera position correction unit 12. That is, the processing of the object detection range determination unit 26 and the object information prompting unit 31 is performed in parallel with the movement of the vision sensor 4, the image processing, and the three-dimensional position and pose measurement.

[0084] Figure 13 shows the display screen of the interface device 30 when the undetectable range of the measured object 7 in the input image after the camera position has been moved exceeds the undetectable threshold. In the example shown in Figure 13, obstructions 8 and 9 overlap the measured object 7 in the input image after the camera position has been moved, and the undetectable range exceeds the undetectable threshold. Therefore, it can be seen that the measured object information prompt unit 31 displays "The undetectable range of the mark is very large. Please change the lighting environment or check whether the mark is obstructed by an obstruction" in a pop-up form on the display screen of the interface device 30.

[0085] According to the position and posture measurement system 1C of the fourth embodiment, when a portion of the object 7 being measured is obscured by external factors such as dust or any obstruction, or when it becomes undetectable in the captured image due to reflections or halos caused by lighting, the system measures the undetectable area. If the undetectable area exceeds a predetermined undetectable threshold, it provides information to attract the user's attention. This not only encourages the user to change the lighting environment to reduce the undetectable area of ​​the object 7, but also encourages the user to remove external factors. Therefore, according to this embodiment, the system can further suppress the increase in the measurement error of the three-dimensional position and posture of the object 7 caused by the expansion of the undetectable area.

[0086] Furthermore, this disclosure is not limited to the above-mentioned forms; variations and improvements within the scope of the purpose of this disclosure are included in this disclosure.

[0087] For example, the position and posture measurement system disclosed herein can also be applied to systems different from the above-described embodiments, where the robot 2 holds the measurement object 7 and the vision sensor 4 is mounted on a machine tool. In this case, the camera position correction unit 12 performs correction processing by moving the measurement object 7 using the position and posture movement unit 14. Furthermore, the camera position correction termination unit 13 performs a termination determination of the correction processing based on the amount of movement of the measurement object 7.

[0088] Alternatively, for example, a vision sensor 4 can be installed on the hands of different robots, and the vision sensor 4 can be used to perform the position and posture measurement system disclosed herein on two robots.

[0089] Alternatively, the camera position correction unit 12 in the above embodiment can be configured to perform at least one correction process based on the following steps: moving the vision sensor 4 or the object being measured 7 to approach the first position posture representing the reference posture of the object being measured 7 (i.e., the position posture when the position is set as the reference for correction), and then performing image processing by the image processing unit 21 and measurement by the position posture measurement unit 11. In this case, the camera position correction termination unit 13 in the above embodiment can be configured to determine whether to terminate the correction process based on the second position posture of the object being measured 7 measured by the position posture measurement unit 11 during the correction process of the camera position correction unit 12 (i.e., any position posture measured by the position posture measurement unit 11 using the correction process).

[0090] 1,1A,1B,1C: Position and attitude measurement system 2: Robot 3: Robotic arm 4: Vision Sensor 5: Flange 6: Tools for the fingers 7: The object being measured 8,9: Coverings 10: Robot control device 11: Position and Attitude Measurement Department 12: Camera position correction section 13: End of Camera Position Correction 14: Position and posture movement unit 20, 20A, 20B, 20C: Vision sensor control device 21: Image Processing Department 22: Correction Data Memory Section 23: Image Acquisition Unit 24: Brightness Judgment Unit of the Measured Object 25: Brightness adjustment section of the measurement object 26: Determination of the Detectable Range of the Measured Object 30: Interface device 31: Measurement Object Information Prompt Department 100: Robotic Systems S: Position when setting the reference position T: Taiwan S11~S17: Steps S21~S27: Steps X, Y, Z: Direction

Claims

1. A position and pose measurement system for measuring the three-dimensional position and pose of a measurement object used for robot control, the system comprising: a calibrated vision sensor; a position and pose movement unit capable of moving the measurement object or the three-dimensional position and pose of the vision sensor; a calibration data memory unit for pre-memorizing calibration data obtained during calibration of the vision sensor; an image processing unit for processing images captured by the vision sensor of the measurement object; and a position and pose measurement unit for measuring the three-dimensional position and pose of the measurement object using the image processing results from the image processing unit. The camera position correction unit performs at least one correction process. This correction process involves using the three-dimensional position and posture of the object being measured, as measured by the position and posture measurement unit, and moving the visual sensor or the object being measured by the position and posture movement unit. Then, image processing by the image processing unit and measurement by the position and posture measurement unit are performed sequentially. The camera position correction termination unit determines whether to terminate the aforementioned correction process. If it is determined that the process is not yet terminated, the correction process is performed again. If it is determined that the process is about to terminate, the three-dimensional position and posture of the object being measured, either at the end or midway by the position and posture measurement unit, is used as the three-dimensional position and posture of the object being measured for control of the robot. The position and posture measurement system further includes: The object brightness determination unit measures the brightness of the object to be measured in the image captured after processing by the image processing unit, and determines whether the difference between the brightness of the object to be measured and the brightness of the object to be measured in the image captured when the robot's reference position is set is greater than a predetermined first threshold; and the object brightness adjustment unit adjusts the brightness of the object to be measured so that it is close to the brightness when the robot's reference position is set when the object brightness determination unit determines that the difference in brightness is greater than the predetermined first threshold.

2. A position and pose measurement system for measuring the three-dimensional position and pose of a measurement object used for robot control, the system comprising: a calibrated vision sensor; a position and pose movement unit capable of moving the measurement object or the three-dimensional position and pose of the vision sensor; a calibration data memory unit for pre-memorizing calibration data obtained during calibration of the vision sensor; an image processing unit for processing images captured by the vision sensor of the measurement object; and a position and pose measurement unit for measuring the three-dimensional position and pose of the measurement object using the image processing results from the image processing unit. The camera position correction unit performs at least one correction process, which involves using the three-dimensional position and posture of the object being measured as measured by the position and posture measurement unit, moving the vision sensor or the object being measured by the position and posture movement unit, and then sequentially performing image processing by the image processing unit and measurement by the position and posture measurement unit; and the camera position correction termination unit determines whether to terminate the aforementioned correction process of the camera position correction unit. If it is determined that the correction process is not yet terminated, the aforementioned correction process is performed again. If it is determined that the correction process is about to terminate, the three-dimensional position and posture of the object being measured by the position and posture measurement unit at the end or in the middle is used as the three-dimensional position and posture of the object being measured for use in the control of the robot. The aforementioned position and posture measurement system further includes: The object brightness determination unit measures the brightness of the object to be measured in the image captured after processing by the image processing unit, and determines whether the difference between the brightness of the object to be measured and the brightness of the object to be measured in the image captured when the robot's reference position is set is greater than a predetermined second threshold; and the object information prompting unit prompts the user when the object brightness determination unit determines that the difference in brightness is greater than the predetermined second threshold, indicating that the brightness of the object to be measured is significantly different from the brightness when the robot's reference position is set.

3. A position and pose measurement system for measuring the three-dimensional position and pose of a measurement object used for robot control, the system comprising: a calibrated vision sensor; a position and pose movement unit capable of moving the measurement object or the three-dimensional position and pose of the vision sensor; a calibration data memory unit for pre-memorizing calibration data obtained during calibration of the vision sensor; an image processing unit for processing images captured by the vision sensor of the measurement object; and a position and pose measurement unit for measuring the three-dimensional position and pose of the measurement object using the image processing results from the image processing unit. The camera position correction unit performs at least one correction process, which involves using the three-dimensional position and posture of the object being measured as measured by the position and posture measurement unit, moving the vision sensor or the object being measured by the position and posture movement unit, and then sequentially performing image processing by the image processing unit and measurement by the position and posture measurement unit; and the camera position correction termination unit determines whether to terminate the aforementioned correction process of the camera position correction unit. If it is determined that the correction process is not yet terminated, the aforementioned correction process is performed again. If it is determined that the correction process is about to terminate, the three-dimensional position and posture of the object being measured by the position and posture measurement unit at the end or in the middle is used as the three-dimensional position and posture of the object being measured for use in the control of the robot. The aforementioned position and posture measurement system further includes: The undetectable range determination unit measures the undetectable range of the aforementioned object in the captured image processed by the aforementioned image processing unit, and determines whether the undetectable range is greater than a predetermined undetectable threshold; and the object information prompting unit prompts the user that the undetectable range of the aforementioned object is large when the undetectable range determination unit determines that the undetectable range is greater than the predetermined undetectable threshold.

4. The position and posture measurement system of any one of claims 1 to 3, wherein the aforementioned camera position correction unit performs the aforementioned correction process after causing the aforementioned robot to move to the aforementioned robot's position, and the aforementioned robot's position is such that the aforementioned visual sensor can approach the position of the aforementioned visual sensor as seen from the aforementioned measurement object when the position is set as a reference position for correction.

5. In any of the claims 1 to 3, the position and attitude measurement system determines that the aforementioned camera position correction termination unit should terminate the aforementioned correction process when the aforementioned correction process has been executed a predetermined number of times.

6. The position and posture measurement system of claim 4, wherein the aforementioned camera position correction termination unit determines that the aforementioned correction process should be terminated when the aforementioned correction process of the aforementioned camera position correction unit has been executed a predetermined number of times.

7. In any of the claims 1 to 3, the position and posture measurement system determines that the aforementioned correction process should be terminated when the movement of the aforementioned visual sensor or the aforementioned measured object is less than a predetermined threshold.

8. The position and posture measurement system of claim 4, wherein the aforementioned camera position correction termination unit determines to terminate the aforementioned correction process when the amount of movement of the aforementioned visual sensor or the aforementioned measured object before the aforementioned camera position correction unit is less than a predetermined threshold.

Images