Imaging System

Abstract

To provide a technique that can obtain an inspecting image photographed at an appropriate position and at an appropriate angle.SOLUTION: A photographing system for an inspection comprises; a camera that receives a signal to instruct the camera to photograph and then photographs objects; a robot arm mounted with the camera; and a control device that makes the robot arm move the camera so that the camera is arranged in directions predetermined respectively at a plurality of preset spots. The control device includes: a detecting part that calculates a current position and a current posture of the camera; a storing part that stores information representing a position and a posture which are preset as a position and a posture in which the camera photographs the object; and an instructing part that when the calculated current position and posture of the camera match the preset position and posture, transmits to the camera the signal to instruct the camera to photograph the object.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to imaging systems. [Background technology]

[0002] A workpiece may be imaged by a camera for use in visual inspection. The captured image data may not only be used for visual inspection, but may also be used as learning data for learning a machine learning model when a machine learning model is adopted for visual inspection. Patent Document 1 describes a method of imaging a workpiece in which, with a camera fixed, images of multiple parts of the workpiece are captured while the workpiece to be inspected is rotated.

[0003] When the shape of the workpiece to be inspected is complex, multiple cameras must be prepared in order to capture images of multiple parts of the workpiece using fixed cameras. Therefore, the inventors thought that it would be desirable to capture images of multiple parts of the workpiece for inspection by controlling a robot arm to which a camera is attached, while appropriately changing the position of the camera attached to the end of the robot arm. In this method, when the robot arm receives an instruction to start operation from the control device, it moves the camera along a preset path. The preset path includes all positions where the camera should capture images of the workpiece. The camera captures images of the workpiece at regular time intervals from the start to the completion of the operation of the robot arm.

[0004] Here, the many image data captured before tracing is completed include image data captured at the position and angle at which the workpiece should be captured, and image data at an undesirable position and angle. The image data required for inspection is image data captured at a specific position and angle at which the workpiece should be captured. For this reason, for example, while the robot arm is moving the camera, it notifies the computer that acquires the image data that the camera position and angle have reached a specific state that has been set in advance. When the computer is notified by the robot arm, it reads out the image data captured immediately before from the camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-160785 A Summary of the Invention [Problem to be solved by the invention]

[0006] In the above-mentioned method, while the robot arm is moving the camera, the robot arm notifies the computer of the timing of acquiring image data, and the computer that receives the notification reads the image data stored in the camera. Therefore, the timing at which the computer reads the image data from the camera is delayed from the timing at which the position and angle of the camera reach a specific state that has been set in advance. Therefore, a situation may occur in which the computer cannot read the image captured at the appropriate position and angle. [Means for solving the problem]

[0007] The present disclosure can be realized in the following forms. According to an embodiment of the present disclosure, there is provided an imaging system for inspection, comprising: a camera that receives a signal instructing imaging to capture an image of an object, a robot arm to which the camera is attached, a control device that moves the camera on the robot arm so that the camera is disposed in a predetermined orientation at a plurality of preset points, and an inspection device that performs an inspection based on the image. The control device includes a detection unit that calculates the current position and attitude of the camera, a memory unit that stores imaging parameters representing a preset position and attitude at which the camera will image the object, and an instruction unit that transmits information indicating the current position and attitude to the camera along with a signal instructing imaging when the current position and attitude of the camera calculated by the detection unit match the preset position and attitude. The camera transmits the information indicating the current position and attitude to the inspection device along with the captured image, and the inspection device determines from the received information indicating the current position and attitude whether the camera has imaged the object at the preset position and attitude, and if it determines that the camera has imaged the object at the preset position and attitude, determines whether there is a misalignment in the image captured by the camera, and if it determines that there is a misalignment in the image, outputs a notification that the image cannot be used as inspection data.

[0008] According to an embodiment of the present disclosure, there is provided an imaging system for inspection. The imaging system includes a camera that receives a signal instructing imaging and images an object, a robot arm to which the camera is attached, and a control device that moves the camera on the robot arm so that the camera is disposed in a predetermined orientation at a plurality of preset points. The control device includes a detection unit that calculates a current position and orientation of the camera, a storage unit that stores information representing a position and orientation that are preset as a position where the camera will image the object, and an instruction unit that transmits a signal instructing the camera to image when the calculated current position and orientation of the camera match the position and orientation that are preset as a position where the image will be captured. According to this aspect, when the camera reaches the imaging point, the control device that controls the robot arm instructs the camera to capture an image, so that the position and orientation at which the camera captures the target part can always be consistent, thereby enabling the computer to obtain images captured at appropriate positions and angles. [Brief description of the drawings]

[0009] [Figure 1] 1 is a perspective view illustrating an appearance of an imaging robot system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of an imaging robot system according to an embodiment. [Diagram 3] 4 is a flowchart of an imaging control process. [Figure 4] 13 is a flowchart of an imaging process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A. Embodiment FIG. 1 is a perspective view showing the appearance of an imaging system 1 for inspection according to an embodiment. FIG. 2 is a block diagram showing the functional configuration of the imaging system 1. The imaging system 1 captures an image of an inspection target in a production line for visual inspection. As shown in FIG. 1, the imaging system 1 captures an image of an inspection target workpiece WK1 that is transported by a conveyor 50 and placed at a fixed position. In the embodiment, it is assumed that the imaging target workpiece WK1 is placed at a fixed position in the transport direction of the conveyor 50 and at a fixed position in the width direction of the conveyor 50. As shown in FIG. 2, the imaging system 1 transmits captured image data to an inspection device 500.

[0011] The imaging system 1 includes a robot 100, a robot control device 200, and a camera 300.

[0012] 1, the robot 100 is a single-arm robot. The robot 100 is also called a robot arm. The robot 100 operates under the control of a robot control device 200. The robot 100 includes an arm 110, an arm flange 120, and a base 130.

[0013] The arm 110 has six joints J1 to J6. The joints J2, J3, and J5 are bending joints, and the joints J1, J4, and J6 are torsion joints. A servo motor 111 and an encoder 112 are provided in each joint. In FIG. 1, the servo motor 111 and the encoder 112 are omitted in order to facilitate understanding of the technology. The robot 100 has six joints J1 to J6, but in FIG. 2, the servo motor 111 and the encoder 112 corresponding to only one joint are illustrated in order to facilitate understanding of the technology. The servo motor 111 drives the corresponding joint shaft according to the control of the robot control device 200. The encoder 112 detects the angular position of the output shaft of the servo motor 111 at a determined time interval, and outputs information indicating the detected angular position to the robot control device 200.

[0014] The robot 100 can position the camera 300 attached to the tip of the arm 110 at a specified position in three-dimensional space in a specified attitude by rotating each of the six joints J1 to J6 with a servo motor 111.

[0015] The arm flange 120 is disposed at the tip of the joint J6 of the arm 110. The arm flange 120 has a camera 300 attached thereto.

[0016] The robot controller 200 controls the operation of the robot 100 by driving the servo motors 111 corresponding to each joint of the robot 100. The robot 100 and the robot controller 200 are communicatively connected via a cable. The robot controller 200 also instructs the camera 300 to capture an image of the workpiece WK1. In the embodiment, when the robot 100 places the camera 300 at a specified position in a specified attitude, the robot controller 200 instructs the camera 300 to capture an image. The robot controller 200 and the camera 300 are communicatively connected via a cable.

[0017] 2, the robot control device 200 is a computer including a CPU (Central Processing Unit) 201 and a memory 202. The CPU 201 realizes various functions by executing various programs stored in the memory 202. The memory 202 includes a volatile memory and a non-volatile memory.

[0018] The robot control device 200 functionally comprises a position detection unit 210, an imaging parameter storage unit 220, an instruction unit 230, and a control unit 240.

[0019] The position detection unit 210 calculates the position and orientation of the camera 300 from the information indicating the angular positions of the joint axes supplied from the encoder 112. The orientation of the camera 300 is expressed by a combination of angular positions around the X-axis, the Y-axis, and the Z-axis in the robot coordinate system. The robot coordinate system is a coordinate system that defines the space in which the robot 100 is installed. The robot coordinate system is a three-dimensional Cartesian coordinate system defined by the X-axis and the Y-axis that are orthogonal to each other on a horizontal plane, and the Z-axis whose positive direction is vertically upward. The coordinate system shown in FIG. 1 is the robot coordinate system. Any position in the three-dimensional space can be expressed by the positions in the X-axis, Y-axis, and Z-axis directions. Any orientation in the three-dimensional space can be expressed by the rotation angles in the X-axis, Y-axis, and Z-axis directions. The position detection unit 210 is also called a detection unit. The function of the position detection unit 210 is realized by the CPU 201.

[0020] The imaging parameter storage unit 220 stores imaging parameters that are set in advance for the position and orientation when the camera 300 images the workpiece WK1. Specifically, the imaging parameter storage unit 220 stores, as the imaging parameters, information indicating the position at which the camera 300 images the workpiece WK1 and information indicating the orientation of the camera 300 at that position. The imaging parameter storage unit 220 is also referred to as a storage unit. The function of the imaging parameter storage unit 220 is realized by the memory 202.

[0021] When the instruction unit 230 determines that the current position and orientation of the camera 300 are the preset position and orientation while the camera 300 is being moved along a predetermined path, the instruction unit 230 instructs the camera 300 to capture an image of the workpiece WK1. The function of the instruction unit 230 is realized by the CPU 201.

[0022] The control unit 240 drives the servo motors 111 that drive the joint shafts based on a program and information indicating the angular positions of the joint shafts supplied from the encoder 112, thereby causing the robot 100 to execute an arm movement operation. The arm movement operation in this embodiment is an operation in which the robot 100 follows a pre-specified trajectory. The robot 100 operates to follow the pre-specified trajectory under the control of the control unit 240. As a result, the camera 300 attached to the arm 110 of the robot 100 is moved sequentially to a plurality of pre-specified points. At each point, the camera 300 is positioned in a pre-specified attitude. The functions of the control unit 240 are realized by the CPU 201.

[0023] The camera 300 captures an image of the workpiece WK1 at a position and posture arranged by the robot 100. The camera 300 includes an imaging unit 310 and a transmission unit 320. When the imaging unit 310 receives a signal instructing imaging from the instruction unit 230 of the robot control device 200, the imaging unit 310 captures an image of the workpiece WK1. The transmission unit 320 transmits the captured image data to the inspection device 500 by wireless communication.

[0024] The inspection device 500 is a computer that performs an appearance inspection of the workpiece WK1 based on the received image data. The inspection device 500 uses the image data supplied from the camera 300 as input data and performs an appearance inspection using a trained machine learning model. The appearance inspection is an inspection that checks for appearance defects such as foreign matter attached to the surface of a part and deformation of the part.

[0025] FIG. 3 is a flowchart of the imaging control process executed by the robot control device 200. In the embodiment, the robot control device 200 starts the imaging control process when it starts its operation. When the robot control device 200 is instructed by a controller (hereinafter referred to as a line controller) that controls the entire production line (not shown) to operate the robot 100 and take an image of the workpiece WK1, the robot control device 200 causes the robot 100 to start the arm movement operation. Note that the line controller operates the conveyor 50 to transport the workpiece WK1 to a fixed position and stops the conveyor 50. The robot control device 200 determines that the arm movement operation is completed when the robot 100 returns to the initial position before starting the arm movement operation after placing the camera 300 at the last point among a plurality of points set in advance.

[0026] In step S 11 , the position detection unit 210 calculates the position and attitude of the camera 300 from information indicating the angular position detected by the servo motor 111 .

[0027] In step S12, the instruction unit 230 determines whether or not the calculated position and orientation of the camera 300 match the position and orientation indicated by the imaging parameters stored in the imaging parameter storage unit 220. If it is determined that they match (step S12; YES), the instruction unit 230 executes the process of step S13. On the other hand, in step S12, if the instruction unit 230 determines that the calculated position and orientation of the camera 300 do not match the position and orientation indicated by the imaging parameters (step S12; NO), the instruction unit 230 executes the process of step S14.

[0028] In step S13, instruction unit 230 transmits a signal instructing camera 300 to capture an image. Instruction unit 230 counts the number of times that instruction unit 230 has instructed camera 300 to capture an image since starting the imaging control process. Thereafter, instruction unit 230 executes the process of step S14.

[0029] In step S14, the instruction unit 230 judges whether or not imaging at all imaging points has been completed. The instruction unit 230 judges that imaging at all imaging points has been completed when the number of times that the camera 300 has been instructed to capture images since the imaging control process started reaches a preset value. The preset value is the total number of preset points at which the camera 300 is placed. If imaging at all points has been completed (step S14; YES), the imaging control process is terminated. Furthermore, the instruction unit 230 notifies the line controller that imaging of the workpiece WK1 to be currently captured has been completed. In response to this, the line controller operates the conveyor 50 to transport the workpiece WK1 to be next captured to a fixed position and stops the conveyor 50. After that, the line controller again instructs the robot control device 200 to capture the workpiece WK1.

[0030] On the other hand, if the movement is not completed (step S14; NO), the process of step S11 is executed again. The imaging control process is as described above.

[0031] FIG. 4 is a flowchart of the imaging process executed by the camera 300. When the camera 300 is powered on, the following imaging process is started. For example, the camera 300 is powered on prior to the start of the imaging control process shown in FIG. 3. The camera 300 executes the following imaging process until the power is cut off. In step S21, the imaging unit 310 waits until it receives an instruction to image from the instruction unit 230 of the robot control device 200 (step S21; NO). When the imaging unit 310 receives an instruction to image from the instruction unit 230 of the robot control device 200 (step S21; YES), in step S22, it images the workpiece WK1 to be inspected. The imaging unit 310 stores the captured image data in an internal memory. In step S23, the transmission unit 320 transmits the captured image data to the inspection device 500 by wireless communication. After that, in step S21, the imaging unit 310 waits for an instruction to image from the instruction unit 230 of the robot control device 200. The imaging process is completed.

[0032] As described above, in the embodiment, when the camera 300 reaches the imaging point, the robot control device 200 instructs the camera 300 to capture an image, so that the position and posture at which the camera 300 captures an image of a target portion of each workpiece WK1 can always be consistent. Therefore, the inspection device 500 can obtain an image captured at an appropriate position and posture.

[0033] Also, for example, when the camera captures the workpiece at a constant time interval from the start to the completion of the robot's operation, the camera may not necessarily be able to capture the workpiece in an appropriate position and posture. It is possible to shorten the time interval for capturing images, but it is expected that the number of captured image data will be enormous. Furthermore, it is also time-consuming to sort the images captured in an appropriate position and posture from the enormous amount of image data. On the other hand, in the aspect according to the embodiment, the camera 300 captures the workpiece WK1 at the timing when the image capturing is instructed, so that the inspection device 500 can obtain the image captured in an appropriate position and posture. Furthermore, since the camera 300 does not capture unnecessary images, there is no need to take the time to sort the image data.

[0034] B1. Other embodiment 1 In the embodiment, an example has been described in which the robot control device 200 calculates the position and orientation of the camera 300 from information on the angular position detected by the encoder 112 provided on each joint axis of the robot 100. However, the robot control device 200 may calculate the position and orientation of the camera 300 by other methods. For example, another camera is prepared to capture an image of the camera 300 attached to the robot 100. While the camera 300 is moving, this other camera captures the image of the camera 300 at short intervals and transmits the captured image to the robot control device 200. The robot control device 200 may calculate the position and orientation of the camera 300 from the image provided by the other camera.

[0035] B2. Other embodiment 2 In the embodiment, it is assumed that the workpiece WK1 is placed at a fixed position when performing the visual inspection, but the position of the workpiece WK1 placed on the conveyor 50 may be shifted. For this reason, the camera 300 may transmit information indicating the position and orientation of the camera 300 at the imaging point together with the captured image data to the inspection device 500. In this case, the robot control device 200 transmits information indicating the position and orientation of the camera 300 calculated by the position detection unit 210 to the camera 300 together with an instruction to capture an image.

[0036] The inspection device 500 judges whether the camera 300 has captured the workpiece WK1 at a determined position and posture from the information indicating the position and posture received together with the image data. When the inspection device 500 judges that the camera 300 has captured the workpiece WK1 at a determined position and posture, it judges whether there is a misalignment in the image data. When the inspection device 500 judges that there is a misalignment in the image data, it notifies, for example, an administrator, that the image data cannot be used as inspection data. Thus, it is possible to prevent inspection from being performed using inappropriate image data. Alternatively, the inspection device 500 may correct the misalignment in the image data if it is within a range that can be corrected. In this case, the corrected image data can be used as inspection data.

[0037] B3. Other embodiment 3 Furthermore, the inspection device 500 may output statistics on the pass / fail results of the appearance inspection using the images of each of the workpieces WK1 captured at the same location. The manager of the imaging system 1 can review whether an imaging location is appropriate or not based on the statistics and the ratio of pass / fail inspection results at the imaging location.

[0038] B4. Other embodiment 4 The robot control device 200 may control the robot 100 to slow down the movement speed of the robot 100 for a determined period of time after the robot 100 reaches a set imaging point. The determined period of time is set to be a very short time. This allows the camera 300 to capture an image of the workpiece WK1 while moving at a low speed, and for example, makes it possible to suppress blurring of the image data.

[0039] B5. Other embodiment 5 1 shows an example in which a pair of the robot 100 and the camera 300 is used to image the workpiece WK1. However, two sets of the robot 100, the robot control device 200, and the camera 300 may be used to image the workpiece WK1. Alternatively, three or more sets of the robot 100, the robot control device 200, and the camera 300 may be used to image the workpiece WK1.

[0040] B6. Other embodiment 6 In the embodiment, an example has been described in which the image data acquired by the imaging system 1 is used for visual inspection. Alternatively, the image data acquired by the imaging system 1 may be used as learning data for machine learning.

[0041] For example, in the imaging system 1, the camera 300 captures images of a large number of workpieces WK1 and transmits the captured image data to a machine learning computer that performs machine learning. The machine learning computer that receives the image data sorts the multiple images received from the camera 300 by imaging location, and uses the sorted images as input data for a machine learning model.

[0042] In machine learning, for example, a large number of image data of good workpieces and image data of defective workpieces are prepared. These image data are used as learning data to make a machine learning computer learn the characteristics of good products and the characteristics of defective products. A machine learning model is generated by machine learning. The computer incorporating the learned machine learning model can output the probability that the workpiece is a good product and the probability that the workpiece is a defective product based on the image data of the captured workpiece.

[0043] For example, assume that image data of specific parts of multiple non-defective workpieces is used as learning data. In order to increase the accuracy of the learning data, it is desirable that there is no misalignment between the multiple image data. In this case, it is preferable that each image data is captured at the same position and posture. As described above, the camera 300 can always capture the image of each workpiece WK1 at the same position and angle, so the machine learning computer can obtain image data that is preferable as learning data.

[0044] B7. Other embodiment 7 In the embodiment, it is assumed that the workpiece WK1 is arranged at a fixed position, but the position of the workpiece WK1 arranged on the conveyor 50 may be shifted. For this reason, the camera 300 may transmit information indicating the position and orientation of the camera 300 at the imaging point together with the captured image data to the machine learning computer. In this case, the robot control device 200 transmits information indicating the position and orientation of the camera 300 calculated by the position detection unit 210 to the camera 300 together with an instruction to capture an image.

[0045] The machine learning computer determines whether the camera 300 has captured the image of the workpiece WK1 at a determined position and posture from the information indicating the position and posture received together with the image data. If the computer determines that the camera 300 has captured the image of the workpiece WK1 at a determined position and posture, it determines whether there is a misalignment in the image data. If the machine learning computer determines that there is a misalignment in the image data, it removes the image data from the learning data. This makes it possible to prevent inappropriate image data from being used as learning data.

[0046] Alternatively, the machine learning computer may correct the misalignment of the image data if the misalignment is within a range that can be corrected, in which case the corrected image data can be used as training data.

[0047] B8. Other embodiment 8 In the embodiment, an example has been described in which the camera 300 transmits image data of the workpiece WK1 to the inspection device 500 by wireless communication. However, the camera 300 may transmit image data of the workpiece WK1 to the inspection device 500 by wired communication.

[0048] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the present disclosure can be appropriately replaced or combined in order to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0049] 1...imaging system, 50...conveyor, 100...robot, 110...arm, 111...servo motor, 112...encoder, 120...arm flange, 130...base, 200...robot control device, 201...CPU, 202...memory, 210...position detection unit, 220...imaging parameter storage unit, 230...instruction unit, 240...control unit, 300...camera, 310...imaging unit, 320...transmission unit, 500...server, J1 to J6...joints, WK1...work

Claims

[Claim 1] 1. An imaging system for inspection, comprising: a camera that receives a signal instructing the camera to capture an image of a target; a robot arm to which the camera is attached; and a control device that causes the robot arm to move the camera so that the camera is disposed in a predetermined orientation at each of a plurality of preset points; an inspection device that performs inspection based on the image; Equipped with The control device includes: A detection unit that calculates the current position and orientation of the camera; a storage unit that stores imaging parameters representing a position and an attitude that are preset as a position at which the camera captures an image of the target; an instruction unit that transmits, when the current position and orientation of the camera calculated by the detection unit coincide with the preset position and orientation, information indicating the current position and orientation to the camera together with a signal instructing the camera to capture an image; Including, the camera transmits the captured image together with the information indicating the current position and orientation to the inspection device; The inspection device includes: determining whether the camera has captured an image of the target at the preset position and orientation based on the received information indicating the current position and orientation; When it is determined that the camera has captured an image of the object at the preset position and orientation, it is determined whether or not there is a misalignment in the image captured by the camera; outputting a notice that the image cannot be used as inspection data when it is determined that the image has the misalignment; Imaging system.

Images