Inspection system and inspection method

Abstract

To provide an inspection system and inspection method that can discriminate between success and failure of a work with respect to a workpiece.SOLUTION: An inspection system 100 includes: a robot 1 for executing a work with respect to a workpiece 3; a camera 2; and a control device 4 for controlling the robot 1 and camera 2. The control device 4 is configured to: acquire a pre-work image of the workpiece 3 before executing the work with respect to the workpiece 3; further, acquire a post-work image of the workpiece 3 after executing the work with respect to the workpiece 3; execute image correction processing to at least one of the pre-work image and post-work image so that positions of a plurality of characteristic points within the pre-work image match up with positions of a plurality of corresponding characteristic points within the post-work image respectively; extract a differential between the pre-work image and the post-work image and thereby acquire a differential image; and discriminate between success and failure of the work with respect to the workpiece 3.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention generally relates to an inspection system and method for determining whether or not work has been performed on a workpiece using a robot, and more specifically, to an inspection system and method for determining whether or not work has been performed on a workpiece using a robot based on a differential image obtained by using a camera to acquire a pre-work image of the workpiece before the work is performed on the workpiece and a post-work image of the workpiece after the work has been performed on the workpiece, and extracting the difference between the pre-work image and the post-work image. [Background technology]

[0002] In recent years, industrial robots for assembly and processing have been widely used in factory production lines to automate and unmanned operations. Industrial robots perform specified tasks on objects to be worked on (referred to as "workpieces") in factory production lines, such as attaching parts, applying liquid materials, and processing such as cutting. Such industrial robots are equipped with movable parts such as a robot arm and a movable stage, and a working part (e.g., a pick-up hand and a nozzle) for performing the specified tasks on the workpiece, and the movable part controls the relative position between the working part and the workpiece to perform the specified tasks on the workpiece.

[0003] In addition, when such an industrial robot is used, it is common to inspect whether an operation on a workpiece has been performed. For example, whether or not a part has been attached to a predetermined work point on the workpiece, a liquid material has been applied, cutting, or the like has been performed without positional deviation is inspected based on an image of the workpiece. In such an inspection, a camera is used to obtain a pre-operation image of the workpiece before the operation on the workpiece is performed and a post-operation image of the workpiece after the operation on the workpiece is performed, and a difference image is obtained by extracting the difference between the two images, and the success or failure of the operation on the workpiece is judged based on the difference image. By referring to the difference image, information on the actually performed operation, such as the attachment position of the part and the area (amount of application) of the liquid material, can be obtained. Based on such information on the actually performed operation, the success or failure of the operation on the workpiece can be judged.

[0004] Also, a technique is known in which a difference image is obtained by extracting the difference between a master image obtained in advance by photographing a sample workpiece on which an operation has been performed accurately and a post-operation image of the workpiece to be inspected, and the difference image is used to determine whether the operation on the workpiece to be inspected has been performed or not. However, in this case, differences due to factors other than the operation on the workpiece, such as differences in the size and shape of the workpiece to be inspected compared to the sample workpiece due to manufacturing errors, assembly errors, etc., are also extracted. Therefore, when it is desired to accurately determine only the success or failure of the operation on the workpiece, it is preferable to use a difference image obtained from the pre-operation image and post-operation image of the workpiece, from which only differences due to the operation on the workpiece are extracted.

[0005] Inspection using a difference image obtained from a pre-operation image and a post-operation image of a workpiece is very useful in the field of FA using industrial robots, but when obtaining the difference image, the position of each element of the workpiece in the pre-operation image must be exactly the same as the position of each corresponding element of the workpiece in the post-operation image. In particular, in the field of FA using industrial robots, the workpiece is often a very small precision part, and the tolerance for positional deviation between the position of each element of the workpiece in the pre-operation image and the position of each corresponding element of the workpiece in the post-operation image is very small.

[0006] In order to reduce the positional deviation between the position of each element of the work in the pre-operation image and the position of the corresponding element of the work in the post-operation image, it is common to adjust the relative position of the camera with respect to the work by controlling an industrial robot when acquiring a post-operation image of the work. However, in this case, the accuracy of the position correction of the relative position of the camera with respect to the work depends on the positioning accuracy and control resolution of the industrial robot itself, so high positioning accuracy cannot be expected. For example, the allowable tolerance is often in units of μm or less, whereas the positioning accuracy and control resolution of the industrial robot itself are often on the level of several μm to several tens of μm. In this case, even if the industrial robot is controlled, it is very difficult to make the positional deviation between the position of each element of the work in the pre-operation image and the position of the corresponding element of the work in the post-operation image equal to or less than the allowable tolerance. In addition, if the position of the work changes during the operation on the work and the position of the work with respect to the jig of the robot occurs, the position correction cannot be performed even if the industrial robot is controlled.

[0007] In response to such a problem, a technique is known in which, when acquiring a difference image from two images, a block matching technique is used to detect a positional shift between the two images, and based on the detected positional shift, one or both images are corrected in the XY direction (see Patent Document 1). In the technique disclosed in Patent Document 1, each of the two images is divided into a plurality of blocks, and corresponding feature points in each block of the two images are identified by feature point matching to detect the amount of positional shift between corresponding feature points in each block of the two images, and further, based on the detected amount of positional shift between corresponding feature points, the positional correction in the XY direction of one or both images is performed. Such a technique disclosed in Patent Document 1 is premised on the fact that no image deformation occurs due to differences in the attitude or orientation of the camera with respect to the subject for acquiring the two images to be compared to acquire the difference image. In other words, in order to use the positional correction using the technique disclosed in Patent Document 1, it is necessary to satisfy the condition that the attitude or orientation of the camera with respect to the subject for acquiring the two images is constant.

[0008] FIG. 1 shows a number of examples of deformation of a camera image 500 obtained when a subject (a crab in the illustrated example) is photographed by changing the camera's posture and orientation in various ways. As shown in FIG. 1, the state of deformation of the camera image 500 changes depending on the camera's posture and orientation. For example, as shown in FIG. 1, a rectangular subject is deformed into a parallelogram or a trapezoid. The technology disclosed in Patent Document 1 does not take into account the influence of image deformation caused by differences in the posture and orientation of the camera with respect to such a subject. Therefore, when the deformations of two images compared to obtain a difference image are different from each other, the technology disclosed in Patent Document 1 cannot be used.

[0009] However, in the field of FA using industrial robots, when performing work on a workpiece, there are cases where the camera rotates or tilts, changing the attitude or orientation of the camera relative to the workpiece, or the workpiece is displaced or tilted on the worktable. In such cases, the deformations of the pre-work image and the post-work image of the workpiece are different from each other. Therefore, even if the position correction technology disclosed in Patent Document 1 that corrects the positions of two images in the XY directions is used in the field of FA using industrial robots, the positional deviation between the positions of each element of the workpiece in the pre-work image and the positions of the corresponding elements of the workpiece in the post-work image cannot be accurately corrected, and there is a problem that it is not possible to accurately determine whether the work on the workpiece has been completed.

[0010] In addition, in order to accurately correct the positional deviation between the position of each element of the work in the pre-operation image and the position of each corresponding element of the work in the post-operation image, it is necessary to accurately identify multiple feature points in the pre-operation image and feature points in the post-operation image that correspond to the multiple feature points in the pre-operation image by feature point matching. In order to improve the accuracy of feature point matching, Patent Document 2 discloses a technology that detects feature vectors including distances and angles between multiple feature points included in each of the two images to be compared, and accurately detects corresponding feature points of the two images to be compared by using the detected feature vectors. However, even if the conventional technology disclosed in Patent Document 2 is used, there is a problem that erroneous feature point matching is performed. Such erroneous feature point matching makes it difficult to accurately correct the positional deviation between the position of each element of the work in the pre-operation image and the position of each corresponding element of the work in the post-operation image and accurately determine whether the work on the work has been completed. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] JP 2013-186562 A [Patent Document 2] JP 2019-101975 A Summary of the Invention [Problem to be solved by the invention]

[0012] The present invention has been made in consideration of the above-mentioned problems in the conventional art, and its object is to provide an inspection system and method that can accurately correct the positional deviation between the position of each element of the work in the before-work image and the position of each corresponding element of the work in the after-work image, even if the deformations in the before-work image and the after-work image of the work are different from each other, and can accurately determine whether the work on the work was successful or not. [Means for solving the problem]

[0013] Such objectives are as follows: 1 This is achieved by the present invention. (1) a robot for performing an operation on a workpiece; A camera capable of photographing the workpiece; a control device for controlling the robot and the camera; The control device includes: Before driving the robot to perform the operation on the workpiece, the camera is driven to acquire a pre-operation image of the workpiece, and further, after driving the robot to perform the operation on the workpiece, the camera is driven to acquire a post-operation image of the workpiece; identifying a plurality of feature points in the before-work image and a plurality of feature points in the after-work image that correspond to the plurality of feature points in the before-work image, and performing feature point matching to obtain a list of a plurality of feature point pairs each composed of the feature points in the before-work image and the feature points in the after-work image that correspond to each other; performing an image correction process on at least one of the before-operation image and the after-operation image such that the positions of the feature points in the before-operation image of each of the plurality of feature point pairs included in the list coincide with the positions of the corresponding feature points in the after-operation image; After performing the image correction process, a difference image is obtained by extracting a difference between the before-operation image and the after-operation image; and and determining whether the operation on the workpiece has been completed or not based on the difference image. And, the robot includes a working unit for performing the task on the workpiece, and a movable unit for changing a relative position between the workpiece and the working unit, When the relative position between the workpiece and the working unit is changed by the movable unit of the robot in order to perform the task on the workpiece, the relative position between the camera and the workpiece is changed; When the pre-operation image is captured by the camera, the working unit and the camera are located at an initial position; When the operation is performed on the workpiece, the working unit and the camera are moved from the initial position; When the post-operation image is captured by the camera, the working unit and the camera are returned to the initial position. 1. An inspection system comprising:

[0014] (2) The inspection system according to (1) above, wherein the image correction processing performed on at least one of the pre-operation image and the post-operation image is an affine transformation or a homography transformation.

[0015] (3) The control device is configured to binarize the difference image and determine whether the operation on the workpiece was successful or not based on the sum of pixel values ​​of the binarized difference image.

[0016] (4) An inspection system described in any of (1) to (3) above, wherein the control device is configured to, after performing the image correction process, cut out inspection areas of each of the pre-work image and the post-work image, and obtain the difference image by extracting the difference between the inspection area of ​​the pre-work image and the inspection area of ​​the post-work image.

[0017] (5) An inspection system described in any of (1) to (4) above, wherein the control device is configured to perform an exclusion process to exclude unreliable feature point pairs from the list after performing the feature point matching.

[0018] (6) The inspection system described in (5) above, wherein the exclusion process identifies low reliability feature points pairs based on an angle difference between an angle between the multiple feature points of the multiple feature points pairs included in the list in the pre-operation image and an angle between the multiple feature points of the multiple feature points pairs included in the list in the post-operation image, and excludes the low reliability feature points pairs from the list.

[0019] (7) An inspection system as described in (5) or (6), wherein the exclusion process calculates a Euclidean distance between the feature points in the pre-operation image and the feature points in the post-operation image for each of the plurality of feature points pairs included in the list, and further identifies the feature points pairs with low reliability based on the calculated Euclidean distance, and excludes the feature points pairs with low reliability from the list.

[0020] (8) An inspection system as described in any of (5) to (7) above, wherein the exclusion process calculates an angle between the feature point in the before-work image and the feature point in the after-work image for each of the multiple feature point pairs included in the list, and further identifies the feature point pairs with low reliability based on the calculated angle, and excludes the feature point pairs with low reliability from the list.

[0022] ( 9 The robot further includes a work table on which the workpiece is placed, When a working surface of the work table is an XY plane, the movable part includes an X-axis movable part that is movable in an X-axis direction, The working unit and the camera are movable in the X-axis direction by the X-axis movable unit. 1 ) An inspection system as described in the above.

[0023] ( 10 ) the movable portion further includes a Z-axis movable portion movable in the Z-axis direction, the Z-axis movable part is attached to the X-axis movable part, The working unit and the camera are attached to the Z-axis movable unit, and are movable in the X-axis direction and the Z-axis direction. 9 ) An inspection system as described in the above.

[0024] ( 11 ) An inspection method for determining whether a robot has performed a task on a workpiece using a camera, comprising: before driving the robot to perform the operation on the workpiece, driving the camera to acquire a pre-operation image of the workpiece, and further, after driving the robot to perform the operation on the workpiece, driving the camera to acquire a post-operation image of the workpiece; performing feature point matching to identify a plurality of feature points in the before-work image and a plurality of feature points in the after-work image that correspond to the plurality of feature points in the before-work image, and to obtain a list of a plurality of feature point pairs each composed of the feature points in the before-work image and the feature points in the after-work image that correspond to each other; performing an image correction process on at least one of the before-operation image and the after-operation image such that the positions of the feature points in the before-operation image of each of the plurality of feature point pairs included in the list coincide with the positions of the corresponding feature points in the after-operation image; obtaining a difference image by extracting a difference between the before-operation image and the after-operation image; and determining whether the operation on the workpiece has been completed or not based on the difference image. fruit, the robot includes a working unit for performing the task on the workpiece, and a movable unit for changing a relative position between the workpiece and the working unit, When the relative position between the workpiece and the working unit is changed by the movable unit of the robot in order to perform the task on the workpiece, the relative position between the camera and the workpiece is changed; When the pre-operation image is captured by the camera, the working unit and the camera are located at an initial position; When the operation is performed on the workpiece, the working unit and the camera are moved from the initial position; When the post-operation image is captured by the camera, the working unit and the camera are returned to the initial position. 13. An inspection method comprising: Effect of the Invention

[0025] According to the present invention, even if the deformations in the pre-work image and post-work image of a workpiece are different from each other, the positional deviation between the position of each element of the workpiece in the pre-work image and the position of each corresponding element of the workpiece in the post-work image can be accurately corrected, and the success or failure of the work on the workpiece can be accurately determined.

[0026] Furthermore, in the present invention, after feature point matching is performed to obtain a list of feature point pairs, an exclusion process is performed to exclude feature point pairs with low reliability from the obtained list of feature point pairs. This makes it possible to more accurately correct positional deviations between the positions of each element of the workpiece in the before-operation image and the positions of the corresponding elements of the workpiece in the after-operation image, and to accurately determine whether the operation on the workpiece has been successful. [Brief description of the drawings]

[0027] [Figure 1] 1A to 1C are diagrams for explaining deformation of a camera image that changes depending on the attitude and direction of the camera relative to a subject. [Diagram 2] 1 is a block diagram illustrating an inspection system according to the present invention. [Diagram 3] FIG. 3 is a perspective view showing the robot, the camera, and the workpiece shown in FIG. 2. [Figure 4] FIG. 4 is an enlarged view of the area around the camera in FIG. [Diagram 5] FIG. 13 is a diagram showing an example of a pre-operation image of a workpiece. [Figure 6] FIG. 3 is a block diagram illustrating a control device shown in FIG. 2. [Figure 7] FIG. 13 is a diagram for explaining an exclusion process for excluding low-reliability feature points pairs from a list of feature points pairs. [Figure 8] FIG. 13 is a diagram for explaining another exclusion process for excluding low-reliability feature points pairs from the list of feature points pairs. [Figure 9] 11A and 11B are diagrams for explaining a process for obtaining a difference image from a pre-operation image and a post-operation image. [Figure 10]2 is a flowchart showing an inspection method of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] The inspection system and inspection method of the present invention will be described below based on the preferred embodiment shown in the accompanying drawings. Note that each of the drawings referred to below is a schematic diagram prepared for the purpose of explaining the present invention. The dimensions (length, width, thickness, etc.) of each component shown in the drawings do not necessarily reflect the actual dimensions. In addition, the same reference numbers are used for the same or corresponding components in each drawing. In the following description, the positive direction of the Z axis in each drawing may be referred to as the "upper side" and the negative direction of the Z axis may be referred to as the "lower side."

[0029] First, the inspection system of the present invention will be described in detail with reference to Figs. 2 to 9. Fig. 2 is a block diagram that shows the inspection system of the present invention in schematic form. Fig. 3 is a perspective view showing the robot, camera, and workpiece shown in Fig. 2. Fig. 4 is an enlarged view of the peripheral area of ​​the camera in Fig. 3. Fig. 5 is a diagram showing an example of a pre-operation image of the workpiece. Fig. 6 is a block diagram that shows the control device shown in Fig. 2 in schematic form. Fig. 7 is a diagram for explaining an exclusion process for excluding feature points pairs with low reliability from a list of feature points pairs. Fig. 8 is a diagram for explaining another exclusion process for excluding feature points pairs with low reliability from a list of feature points pairs. Fig. 9 is a diagram for explaining a process for obtaining a difference image from a pre-operation image and a post-operation image.

[0030] <Inspection system 100> 2 is used to determine whether or not an operation performed on a workpiece 3 by a robot 1 has been successful. The inspection system 100 includes the robot 1, a camera 2, the workpiece 3 placed on a worktable 11 of the robot 1, and a control device 4 for controlling the robot 1 and the camera 2.

[0031] The robot 1 has a function of performing predetermined operations such as attaching parts and processing such as coating and cutting on a workpiece 3 placed on a worktable 11. Typically, the robot 1 is an industrial robot that performs precision operations on the workpiece 3 (e.g., a circuit board or precision component) to obtain a product or work in progress. The robot 1 includes the worktable 11, a working unit 12 for performing operations on the workpiece 3 placed on the worktable 11, and a movable unit 13 for changing the relative position of the workpiece 3 placed on the worktable 11 and the working unit 12.

[0032] As shown in Fig. 3, the work table 11 is a flat member on which the work 3 to be operated by the working unit 12 is placed. The working unit 12 is used to perform a predetermined operation on the work 3 placed on the work table 11. Various tools can be used as the working unit 12 depending on the content of the operation to be performed on the work 3. For example, when applying an adhesive such as a thermosetting resin to the work 3, a nozzle can be used as the working unit 12, and when assembling a part on the work 3, a pick-up hand can be used as the working unit 12.

[0033] The working unit 12 is connected to the control device 4 by a wiring not shown, and executes a predetermined operation in response to the control from the control device 4. For example, when the working unit 12 is a nozzle, the working unit 12 executes an operation of applying a liquid material (e.g., an adhesive such as a thermosetting resin) filled inside the nozzle to the workpiece 3 in response to the control from the control device 4. Hereinafter, the working unit 12 is described as a nozzle for applying the liquid material to one or more work points 32 (work target points, see FIG. 5) of the workpiece 3 in response to the control from the control device 4, but it should be noted that the working unit 12 is not limited to a nozzle as described above. After the movable part 13 is driven in response to the control from the control device 4 and the relative position between the working unit 12 and the workpiece 3 placed on the work table 11 is adjusted, the control device 4 transmits a control signal to the working unit 12, and the working unit 12 discharges the liquid material filled inside itself to one of the work points 32 of the workpiece 3.

[0034] The movable part 13 comprises a base 131 which functions as the foundation of the robot 1, an X-axis movable part 13X which is arranged above the base 131 and spaced apart from the upper surface of the base 131, a Y-axis movable part 13Y which is arranged on the upper surface of the base 131, and a Z-axis movable part 13Z which is attached to the X-axis movable part 13X.

[0035] The X-axis movable part 13X is connected to the control device 4 by wiring (not shown) and has a function of sliding the X-axis stage 133X along the X-axis direction in Fig. 3 according to the control from the control device 4. The X-axis movable part 13X includes a pair of legs 131X provided to stand on both ends of the base 131 in the X-axis direction, an X-axis rail 132X provided to bridge the pair of legs 131X apart from the upper surface of the base 131 and extending along the X-axis in the figure, a box-shaped X-axis stage 133X provided to be slidable on the X-axis rail 132X, and an X-axis actuator (not shown) such as a stepping motor for sliding the X-axis stage 133X. When a control signal is transmitted from the control device 4 to the X-axis actuator, the X-axis actuator is driven according to the control signal to slide the X-axis stage 133X on the X-axis rail 132X.

[0036] The Y-axis movable part 13Y is connected to the control device 4 by wiring (not shown), and has a function of sliding the Y-axis stage 132Y along the Y-axis direction in FIG. 3 in response to control from the control device 4. The Y-axis movable part 13Y is provided on the upper surface of the base 131, and includes a Y-axis rail 131Y extending along the Y-axis in the figure, a flat Y-axis stage 132Y slidably provided on the Y-axis rail 131Y, and a Y-axis actuator (not shown) such as a stepping motor for sliding the Y-axis stage 132Y. When a control signal is transmitted from the control device 4 to the Y-axis actuator, the Y-axis actuator is driven in response to the control signal, and the Y-axis stage 132Y is slidably moved on the Y-axis rail 131Y. As shown in FIG. 3, the surface of the work table 11 and the surface of the Y-axis stage 132Y coincide with the XY plane in the figure. The work table 11 of the robot 1 is fixed on the Y-axis stage 132Y.

[0037] The Z-axis movable part 13Z is connected to the control device 4 by wiring (not shown), and has a function of sliding the Z-axis stage 131Z along the Z-axis direction in FIG. 3 according to the control from the control device 4. The Z-axis movable part 13Z includes a flat Z-axis stage 131Z provided on the X-axis stage 133X of the X-axis movable part 13X, and a Z-axis actuator (not shown) such as a stepping motor for sliding the Z-axis stage 131Z. When a control signal is transmitted from the control device 4 to the Z-axis actuator, the Z-axis actuator is driven according to the control signal, and slides the Z-axis stage 131Z along the Z-axis direction on the X-axis stage 133X. As shown in FIG. 3, the surface direction of the Z-axis stage 131Z coincides with the XZ plane in the figure. The working part 12 and the camera 2 are fixed on the surface of the Z-axis stage 131Z by a fixture.

[0038] In the illustrated embodiment, the work table 11 is mounted on the Y-axis stage 132Y of the Y-axis movable part 13Y. Meanwhile, the Z-axis stage 131Z of the Z-axis movable part 13Z is mounted on the X-axis stage 133X of the X-axis movable part 13X, and further, the working part 12 and the camera 2 are mounted on the Z-axis stage 131Z. Therefore, the control device 4 can change the relative position between the working part 12 and the workpiece 3 placed on the work table 11 by driving the X-axis movable part 13X, the Y-axis movable part 13Y, and the Z-axis movable part 13Z of the movable part 13. The positions of the working part 12 and the camera 2 in the initial state in which the X-axis movable part 13X, the Y-axis movable part 13Y, and the Z-axis movable part 13Z of the robot 1 are not driven by the control device 4 are referred to as the initial positions.

[0039] The camera 2 is an arbitrary imaging device provided above the work table 11 so as to be able to photograph the work 3 placed on the work table 11. As shown in FIG. 3, the camera 2 is attached to the Z-axis stage 131Z of the Z-axis movable part 13Z of the movable part 13. The camera 2 is connected to the control device 4 by wiring (not shown), and acquires a camera image including the working part 12 and the work table 11 in real time and transmits the acquired camera image to the control device 4. The control device 4 can check the relative position between the working part 12 and the work 3 placed on the work table 11 in real time by referring to the camera image transmitted from the camera 2. Therefore, the camera 2 functions as a visual sensor for observing the relative position between the working part 12 and the work 3 placed on the work table 11.

[0040] FIG. 4 shows an enlarged view of the peripheral area of ​​the camera 2 in FIG. 3. In FIG. 4, the dotted lines extending radially downward from the tip of the camera 2 represent the field of view of the camera 2. As shown in FIG. 4, the camera 2 is attached to the Z-axis stage 131Z in such an orientation and position that the tip 121 of the working unit 12 that performs the work on the workpiece 3 is always located within the field of view of the camera 2. In this way, since both the camera 2 and the working unit 12 are attached to the Z-axis stage 131Z, the relative position between the camera 2 and the working unit 12 is always constant. In addition, the tip 121 of the working unit 12 is the nozzle tip that ejects droplets when the working unit 12 is a nozzle, and is the hand tip that grips the workpiece 3 when the working unit 12 is a pickup hand.

[0041] With this configuration, the camera image captured in real time by the camera 2 when the robot 1 performs an operation on the workpiece 3 always includes the tip 121 of the working unit 12 and the workpiece 3 on the worktable 11. Therefore, by referring to the camera image, the control device 4 can directly confirm the relative position between the working unit 12 and the workpiece 3 placed on the worktable 11 in real time, and can more accurately control the relative position between the working unit 12 and the workpiece 3 placed on the worktable 11.

[0042] As described later, in the inspection system 100 of the present invention, the camera 2 is controlled by the control device 4 to photograph the workpiece 3 before the robot 1 performs the operation on the workpiece 3, thereby obtaining a pre-operation image of the workpiece 3, and to photograph the workpiece 3 after the robot 1 performs the operation on the workpiece 3, thereby obtaining a post-operation image of the workpiece 3. The pre-operation image and post-operation image of the workpiece 3 obtained by the camera 2 are used for inspection to determine whether the robot 1 has performed the operation on the workpiece 3 successfully or not.

[0043] In addition, since the camera 2 is attached to the Z-axis stage 131Z of the Z-axis movable part 13Z of the movable part 13, when the control device 4 drives the X-axis movable part 13X and the Z-axis movable part 13Z of the movable part 13 to change the relative position between the workpiece 3 and the working part 12 in order to perform the work on the workpiece 3 by the working part 12, the relative position between the camera 2 and the workpiece 3 also changes. Thus, in the inspection system 100 of the present invention, the posture and orientation of the camera 2 with respect to the workpiece 3 placed on the work table 11 are not constant. Therefore, even if the control device 4 drives the movable part 13 of the robot 1 and returns the working part 12 and the camera 2 to their initial positions after the robot 1 performs the work on the workpiece 3, the posture and orientation of the camera 2 with respect to the workpiece 3 will be different from those before the working part 12 performs the work on the workpiece 3 due to multiple factors such as the control resolution of the movable part 13 and the positional deviation and inclination of the workpiece 3 during the work. Therefore, the deformation of the pre-work image of the workpiece 3 acquired before the robot 1 performs the work on the workpiece 3 differs from the deformation of the post-work image of the workpiece 3 acquired after the robot 1 performs the work on the workpiece 3. As will be described later, the inspection system 100 of the present invention is configured to be able to accurately align the pre-work image and the post-work image of the workpiece 3 even if the deformations of the pre-work image and the post-work image of the workpiece 3 differ from each other.

[0044] Fig. 5 shows an example of a pre-operation image of the workpiece 3. In the example shown in Fig. 5, the workpiece 3 is a precision part to which a liquid material such as adhesive is applied by the working unit 12 of the robot 1. The workpiece 3 includes a plurality of reference markers 31 (four in the illustrated example) that can be used as feature points for performing image correction processing for aligning the pre-operation image and the post-operation image, and one or more work points 32 to which the liquid material such as adhesive is applied.

[0045] The multiple reference markers 31 are portions that can be used as feature points for aligning the pre-work image and the post-work image in an image correction process performed on at least one of the pre-work image and the post-work image in order to align the pre-work image and the post-work image. In the illustrated embodiment, each of the multiple reference markers 31 is a black point formed on the upper surface of the workpiece 3 so as to be displayed with high contrast in the pre-work image and the post-work image acquired by the camera 2. As described later, in order to perform the image correction process performed for aligning the pre-work image and the post-work image, it is necessary to identify multiple feature points in the pre-work image and multiple corresponding feature points in the post-work image. Typically, the edge parts of the workpiece 3, particularly the edge parts at the four corners, are used as feature points in the pre-work image and the post-work image, but depending on the shooting conditions of the camera 2, there may be cases where the edge parts of the workpiece 3 are not clearly displayed in the pre-work image and the post-work image. In such a case, it is preferable that the multiple reference markers 31 displayed with high contrast in the pre-work image and the post-work image are used as feature points in the pre-work image and the post-work image. As described below, in the inspection system 100 of the present invention, an image correction process is performed on at least one of the pre-work image and the post-work image so that the positions of multiple feature points in the pre-work image match the positions of corresponding multiple feature points in the post-work image.

[0046] When the deformations of the pre-work image and the post-work image of the workpiece 3 are different from each other, performing position correction in the XY direction on at least one of the pre-work image and the post-work image of the workpiece 3 as described in the section on the prior art can match the position of any one feature point in the pre-work image with the position of a corresponding one feature point in the post-work image, but cannot match the positions of other feature points in the pre-work image with the positions of corresponding feature points in the post-work image. In the inspection system 100 of the present invention, even if the deformations of the pre-work image and the post-work image of the workpiece 3 are different from each other, an image correction process is performed on at least one of the pre-work image and the post-work image so that the positions of the feature points in the pre-work image and the positions of the corresponding feature points in the post-work image match each other.

[0047] One or more work points 32 are locations where work should be performed by the working unit 12 of the robot 1. In the illustrated example, a predetermined amount of liquid material is applied to each of the six work points 32 by the working unit 12. The number and positions of the work points 32 are set in advance, and the control device 4 can identify the positions of each of the multiple work points 32 by referring to the camera image acquired by the camera 2. For example, the control device 4 can identify the position of any reference marker 31 in the camera image using any object recognition technology such as a known pattern recognition algorithm, and identify an area that has a predetermined positional relationship with the identified reference marker 31 as the work point 32.

[0048] 6 shows a block diagram of the control device 4. The control device 4 is connected to each component of the robot 1 (specifically, the working unit 12 and the movable unit 13) and the camera 2. The control device 4 has a function of controlling each component of the robot 1 and the camera 2. The control device 4 may be implemented as a standalone device, or may be implemented in any computing device such as a desktop computer, a laptop computer, a notebook computer, a workstation, a tablet computer, a mobile phone, a smartphone, a PDA, a wearable terminal, a server, or the like.

[0049] As shown in FIG. 6, the control device 4 includes one or more processors 41 for executing the operations of the control device 4, an I / O (input / output) interface 42 for executing input to and output from the control device 4, and one or more memories 43 that store data 44 and modules 45 used to execute the processing of the control device 4.

[0050] The one or more processors 41 are computing units that perform computations, such as signal manipulation, based on computer-readable instructions, such as one or more microprocessors, microcomputers, microcontrollers, digital signal processors (DSPs), central processing units (CPUs), memory control units (MCUs), graphics processing units (GPUs), state machines, logic circuits, application specific integrated circuits (ASICs), or combinations thereof. In particular, the processor 41 is configured to fetch computer-readable instructions (e.g., data, programs, modules, etc.) stored in the memory 43 and perform computations, signal manipulations, and control.

[0051] The I / O interface 42 includes various software interfaces and hardware interfaces, such as a web interface and a graphical user interface (GUI). For example, the I / O interface 42 is an interface for peripheral devices, such as a keyboard, a mouse, a touch panel display, an external memory, a printer, and a display. The I / O interface 42 enables input to the control device 4 using an input device, such as a keyboard, a mouse, and a touch panel display, and output from the control device 4 to an external device, such as a display, a printer, and an external memory. The I / O interface 42 may also enable the control device 4 to communicate with any external device, such as an external web server or data server, via a network, such as the Internet.

[0052] The memory 43 is a computer-readable medium including a volatile storage medium (e.g., RAM, SRAM, DRAM), a non-volatile storage medium (e.g., ROM, EPROM, EEPROM, flash memory, a hard disk, a solid-state drive, an SD card, an optical disk, a CD-ROM, a digital versatile disk (DVD), a Blu-ray disk, a magnetic cassette, a magnetic tape, a magnetic disk), or a combination thereof.

[0053] The memory 43 is communicatively connected to the processor 41, and stores a plurality of modules 45 executable by the processor 41, and data 44 required for processing by the plurality of modules 45. The memory 43 also has a function of temporarily storing data received, processed, or generated by one or more of the plurality of modules 45, and data required for executing processing by the plurality of modules 45.

[0054] The data 44 includes inspection area information 441 for extracting inspection areas required to determine whether the robot 1 has performed the work on the work 3 from each of the pre-work image and post-work image of the work 3, success / failure determination conditions 442 used to determine whether the work on the work 3 has been successful based on a differential image obtained by extracting the difference between the pre-work image and post-work image of the work 3, and any number of other data 443 required to execute the processing of the control device 4.

[0055] The inspection area information 441 is information for specifying the inspection area to be cut out from each of the before-operation image and the after-operation image of the workpiece 3. Specifically, the inspection area information 441 includes the start point coordinates (x1, y1) of the inspection area, and the widths in the x and y directions (x_length, y_length) of the inspection area or the end point coordinates (x2, y2) of the inspection area. The control device 4 cuts out an area specified by the start point coordinates (x1, y1), the widths in the x and y directions (x_length, y_length) or the end point coordinates (x2, y2) from the before-operation image of the workpiece 3 in accordance with the inspection area information 441, and acquires the inspection area of ​​the before-operation image. Similarly, the control device 4 cuts out an area specified by the start point coordinates (x1, y1), the widths in the x and y directions (x_length, y_length) or the end point coordinates (x2, y2) from the after-operation image of the workpiece 3 in accordance with the inspection area information 441, and acquires the inspection area of ​​the after-operation image.

[0056] The inspection area information 441 can be appropriately set by a user or administrator of the inspection system 100. For example, the user or administrator of the inspection system 100 may set the inspection area information 441 so that all work points 32 of the work 3 are included in the inspection area of ​​the before-work image and the after-work image, or may set the inspection area information 441 so that any one or more work points 32 of the work 3 are included in the inspection area of ​​the before-work image and the after-work image. When multiple work points 32 of the work 3 are included in the inspection area of ​​the before-work image and the after-work image, the inspection system 100 does not individually determine whether the work for each work point 32 was successful, but determines whether the work for each of the multiple work points 32 included in the inspection area was successful as a whole. On the other hand, when one work point 32 of the work 3 is included in the inspection area of ​​the before-work image and the after-work image, the inspection system 100 determines whether the work for the one work point 32 was successful.

[0057] The success / failure determination condition 442 is a condition used to determine whether the robot 1 has performed an operation on the workpiece 3, based on a difference image obtained by extracting the difference between the pre-operation image and the post-operation image. As described below, when determining whether the robot 1 has performed an operation on the workpiece 3, the difference image is binarized. Therefore, in the binarized difference image, the pixel values ​​of the different parts between the pre-operation image and the post-operation image are 1 (white), and the pixel values ​​of the common parts between the pre-operation image and the post-operation image are 0 (black). Therefore, the white parts (parts with a pixel value of 1) of the binarized difference image are the different parts between the pre-operation image and the post-operation image.

[0058] When performing work such as applying liquid material or attaching parts to one or more work points 32 of the work 3, the area of ​​the difference between the before-work image and the after-work image in the inspection area specified by the inspection area information 441 can be used as a criterion for judging whether the work on the work 3 by the robot 1 has been successful. For example, if the area of ​​the difference between the before-work image and the after-work image is smaller than a predetermined lower threshold, an error has occurred, such as no liquid material has been applied to the work point 32 of the work 3, the amount of liquid material applied is small, or a part has not been attached. On the other hand, if the area of ​​the difference between the before-work image and the after-work image is larger than a predetermined upper threshold, an error has occurred, such as more liquid material has been applied to the work point 32 of the work 3 than necessary, or a different part has been attached.

[0059] Since the area of ​​the difference between the before-operation image and the after-operation image is equal to the sum of the pixel values ​​of the binarized difference image (the total number of pixels in the white part), the success or failure of the operation of the robot 1 on the workpiece 3 can be determined based on whether the sum of the pixel values ​​of the binarized difference image is within a range defined by a predetermined lower limit threshold value and an upper limit threshold value. Therefore, the success or failure determination condition 442 is the lower limit threshold value and the upper limit threshold value of the sum of the pixel values ​​of the binarized difference image. The success or failure determination condition 442 is appropriately set by the user or administrator of the inspection system 100 depending on the content of the operation on the workpiece 3 by the robot 1, the size of the inspection area, etc.

[0060] Modules 45 are computer-readable instructions executable by processor 41, such as routines, applications, programs, algorithms, libraries, objects, components, data structures, or combinations thereof.

[0061] The module 45 includes a camera control module 451 for controlling the camera 2, a robot control module 452 for controlling the robot 1, a feature point matching module 453 for identifying a plurality of feature points in a before-operation image of the workpiece 3 and a plurality of feature points in a after-operation image of the workpiece 3 that correspond to the plurality of feature points in the before-operation image of the workpiece 3, and performing feature point matching to obtain a list of a plurality of feature point pairs each composed of the feature points in the before-operation image and the feature points in the after-operation image that correspond to each other, and a feature point matching module 454 for performing an exclusion process to exclude feature point pairs with low reliability from the list of feature point pairs. the control device 4 includes an exclusion module 454 for detecting the presence or absence of an object to be worked on by the robot 1, an image correction module 455 for performing image correction processing on at least one of the before-work image and the after-work image of the workpiece 3 in order to align the before-work image and the after-work image of the workpiece 3, a difference image creation module 456 for obtaining a difference image by extracting the difference between the before-work image and the after-work image of the workpiece 3 after performing the image correction processing, an inspection module 457 for determining whether the work performed by the robot 1 on the workpiece 3 was successful based on the difference image, and any number of other modules 458 for supplementing the functions provided by the control device 4.

[0062] The camera control module 451 has a function of driving the camera 2 and acquiring camera images. The camera images acquired and corrected by the camera control module 451 are temporarily stored in the memory 43 and used by each module of the control device 4. In particular, the camera control module 451 is used by the control device 4 to acquire a pre-operation image of the workpiece 3 by photographing the workpiece 3 before the robot 1 executes an operation on the workpiece 3, and to acquire a post-operation image of the workpiece 3 by photographing the workpiece 3 after the robot 1 executes an operation on the workpiece 3.

[0063] The robot control module 452 has a function of controlling the robot 1. More specifically, the robot control module 452 has a function of driving the movable part 13 of the robot 1 to change the relative position between the working part 12 of the robot 1 and the workpiece 3 on the worktable 11, and a function of driving the working part 12 to execute a predetermined task on the workpiece 3.

[0064] The robot control module 452 can independently drive the X-axis movable part 13X, the Y-axis movable part 13Y, and the Z-axis movable part 13Z of the movable part 13. The robot control module 452 refers to the camera image acquired by the camera 2 and identifies the positions of the multiple work points 32 of the workpiece 3. Thereafter, the robot control module 452 determines the target amount of drive of each of the X-axis movable part 13X, the Y-axis movable part 13Y, and the Z-axis movable part 13Z of the movable part 13 in order to change the relative position between the working part 12 of the robot 1 and any one of the work points 32 of the workpiece 3 on the worktable 11, and determines the output value for each of the X-axis movable part 13X, the Y-axis movable part 13Y, and the Z-axis movable part 13Z. Thereafter, the robot control module 452 outputs the determined output values ​​to the X-axis movable part 13X and the Y-axis movable part 13Y of the movable part 13, drives the X-axis movable part 13X and the Y-axis movable part 13Y of the movable part 13 independently, and sets the relative position between the working part 12 and the workpiece 3 to a predetermined state appropriate for performing work on one work point 32 of the workpiece 3.

[0065] More specifically, the robot control module 452 first drives the X-axis movable part 13X and the Y-axis movable part 13Y so that the relative position between the tip 121 of the working unit 12 and one work point 32 of the workpiece 3 satisfies the approach condition. The "approach condition" here is a condition for determining whether the relative position between the tip 121 of the working unit 12 and one work point 32 of the workpiece 3 on the XY plane is in a suitable condition for the robot control module 452 to drive the Z-axis movable part 13Z and cause the tip 121 of the working unit 12 to approach one work point 32 of the workpiece 3. For example, the approach condition is the distance between the tip 121 of the working unit 12 and one work point 32 of the workpiece 3 in the camera image acquired by the camera 2, the angle of the tip 121 of the working unit 12 with respect to one work point 32 of the workpiece 3, etc. The approach conditions are set in advance according to the shape of the working unit 12, the type of work performed by the working unit 12, and the shape and size of the workpiece 3. In this manner, the robot control module 452 drives the X-axis movable unit 13X and the Y-axis movable unit 13Y so that the relative position between the tip 121 of the working unit 12 and one work point 32 of the workpiece 3 satisfies the approach conditions.

[0066] Thereafter, the robot control module 452 drives the Z-axis movable part 13Z to make the tip 121 of the working part 12 approach one of the work points 32 of the workpiece 3, and makes the tip 121 of the working part 12 contact or approach one of the work points 32 of the workpiece 3. Thereafter, the robot control module 452 drives the working part 12 to perform a predetermined operation on one of the work points 32 of the workpiece 3. After the predetermined operation on one of the work points 32 of the workpiece 3 is completed, the robot control module 452 drives the movable part 13 and the working part 12 to perform an operation on another work point 32 of the workpiece 3. When the operation on all the work points 32 of the workpiece 3 by the robot 1 is completed, the robot control module 452 returns the working part 12 to its initial position. At this time, the camera 2 attached to the Z-axis movable part 13Z of the movable part 13 of the robot 1 is also returned to its initial position. In this state, the camera control module 451 drives the camera 2 to acquire an image of the workpiece 3 after the operation.

[0067] The feature point matching module 453 has a function of performing feature point matching to identify a plurality of feature points in the pre-operation image of the work 3 and a plurality of feature points in the post-operation image of the work 3 that correspond to the plurality of feature points in the pre-operation image of the work 3, and to obtain a list of a plurality of feature point pairs each composed of a feature point in the pre-operation image and a feature point in the post-operation image that correspond to each other. The feature point pair here is a pair composed of any one feature point in the pre-operation image and a corresponding feature point in the post-operation image. The feature point matching module 453 can identify a plurality of feature points in the pre-operation image of the work 3 and a plurality of feature points in the post-operation image of the work 3 that correspond to the plurality of feature points in the pre-operation image of the work 3, respectively, using any feature point matching algorithm known in the field of feature point matching, for example, SIFT, SURF, AKAZE, ORB, etc., to obtain a list of a plurality of feature point pairs.

[0068] Typically, the feature point matching module 453 first detects a plurality of feature points in the pre-operation image of the workpiece 3 and a plurality of feature points in the post-operation image of the workpiece 3 using any edge extraction method such as Canny edge detector. At this time, it is preferable that the edge portions of the four corners of the workpiece 3 and the plurality of reference markers 31 in each of the pre-operation image and the post-operation image are detected as feature points. Thereafter, the feature point matching module 453 calculates the similarity between any feature point in the pre-operation image and each of all feature points in the post-operation image, and identifies the feature point in the post-operation image having the highest similarity as the feature point in the post-operation image corresponding to the any feature point in the pre-operation image. Next, the feature point matching module 453 associates any feature point in the pre-operation image with the corresponding feature point in the identified post-operation image to form a set of feature point pairs. This construction of feature point pairs is performed for all feature points in the pre-operation image to obtain a first tentative list of feature point pairs. Obtaining such a list of feature point pairs is called brute force matching in the field of feature point matching, and is performed using an arbitrary similarity measure calculated by a known method such as the Hamming distance.

[0069] Next, the feature point matching module 453 performs the above-mentioned brute force matching performed on all feature points in the before-operation image on all feature points in the after-operation image to obtain a second tentative list of feature point pairs. Thereafter, the feature point matching module 453 compares the first tentative list of feature point pairs with the second tentative list of feature point pairs (such a comparison is called a "cross check" in the field of feature point matching), and creates a list of feature point pairs with only feature point pairs that match in the first tentative list of feature point pairs and the second tentative list of feature point pairs. The list of feature point pairs obtained by the feature point matching module 453 is temporarily stored in the memory 43.

[0070] The exclusion module 454 has a function of executing an exclusion process for excluding low-reliability feature points pairs from the list of multiple feature points pairs acquired by the feature point matching module 453. The exclusion process executed by the exclusion module 454 includes: (1) a first exclusion process of determining an unreliable feature points pair based on an angle difference between an angle between multiple feature points of multiple feature points pairs in the pre-operation image and an angle between multiple feature points of multiple feature points pairs in the post-operation image, and excluding the unreliable feature points pair from the list of feature points pairs; (2) a second exclusion process of calculating a Euclidean distance between a feature point in the pre-operation image and a feature point in the post-operation image, for each of the multiple feature points pairs included in the list of feature points pairs, and further determining an unreliable feature points pair based on the calculated Euclidean distance and excluding the unreliable feature points pair from the list of feature points pairs; and (3) a third exclusion process of calculating an angle between a feature point in the pre-operation image and a feature point in the post-operation image, for each of the multiple feature points pairs included in the list of feature points pairs, and further determining an unreliable feature points pair based on the calculated angle and excluding the unreliable feature points pair from the list of feature points pairs.

[0071] FIG. 7 is a diagram for explaining the first exclusion process. FIG. 8 is a diagram for explaining the second exclusion process and the third exclusion process. In FIG. 7 and FIG. 8, for ease of viewing, it is assumed that an M-shaped subject is included in each of the before-operation image and the after-operation image. When the inspection system 100 is actually operated, it should be noted that the workpiece 3 is included in each of the before-operation image and the after-operation image. When feature point matching is performed by the feature point matching module 453 on the before-operation image and the after-operation image as shown in FIG. 7, for example, a plurality of feature points FP as marked with circles in FIG. 7 and FIG. 8 are detected. In FIG. 7 and FIG. 8, the feature point FP 1n represents the feature points in the before-operation image that constitute the n-th feature point pair included in the list of feature point pairs obtained by the feature point matching module 453, and the feature point FP 2nrepresents the feature points in the post-operation image that constitute the n-th feature point pair included in the list of feature point pairs obtained by the feature point matching module 453. Therefore, the n-th feature point pair included in the list of feature point pairs is the feature points FP 1n and the feature point FP in the post-processing image 2n It consists of:

[0072] First, the first elimination process will be described in detail with reference to Fig. 7. In the first elimination process, each feature point FP 1n Other feature points of FP 1m The angle θ 1nm is calculated by the following formula (1), and each feature point FP 1n Other feature points of FP 1m Set of angles to Θ 1n =(θ 1n0 ,θ 1n1 ,θ 1n2 ,...,θ 1nm-1 ,θ 1nm ) is obtained.

[0073]

number

[0074] Next, for each feature point FP 2n Other feature points of FP 2m The angle θ 2nm is calculated by the following formula (2), and each feature point FP2n Other feature points of FP 2m Set of angles to Θ 2n =(θ 2n0 ,θ 2n1 ,θ 2n2 ,...,θ 2nm-1 ,θ 2nm ) is obtained.

[0075]

number

[0076] Next, according to the following formula (3), the feature points FP 1n Angle θ between 1nm and multiple feature points FP in the before image of multiple feature point pairs 1n Angle θ between 2nm The angle difference Δθ nm is calculated, and the angular difference Δθ nm Set ΔΘ n =(Δθ n0 ,Δθ n1 ,...,Δθ nm-1, Δθ nm ) and the angle difference Δθ between all feature point pairs nm Set ΔΘ=(ΔΘ0,ΔΘ1,...,ΔΘ n-1, ΔΘ n ) is obtained.

[0077]

number

[0078] Next, the angle difference Δθ nm Set ΔΘ n =(Δθ n0 ,Δθ n1 ,...,Δθ nm-1, Δθ nm The first quartile (25th percentile), the third quartile (75th percentile), and the interquartile range (=third quartile-first quartile) obtained by subtracting the first quartile from the third quartile are calculated and temporarily stored in the memory 43. Next, the angle difference Δθ nm Set ΔΘ n Angle difference Δθ less than (1st quartile - 1.5 × interquartile range) calculated from its first quartile, third quartile, and interquartile range nm and the angle difference Δθ exceeds (first quartile + 1.5 × interquartile range) nm Then, the angle difference Δθ of each feature point pair from which the outliers have been removed is calculated. nm Set ΔΘ n The median (50th percentile) of the feature point pair is calculated, and the angular difference Δθ nm Set ΔΘ n Each of (ΔΘ0,ΔΘ1,...,ΔΘ n-1, ΔΘ n The median value of each of the feature point pairs after removing the outliers is temporarily stored in the memory 43. nm Set ΔΘ n The median of the angle difference Δθ nm This is called the median.

[0079] Next, the angular difference Δθ of all feature point pairs that make up the set ΔΘ nm The first quartile, the third quartile, and the interquartile range obtained by subtracting the first quartile from the third quartile are calculated and temporarily stored in the memory 43. Then, from the set ΔΘ, an angle difference Δθ less than (first quartile-1.5×interquartile range) is calculated. nm and the angle difference Δθ exceeds (first quartile + 1.5 × interquartile range) nm Then, the angle difference Δθ of all the feature point pairs that compose the set ΔΘ from which the outliers are removed is calculated. nm Median and standard deviation σ θ is calculated and temporarily stored in the memory 43. Hereinafter, the angle difference Δθ of all the feature point pairs constituting the set ΔΘ from which the outliers have been removed is nm Median and standard deviation σ θ , and the angle difference Δθ between all feature point pairs. nm Median and standard deviation σ θ He said.

[0080] Feature points FP in the before image 1n and the feature point FP in the image after processing 2n If the matching is done correctly, even if the deformations of the before and after images are different, the feature points FP 1n Other feature points of FP 1m The angle to the target object and the feature point FP in the post-processing image 2n Other feature points of FP 2m The angle difference Δθ nm does not change significantly from a certain value. Therefore, the angle difference Δθ nm The median of the angle difference Δθ of all feature point pairs nm If the n-th feature pair is significantly different from the median of , it is highly likely that the n-th feature pair was obtained by an incorrect matching result, and it can be determined that the feature pair is of low reliability. For example, the angle difference Δθ of the first feature pair is 1m The median of the angle difference Δθ of all feature point pairs nmIf the first feature pair is significantly different from the median value of , the first feature pair can be determined to be a low-reliability feature pair. nm The median of and the angle difference Δθ of all feature point pairs nm By comparing the median of the matched pairs, it is possible to identify matched pairs with low reliability.

[0081] The first elimination process is to eliminate the angular difference Δθ nm The median of and the angle difference Δθ of all feature point pairs nm By comparing the median value of each feature point pair with the median value of , the matching pairs with low reliability are identified. nm The median of the angle difference Δθ of all feature point pairs nm Median ±Nσ θ (where N is an integer equal to or greater than 1 and is set appropriately as necessary) nm A feature points pair whose median value is outside the range is determined to be a low-reliability feature points pair and is excluded from the list of feature points pairs acquired by the feature point matching module 453. In this way, the first exclusion process excludes low-reliability feature points pairs from the list of feature points pairs acquired by the feature point matching module 453.

[0082] Next, the second exclusion process will be described in detail with reference to Fig. 8. In the second exclusion process, the feature points FP 1n and the feature point FP in the image after processing 2n The Euclidean distance d between n is calculated by the following formula (4), and the Euclidean distance d n A set D=(d0,d1,...,d n-1 ,d n ) is acquired and temporarily stored in the memory 43.

[0083]

number

[0084] Next, the Euclidean distance d n From the set D, the Euclidean distance d n The first quartile, the third quartile, and the interquartile range obtained by subtracting the first quartile from the third quartile are calculated and temporarily stored in the memory 43. Next, the Euclidean distance d n From the set D, the Euclidean distance d is less than (first quartile - 1.5 × interquartile range) n and the Euclidean distance d exceeding (first quartile + 1.5 × interquartile range) n Then, the Euclidean distance d n The median and standard deviation σ of the set D of d is calculated and temporarily stored in the memory 43.

[0085] Feature points FP in the before image 1n and the feature point FP in the image after processing 2n If the matching is done correctly, even if the deformations of the before and after images are different, the feature points FP 1n and the feature point FP in the post-processing image. 2n The Euclidean distance d between n does not change significantly from a certain value. Therefore, the Euclidean distance d n is the Euclidean distance d nIf the Euclidean distance d of each feature pair is significantly different from the median of the set D, then such a feature pair is likely to have been obtained by an incorrect matching result, and can be determined to be a low-reliability feature pair. Therefore, the Euclidean distance d n and the Euclidean distance d n By comparing the median of set D with the set D, it is possible to identify unreliable matching pairs.

[0086] The second elimination process is to calculate the Euclidean distance d n and the Euclidean distance d n By comparing the Euclidean distance d n is the Euclidean distance d n The median of the set D of ±Nσ d (where N is an integer equal to or greater than 1 and is set appropriately as necessary) is determined. n is excluded from the list of feature points pairs obtained by the feature point matching module 453 as a low-reliability feature points pair. In this way, the second exclusion process excludes low-reliability feature points pairs from the list of feature points pairs obtained by the feature point matching module 453.

[0087] Next, the third elimination process will be described in detail. In the third elimination process, the feature points FP 1n and the feature point FP in the image after processing 2n The angle φ between n is calculated by the following formula (5), and the Euclidean distance φ n The set Φ=(φ0,φ1,...,φ n-1 ,φ n ) is acquired and temporarily stored in the memory 43.

[0088]

number

[0089] Next, the angle φ of the feature point pair n From the set Φ, the angle φ of the feature point pair n The first quartile, the third quartile, and the interquartile range obtained by subtracting the first quartile from the third quartile are calculated and temporarily stored in the memory 43. Next, the angle φ n From the set Φ, the angle φ is less than (first quartile - 1.5 × interquartile range). n and the angle φ that exceeds (first quartile + 1.5 × interquartile range) n Then, the angle φ of the feature point pair from which the outlier was removed is calculated. n The median and standard deviation σ of the set Φ φ is calculated and temporarily stored in the memory 43.

[0090] Feature points FP in the before image 1n and the feature point FP in the image after processing 2n If the matching is done correctly, even if the deformations of the before and after images are different, the feature points FP 1n and the feature point FP in the post-processing image. 2n The angle φ between n does not change significantly from a certain value. Therefore, the angle φ n is the angle φ of the feature point pair. n If the angle φ of each feature point pair is significantly different from the median of the set Φ, then such a feature point pair is likely to have been obtained by an incorrect matching result, and can be determined to be a low-reliability feature point pair. n and the angle between the feature point pairs, φ n By comparing the median of the set Φ of , it is possible to identify unreliable matching pairs.

[0091] In the third elimination step, the angle φ n and the angle between the feature point pairs, φ n By comparing the median of the set Φ of each feature point pair with the median of the set Φ, we distinguish low-reliability matching pairs. n is the angle φ of the feature point pair. n The median of the set Φ of ±Nσ φ (where N is an integer equal to or greater than 1 and is appropriately set as necessary) is determined. n is excluded from the list of feature points pairs obtained by the feature point matching module 453 as a low-reliability feature points pair. In this way, the third exclusion process excludes low-reliability feature points pairs from the list of feature points pairs obtained by the feature point matching module 453.

[0092] The exclusion module 454 can exclude low-reliability feature pairs from the list of feature pairs obtained by the feature matching module 453 by performing at least one of the first exclusion process, the second exclusion process, and the third exclusion process described above.

[0093] The image correction module 455 has a function of performing image correction processing on at least one of the pre-operation image and the post-operation image of the work 3 in order to perform alignment between the pre-operation image and the post-operation image of the work 3. More specifically, the image correction module 455 performs affine transformation or homography transformation on at least one of the pre-operation image and the post-operation image of the work 3 so that the positions of the multiple feature points of the work 3 included in the pre-operation image of the work 3 of the feature point pair included in the list of feature point pairs acquired by the feature point matching module 453 and further excluded by the exclusion module 454 from which the matching pairs with low reliability are excluded, match with the positions of the multiple corresponding feature points of the work 3 included in the post-operation image of the work 3. By using the affine transformation and homography transformation, not only two-dimensional position correction of the image (i.e., position shift and size conversion on the XY plane) but also three-dimensional position correction of the image can be performed.

[0094] As described above, when the deformations of the pre-work image and the post-work image of the workpiece 3 are different from each other, the positions of the multiple feature points in the pre-work image cannot be matched with the positions of the multiple corresponding feature points in the post-work image simply by performing position correction in the XY direction on at least one of the pre-work image and the post-work image of the workpiece 3. On the other hand, even when the deformations of the pre-work image and the post-work image of the workpiece 3 are different from each other, the positions of the multiple feature points in the pre-work image can be matched with the positions of the multiple corresponding feature points in the post-work image by appropriately performing affine transformation or homography transformation on at least one of the pre-work image and the post-work image of the workpiece 3.

[0095] As is well known in the image processing field, the affine transformation of an image is a combination of linear transformation (rotation, scaling, shear transformation) and translation of an image, and can transform a rectangular image into an image of a parallelogram. In order to align the pre-operation image and the post-operation image of the workpiece 3, the image correction module 455 performs an affine transformation on at least one of the pre-operation image and the post-operation image of the workpiece 3 so that the positions of three or more feature points in the pre-operation image of the workpiece 3 coincide with the positions of the corresponding three or more feature points in the post-operation image of the workpiece 3. As is well known in the image processing field, the affine transformation is performed using a determinant that transforms the coordinates before the transformation into the coordinates after the affine transformation. In the inspection system 100 of the present invention, a determinant is calculated to match the positions of three or more feature points in the pre-operation image of the workpiece 3 with the positions of the corresponding three or more feature points in the post-operation image of the workpiece 3, and the affine transformation is performed on at least one of the pre-operation image and the post-operation image of the workpiece 3 using the determinant. If affine transformation is sufficient for image correction of at least one of the pre-work image and post-work image of the work 3 to align the pre-work image and post-work image of the work 3, the image correction module 455 corrects at least one of the pre-work image and post-work image of the work 3 using affine transformation.

[0096] On the other hand, the homography transformation of an image is an extension of the affine transformation, and in addition to the function of the affine transformation, it can perform a transformation from a rectangular image, which cannot be handled by the affine transformation, to a trapezoidal image. In order to align the pre-operation image and the post-operation image of the workpiece 3, the image correction module 455 performs a homography transformation on at least one of the pre-operation image and the post-operation image of the workpiece 3 so that the positions of four or more feature points in the pre-operation image of the workpiece 3 coincide with the positions of the corresponding four or more feature points in the post-operation image of the workpiece 3. As with the affine transformation, the homography transformation is performed using a determinant (homography matrix) that transforms the coordinates before the transformation into the coordinates after the homography transformation. In the inspection system 100 of the present invention, a determinant is calculated to match the positions of four or more feature points in the pre-operation image of the workpiece 3 with the positions of the corresponding four or more feature points in the post-operation image of the workpiece 3, and the homography transformation is performed on at least one of the pre-operation image and the post-operation image of the workpiece 3 using the determinant. When the affine transformation is insufficient for image correction of at least one of the pre-operation image and the post-operation image of the workpiece 3, the image correction module 455 uses a homography transformation to correct at least one of the pre-operation image and the post-operation image of the workpiece 3. Whether the image correction module 455 performs an affine transformation or a homography transformation on at least one of the pre-operation image and the post-operation image of the workpiece 3 is appropriately set in advance by a user or administrator of the inspection system 100.

[0097] The image correction module 455 uses the affine transformation or homography transformation as described above to correct at least one of the pre-operation image and the post-operation image of the work 3 so that the positions of multiple feature points of the work 3 included in the pre-operation image of the work 3 coincide with the positions of multiple corresponding feature points of the work 3 included in the post-operation image of the work 3. The pre-operation image and the post-operation image of the work 3 aligned by the image correction process of the image correction module 455 on at least one of the pre-operation image and the post-operation image of the work 3 are temporarily stored in the memory 43 and used to create a difference image.

[0098] The difference image creation module 456 has a function of extracting the difference between the before-operation image and the after-operation image of the work 3 aligned by the image correction module 455, thereby acquiring a difference image. FIG. 9 shows an overview of the difference image extraction process executed by the difference image creation module 456. As shown in FIG. 9, first, the difference image creation module 456 cuts out the respective inspection areas of the before-operation image and the after-operation image of the work 3 with reference to the inspection area information 441 stored in the memory 43. Then, the difference image creation module 456 extracts the difference between the inspection area of ​​the before-operation image and the inspection area after the operation, thereby acquiring a difference image. More specifically, the difference image creation module 456 creates a difference image by obtaining the absolute value of the difference between the absolute value of the pixel value of the inspection area of ​​the before-operation image and the absolute value of the corresponding pixel value of the inspection area of ​​the after-operation image. Then, the difference image creation module 456 performs binarization processing using an appropriate threshold value on the created difference image, thereby acquiring a binarized difference image. 9, in the binarized difference image, the pixel values ​​of the different parts between the before-operation image and the after-operation image are 1 (white), and the pixel values ​​of the common parts between the before-operation image and the after-operation image are 0 (black). Such a binarized difference image is temporarily stored in memory 43 and is used to determine whether the operation of the robot 1 on the workpiece 3 was successful or not.

[0099] The inspection module 457 has a function of determining whether the robot 1 has performed an operation on the workpiece 3 based on the differential image created by the differential image creation module 456. More specifically, the inspection module 457 first calculates the sum of pixel values ​​(total number of pixels in white parts) of the binarized differential image created by the differential image creation module 456. Next, the inspection module 457 refers to the success / failure determination condition 442 stored in the memory 43, and determines whether the calculated sum of pixel values ​​of the binarized differential image (total number of pixels in white parts) is within a range defined by the lower limit threshold and upper limit threshold of the success / failure determination condition 442.

[0100] If the calculated sum of pixel values ​​of the binarized difference image is within the range defined by the lower and upper thresholds, the inspection module 457 determines that the work performed by the robot 1 on the workpiece 3 was successful. On the other hand, if the calculated sum of pixel values ​​of the binarized difference image is not within the range defined by the lower and upper thresholds, the inspection module 457 determines that the work performed by the robot 1 on the workpiece 3 was unsuccessful. The inspection module 457 then notifies the user or administrator of the inspection system 100 of the determination result using the I / O interface 42. This allows the user or administrator of the inspection system 100 to take appropriate measures, such as inspecting the robot 1, if the work performed by the robot 1 on the workpiece 3 has failed.

[0101] In this way, in the inspection system 100 of the present invention, image correction processing (affine transformation or homography transformation) is performed on at least one of the pre-work image and the post-work image of the work 3 so that the positions of multiple feature points in the pre-work image of the work 3 match the positions of multiple corresponding feature points in the post-work image of the work 3. Therefore, even if the deformations of the pre-work image and the post-work image of the work 3 are different due to differences in the attitude and orientation of the camera 2 with respect to the work 3, the positional deviation between the position of each element of the work 3 in the pre-work image and the position of the corresponding element of the work 3 in the post-work image can be accurately corrected, and it is possible to accurately determine whether the work on the work 3 has been successful.

[0102] <Testing method S100> The inspection method of the present invention is carried out using the above-described inspection system 100. The inspection method of the present invention will be described in detail below with reference to Fig. 10. Fig. 10 is a flow chart showing the inspection method of the present invention.

[0103] The inspection method S100 starts when a user or administrator of the inspection system 100 executes an operation on the control device 4 to start a predetermined operation on the workpiece 3. First, in step S110, the processor 41 of the control device 4 drives the camera 2 using the camera control module 451 to capture an image of the workpiece 3 placed on the work table 11 and obtain a pre-operation image of the workpiece 3.

[0104] Next, in step S120, the processor 41 of the control device 4 uses the robot control module 452 to drive the movable unit 13 and working unit 12 of the robot 1 to perform work on the workpiece 3. Specifically, the robot control module 452 first refers to the before-work image of the workpiece 3 acquired in step S110, and identifies the respective positions of the tip 121 of the working unit 12 and the work point 32 of the workpiece 3 in the before-work image. Thereafter, the robot control module 452 determines the target amount of drive of each of the X-axis movable unit 13X, the Y-axis movable unit 13Y, and the Z-axis movable unit 13Z of the movable unit 13 in order to change the relative position between the working unit 12 of the robot 1 and any one of the work points 32 of the workpiece 3. Thereafter, the robot control module 452 outputs the determined output values ​​to the X-axis movable part 13X and the Y-axis movable part 13Y, drives the X-axis movable part 13X and the Y-axis movable part 13Y independently, and sets the relative position between the working part 12 and the workpiece 3 to a predetermined state appropriate for performing work on one work point 32 of the workpiece 3.

[0105] Thereafter, the robot control module 452 outputs the determined output value to the Z-axis movable part 13Z, drives the Z-axis movable part 13Z, and causes the tip part 121 of the working part 12 to approach one of the work points 32 of the workpiece 3, and causes the tip part 121 of the working part 12 to contact or approach one of the work points 32 of the workpiece 3. Thereafter, the robot control module 452 drives the working part 12 to perform a predetermined operation on one of the work points 32 of the workpiece 3. After the predetermined operation on one of the work points 32 of the workpiece 3 is completed, the robot control module 452 drives the movable part 13 and the working part 12 to perform an operation on another work point 32 of the workpiece 3. When the operation on all the work points 32 of the workpiece 3 by the robot 1 is completed, the working part 12 is returned to the initial position by the robot control module 452. At this time, the camera 2 attached to the Z-axis movable part 13Z of the movable part 13 of the robot 1 is also returned to the initial position.

[0106] Next, in step S130, the processor 41 of the control device 4 uses the camera control module 451 to drive the camera 2 and acquire a post-operation image of the workpiece 3. In step S140, the processor 41 of the control device 4 uses the feature point matching module 453 to identify a plurality of feature points in the pre-operation image of the workpiece 3 and a plurality of feature points in the post-operation image of the workpiece 3 that correspond to the plurality of feature points in the pre-operation image of the workpiece 3, and executes feature point matching to acquire a list of a plurality of feature point pairs each composed of the feature points in the pre-operation image and the feature points in the post-operation image that correspond to each other. At this time, it is preferable that the edge portions of the four corners of the workpiece 3 and the plurality of reference markers 31 in each of the pre-operation image and the post-operation image are detected as feature points.

[0107] Next, in step S150, the processor 41 of the control device 4 uses the exclusion module 454 to execute at least one of the first exclusion process, the second exclusion process, and the third exclusion process described above, to exclude feature points pairs with low reliability from the list of the multiple feature points pairs acquired in step S140. Note that, in a case where two or more of the first exclusion process, the second exclusion process, and the third exclusion process are executed in step S150, the order in which the first exclusion process, the second exclusion process, and the third exclusion process are executed is not particularly limited. For example, in a case where all of the first exclusion process, the second exclusion process, and the third exclusion process are executed in step S150, the first exclusion process, the second exclusion process, and the third exclusion process may be executed in this order.

[0108] In step S160, affine transformation or homography transformation is performed on at least one of the pre-operation image and the post-operation image of the work 3 so that the positions of the multiple feature points of the work 3 in the pre-operation image of the work 3 included in the list of feature point pairs obtained in step S140 and further excluding the feature point pairs with low reliability in step S150 match the positions of the multiple corresponding feature points of the work 3 in the post-operation image of the work 3. The pre-operation image and the post-operation image of the work 3 aligned by the image correction process on at least one of the pre-operation image and the post-operation image of the work 3 by the image correction module 455 are temporarily stored in the memory 43 and used to create a difference image. Note that whether the image correction module 455 performs the affine transformation or the homography transformation on at least one of the pre-operation image and the post-operation image of the work 3 is appropriately set in advance by the user or administrator of the inspection system 100.

[0109] In step S170, the processor 41 of the control device 4 uses the difference image creation module 456 to extract the difference between the before-operation image and the after-operation image of the workpiece 3 aligned by the image correction module 455, thereby acquiring a difference image. More specifically, the difference image creation module 456 refers to the inspection area information 441 stored in the memory 43 and cuts out the inspection areas of the before-operation image and the after-operation image of the workpiece 3. Then, the difference image creation module 456 creates a difference image by obtaining the absolute value of the difference between the absolute value of the pixel value of the inspection area of ​​the before-operation image and the absolute value of the corresponding pixel value of the inspection area of ​​the after-operation image. Then, the difference image creation module 456 performs a binarization process using an appropriate threshold value on the created difference image to acquire a binarized difference image. The binarized difference image is temporarily stored in the memory 43 and is used to determine whether the work on the workpiece 3 by the robot 1 has been successful.

[0110] In step S180, the processor 41 of the control device 4 uses the inspection module 457 to determine whether the robot 1 has performed the work on the workpiece 3. More specifically, the inspection module 457 first calculates the sum of pixel values ​​(total number of pixels in white parts) of the binarized difference image created by the difference image creation module 456. Next, the inspection module 457 refers to the success / failure determination condition 442 stored in the memory 43, and determines whether the calculated sum of pixel values ​​of the binarized difference image (total number of pixels in white parts) is within a range defined by the lower limit threshold and upper limit threshold of the success / failure determination condition 442.

[0111] If the calculated sum of pixel values ​​of the binarized difference image is within the range defined by the lower and upper thresholds, the inspection module 457 determines that the work performed by the robot 1 on the workpiece 3 has been successful. On the other hand, if the calculated sum of pixel values ​​of the binarized difference image is not within the range defined by the lower and upper thresholds, the inspection module 457 determines that the work performed by the robot 1 on the workpiece 3 has failed. Thereafter, the inspection module 457 notifies the user or administrator of the inspection system 100 of the determination result using the I / O interface 42. As a result, if the work performed by the robot 1 on the workpiece 3 has failed, the user or administrator of the inspection system 100 can take appropriate measures, such as inspecting the robot 1. Thereafter, the inspection method S100 ends.

[0112] Although the inspection system and inspection method of the present invention have been described above based on the illustrated embodiment, the present invention is not limited thereto. The configuration of each component of the present invention can be replaced with any other component that can exert a similar function, or any other component can be added to the configuration of the present invention.

[0113] A person skilled in the art in the field and technology to which the present invention pertains will be able to modify the configuration of the described inspection system and inspection method of the present invention without significantly departing from the principles, concepts, and scope of the present invention, and inspection systems and inspection methods having modified configurations are also within the scope of the present invention.

[0114] 10 is merely an example for the purpose of explanation, and the present invention is not necessarily limited thereto. Within the scope of the principle and intent of the present invention, any step may be added or combined for any purpose, or any step may be deleted. [Explanation of symbols]

[0115] 1...Robot 11...Work table 12...Working part 121...Tip part 13...Moving part 131...Base 13X...X-axis moving part 131X...Leg part 132X...X-axis rail 133X...X-axis stage 13Y...Y-axis moving part 131Y...Y-axis rail 132Y...Y-axis stage 13Z...Z-axis moving part 131Z...Z-axis stage 2...Camera 3...Work 31...Reference marker 32...Work point 4...Control device 41...Processor 42...I / O interface 43...Memory 44...Data 441...Inspection area information 442...Success / failure determination condition 443...Other data 45...Module 451...Camera control module 452...Robot control module 453...Feature point matching module 454...Exclusion module 455...Image correction module 456...Difference image creation module 457... Inspection module 458... Other modules 100... Inspection system 500... Camera image S100... Inspection method S110, S120, S130, S140, S150, S160, S170, S180... Process

Claims

1. A robot for performing an operation on a workpiece; A camera capable of photographing the workpiece; a control device for controlling the robot and the camera; The control device includes: Before driving the robot to perform the operation on the workpiece, the camera is driven to acquire a pre-operation image of the workpiece, and further, after driving the robot to perform the operation on the workpiece, the camera is driven to acquire a post-operation image of the workpiece; identifying a plurality of feature points in the before-work image and a plurality of feature points in the after-work image that correspond to the plurality of feature points in the before-work image, and performing feature point matching to obtain a list of a plurality of feature point pairs each composed of the feature points in the before-work image and the feature points in the after-work image that correspond to each other; performing an image correction process on at least one of the before-operation image and the after-operation image such that the positions of the feature points in the before-operation image of each of the plurality of feature point pairs included in the list coincide with the positions of the corresponding feature points in the after-operation image; After performing the image correction process, a difference image is obtained by extracting a difference between the before-operation image and the after-operation image; and The system is configured to determine whether the operation on the workpiece has been successful or not based on the difference image, the robot includes a working unit for performing the task on the workpiece, and a movable unit for changing a relative position between the workpiece and the working unit, When the relative position between the workpiece and the working unit is changed by the movable unit of the robot in order to perform the task on the workpiece, the relative position between the camera and the workpiece is changed; When the pre-operation image is captured by the camera, the working unit and the camera are located at an initial position; When the operation is performed on the workpiece, the working unit and the camera are moved from the initial position; An inspection system, characterized in that when the post-operation image is acquired by the camera, the working unit and the camera are returned to the initial positions.

2. The inspection system according to claim 1 , wherein the image correction process performed on at least one of the before-operation image and the after-operation image is an affine transformation or a homography transformation.

3. The inspection system according to claim 1 or 2, wherein the control device is configured to binarize the difference image and determine whether the operation on the workpiece was successful or not based on the sum of pixel values ​​of the binarized difference image.

4. An inspection system as described in any one of claims 1 to 3, wherein the control device is configured to, after performing the image correction process, cut out inspection areas of each of the pre-work image and the post-work image, and obtain the difference image by extracting the difference between the inspection area of ​​the pre-work image and the inspection area of ​​the post-work image.

5. 5. The inspection system according to claim 1, wherein the control device is configured to, after performing the feature point matching, perform an exclusion process for excluding unreliable feature point pairs from the list.

6. 6. The inspection system of claim 5, wherein the exclusion process identifies low-reliability feature points pairs based on an angle difference between an angle between the multiple feature points of the multiple feature points pairs included in the list in the pre-operation image and an angle between the multiple feature points of the multiple feature points pairs included in the list in the post-operation image, and excludes the low-reliability feature points pairs from the list.

7. 7. The inspection system of claim 5, wherein the exclusion process calculates a Euclidean distance between the feature points in the before-operation image and the feature points in the after-operation image for each of the plurality of feature points pairs included in the list, and further identifies the feature points pairs with low reliability based on the calculated Euclidean distance, and excludes the feature points pairs with low reliability from the list.

8. 8. The inspection system of claim 5, wherein the exclusion process calculates an angle between the feature point in the before-operation image and the feature point in the after-operation image for each of the plurality of feature point pairs included in the list, and further identifies the feature points pairs with low reliability based on the calculated angle, and excludes the feature points pairs with low reliability from the list.

9. The robot further includes a work table on which the workpiece is placed, When a working surface of the work table is an XY plane, the movable part includes an X-axis movable part that is movable in an X-axis direction, The inspection system according to claim 1 , wherein the working unit and the camera are provided so as to be movable in the X-axis direction by the X-axis movable unit.

10. The movable portion further includes a Z-axis movable portion movable in a Z-axis direction, The Z-axis movable part is attached to the X-axis movable part, The inspection system according to claim 9 , wherein the working unit and the camera are attached to the Z-axis movable unit, thereby allowing movement in the X-axis direction and the Z-axis direction.

11. An inspection method for determining whether a robot has performed a task on a workpiece using a camera, comprising: before driving the robot to perform the operation on the workpiece, driving the camera to acquire a pre-operation image of the workpiece, and further, after driving the robot to perform the operation on the workpiece, driving the camera to acquire a post-operation image of the workpiece; performing feature point matching to identify a plurality of feature points in the before-work image and a plurality of feature points in the after-work image that correspond to the plurality of feature points in the before-work image, and to obtain a list of a plurality of feature point pairs each composed of the feature points in the before-work image and the feature points in the after-work image that correspond to each other; performing an image correction process on at least one of the before-operation image and the after-operation image such that the positions of the feature points in the before-operation image of each of the plurality of feature point pairs included in the list coincide with the positions of the corresponding feature points in the after-operation image; obtaining a difference image by extracting a difference between the before-operation image and the after-operation image; and determining whether the operation on the workpiece has been completed or not based on the difference image. the robot includes a working unit for performing the task on the workpiece, and a movable unit for changing a relative position between the workpiece and the working unit, When the relative position between the workpiece and the working unit is changed by the movable unit of the robot in order to perform the task on the workpiece, the relative position between the camera and the workpiece is changed; When the pre-operation image is captured by the camera, the working unit and the camera are located at an initial position; When the operation is performed on the workpiece, the working unit and the camera are moved from the initial position; An inspection method, characterized in that when the post-work image is acquired by the camera, the working unit and the camera are returned to the initial positions.

Images