Visual system and visual inspection method

Abstract

Based on a detection result of visual detection of an image, a taking-out position of a workpiece to be taken out by a robot is obtained with high precision. The vision system includes: an acquisition unit that acquires a first image in which a presence region of a plurality of workpieces is captured; a detection unit that outputs detection results of the plurality of workpieces; and a robot that performs an operation of changing the position of at least one of the plurality of workpieces. The robot executes a first operation of changing positions of the plurality of workpieces, the acquisition unit acquires a second image in which a presence region of the plurality of workpieces after the execution of the first operation is captured, and the detection unit detects the first image and the second image. At least one of the position, orientation, and appearance information of the plurality of workpieces is output as the detection result on the basis of at least the amount of change between the first image data including the detection result and the second image data including the detection result.

Description

Technical Field

[0001] The present invention relates to a visual system and a visual inspection method. Background Art

[0002] There is an application that visually inspects an image captured by a two-dimensional / three-dimensional camera and causes a robot to pick up a workpiece based on information such as the inspection position, posture, or shape of the workpiece included in the inspection result.

[0003] For example, Patent Document 1 proposes the following technology: a robot arm is moved to induce the collapse of a workpiece, and after the collapse, the robot arm is photographed again to find a workpiece that can be grasped again.

[0004] Patent document 2 proposes the following technology: performing a contact action on the workpiece, measuring the force applied to the robot during contact and the position of each joint of the robot, performing a positive transformation from the joint position to calculate the position of the front end of the end effector, updating the surface shape of the workpiece, and finding a new holding position to hold the workpiece.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2019-198949

[0008] Patent Document 2: Japanese Patent Application Publication No. 2017-136677 Summary of the invention

[0009] Problems to be solved by the invention

[0010] In Patent Document 1, the inducing action of workpiece collapse may cause two workpieces to be in close contact with each other. In this case, the two closely contacted workpieces may be mistakenly detected as one large workpiece, and the position, posture or external dimensions of the workpieces may not be correctly detected.

[0011] On the other hand, although Patent Document 2 stops the contact action when the force measured during contact reaches the upper limit value, the force information is not used in the update of the map information for recalculating the surface shape of the workpiece (i.e., visual detection). In addition, although Patent Document 2 uses the measured values ​​of the positions of each joint of the robot to calculate the position of the front end of the end effector in order to update the information of the three-dimensional point group, i.e., the map information, the accuracy of the calculated front end position of the end effector is sometimes poor due to the adverse effects of the backlash of the reduction gear in the robot body or the manufacturing errors and deflections of the robot arm and the end effector / hand. Therefore, Patent Document 2 has the following problems: it cannot produce high-precision map information, and cannot produce high-precision surface shape of the workpiece, and cannot obtain high-precision detection results.

[0012] In addition, when performing visual inspection on captured images, the position and external dimensions of the workpiece may be misdetected due to the influence of tape or labels affixed to the cardboard box serving as the workpiece, resulting in the output of erroneous visual inspection results.

[0013] In addition, for example, the gaps between densely stacked cartons may not be visually detected, and the area containing the gaps may be mistakenly detected as the detection position, or small grooves, steps, holes, etc. may not be visually detected, and the area containing these may be mistakenly detected as the detection position. In these cases, when a robot with an adsorption pad goes to the detected detection position to pick up a workpiece such as a carton, air will leak, causing the removal action to fail.

[0014] Therefore, it is desired to obtain the removal position of the workpiece to be removed by the robot with high accuracy based on the detection result of visual detection of images.

[0015] Means for solving problems

[0016] One mode of the visual system disclosed in the present invention comprises: an acquisition unit, which acquires a first image of an area where a plurality of workpieces exist; a detection unit, which outputs detection results of the plurality of workpieces; a robot, which performs an action to change the position of at least one of the plurality of workpieces; the robot performs the first action to change the positions of the plurality of workpieces, the acquisition unit acquires a second image of an area where a plurality of workpieces exist after performing the first action, the detection unit detects the first image and the second image, and outputs at least one of the positions, postures and appearance information of the plurality of workpieces as the detection result based at least on the amount of change between the first image data including the detection result and the second image data including the detection result.

[0017] One mode of the visual inspection method disclosed in the present invention comprises: an acquisition step, which acquires a first image of an area where a plurality of workpieces exist; a detection step, which outputs detection results of the plurality of workpieces; an execution step, which causes a robot to execute an action to change the position of at least one of the plurality of workpieces; the execution step causes the robot to execute the first action to change the positions of the plurality of workpieces, the acquisition step acquires a second image of an area where a plurality of workpieces exist after executing the first action, the detection step detects the first image and the second image, and outputs at least one of the positions, postures and appearance information of the plurality of workpieces as the detection result based at least on the amount of change between the first image data including the detection result and the second image data including the detection result.

[0018] Effects of the Invention

[0019] According to one aspect, the removal position of a workpiece to be removed by a robot can be obtained with high accuracy based on the detection result of visual detection of an image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG. 1 is a diagram showing an example of the configuration of a visual system according to a first embodiment.

[0021] FIG. 2 is a functional block diagram showing an example of a functional configuration of a visual inspection device.

[0022] FIG. 3 is a diagram showing an example of an image of an area where a plurality of cartons exist before the positions of the cartons are changed.

[0023] FIG. 4 is a diagram showing an example of an image captured after the position of the carton in FIG. 3 has been changed.

[0024] FIG. 5 is a diagram showing an example of an image captured after the position of the carton in FIG. 4 has been changed.

[0025] FIG. 6 is a diagram showing an example of the final detection result.

[0026] FIG. 7 is a flowchart illustrating the detection process of the visual system.

[0027] FIG. 8 is a diagram showing an example of the configuration of a visual system according to the second embodiment.

[0028] FIG. 9 is a functional block diagram showing an example of a functional configuration of a visual inspection device.

[0029] FIG. 10 is a flowchart illustrating the detection process of the visual system.

[0030] FIG. 11 is a diagram showing an example of the configuration of a visual system according to the third embodiment.

[0031] FIG. 12 is a functional block diagram showing an example of a functional structure of a visual inspection device.

[0032] FIG. 13 is a diagram showing an example of a waveform of a contact force caused by contact with the surface of a carton measured by a tactile / force sensor.

[0033] FIG. 14 is a flowchart illustrating the detection processing of the visual system.

[0034] FIG. 15 is a diagram showing an example of a waveform of a force caused by contact with the upper surface of a workpiece measured by the tactile / force sensor. DETAILED DESCRIPTION

[0035] Modes for carrying out the invention

[0036] Hereinafter, the first to third embodiments will be described in detail with reference to the drawings.

[0037] Here, each embodiment is common in the following structure: performing visual detection of a first image of an area where a workpiece exists, causing the robot to perform a first action on the workpiece to obtain at least one of the position, posture, and shape information of the workpiece, and outputting at least one of the position, posture, and shape information of the workpiece as a detection result.

[0038] Among them, in the first embodiment, in a state where a plurality of workpieces are densely stacked, the robot is caused to execute a first action of changing the positions of the plurality of workpieces, a second image of the region where the plurality of workpieces exist after the execution of the first action is obtained, the first image and the second image are visually inspected, and the inspection results of the plurality of workpieces are output based on the amount of change between the first image data containing the inspection results and the second image data containing the inspection results. In contrast, in the second embodiment, the difference from the first embodiment is that in the first image, when there is an area where the three-dimensional position cannot be obtained by visual inspection (for example, when there is an area where halation occurs, or when the workpiece is transparent), the robot is caused to execute a first action of touching the workpiece for the failed area, obtain contact information indicating the three-dimensional position of the area, complete the three-dimensional position of the failed area based on the obtained contact information, and correct the inspection result. In addition, the third embodiment is different from the first and second embodiments in that when the surface of a workpiece is not in a uniform state, but has features that separate areas of the surface (for example, different areas of the surface have different textures or materials), and is thereby mistakenly determined to be a plurality of workpieces, the robot executes the first action of touching the surface of the workpiece, identifies the change in texture on the workpiece to distinguish the area, and corrects the detection result of the workpiece.

[0039] Hereinafter, the first embodiment will be described in detail, and then the second and third embodiments will be described with particular reference to portions different from the first embodiment.

[0040] <First Embodiment>

[0041] Fig. 1 is a diagram showing an example of the structure of a visual system according to the first embodiment. Here, a case where a plurality of cartons are densely stacked as a plurality of workpieces is exemplified. In addition, the present invention can be applied to a workpiece other than cardboard, a fixture, etc., and can also be applied to a workpiece of a shape such as a triangle or a hexagon.

[0042] As shown in FIG. 1 , the vision system 1 includes a vision inspection device 10 , a robot control device 20 , a robot 30 , a photographing device 40 , a plurality of cartons 50 , and a pallet 60 .

[0043] The visual inspection device 10, the robot control device 20, the robot 30 and the camera 40 may also be directly connected to each other via a connection interface not shown in the figure. In addition, the visual inspection device 10, the robot control device 20, the robot 30 and the camera 40 may also be connected to each other via a network not shown in the figure, such as a LAN (Local Area Network) or the Internet. In this case, the visual inspection device 10, the robot control device 20, the robot 30 and the camera 40 are provided with a communication unit not shown in the figure for communicating with each other through the connection. In addition, for ease of explanation, FIG. 1 depicts the visual inspection device 10 and the robot control device 20 separately, and the visual inspection device 10 in this case may also be constituted by, for example, a computer. Not limited to this type of structure, for example, the visual inspection device 10 may also be installed inside the robot control device 20 as described later and integrated with the robot control device 20.

[0044] The robot control device 20 is a device known to those skilled in the art for controlling the motion of the robot 30. In addition, in Fig. 1, a teaching operation panel 25 for teaching the motion to the robot 30 is connected to the robot control device 20. In addition, the robot control device 20 has a display unit 21 such as a liquid crystal display.

[0045] The robot control device 20, for example, operates the robot 30 based on the detection result output from the visual inspection device 10 described later, so as to change the position of at least one of the densely stacked cartons 50. The robot control device 20 receives the detection result of the cartons 50 detected by the visual inspection device 10 described later, using the image of the existence area of ​​the plurality of cartons 50 captured by the imaging device 40 before the position change and the image of the existence area of ​​the plurality of cartons 50 after the change. The robot control device 20 displays the received detection result together with the image captured by the imaging device 40 on the display unit 21 of the robot control device 20. The robot control device 20 selects the carton 50 to be taken out based on the received detection result or the selection instruction of the user via the teaching operation panel 25. The robot control device 20 generates a control signal for controlling the operation of the robot 30 in order to move the hand 31 of the robot 30 to the removal position of the selected carton 50. Then, the robot control device 20 outputs the generated control signal to the robot 30.

[0046] Furthermore, the display unit 21 may be disposed on the teaching operation panel 25 .

[0047] In addition, the robot control device 20 may include the visual inspection device 10 as described later.

[0048] The robot 30 is a robot that moves under the control of the robot control device 20. The robot 30 includes a base portion that rotates around a vertical axis, a movable and rotating arm, and a hand 31 mounted on the arm to hold the carton 50. In addition, in FIG. 1 , an air suction type removal hand is mounted on the hand 31 of the robot 30, but a gripping type removal hand may also be mounted.

[0049] The robot 30 drives the arm or hand 31 according to the control signal output by the robot control device 20, moves the hand 31 to the removal position of the selected carton 50, holds the selected carton 50 and removes it from the pallet 60.

[0050] In addition, illustration is omitted about the transfer destination of the taken-out carton 50. In addition, since the specific structure of the robot 30 is well known to those skilled in the art, detailed description is omitted.

[0051] In addition, the visual inspection device 10 or the robot control device 20 has previously calibrated to associate the mechanical coordinate system used to control the robot 30 with the camera coordinate system of the detection results representing the position, posture, and shape information of the carton 50.

[0052] The photographing device 40 is a digital camera or the like, and photographs a two-dimensional image that projects the existence area of ​​the plurality of cartons 50 densely stacked on the pallet 60 onto a plane perpendicular to the optical axis of the photographing device 40. The image photographed by the photographing device 40 may also be an RGB color image or a visible light image such as a grayscale image or a depth image. In addition, the photographing device 40 may also be configured to include an infrared sensor and photograph thermal images, or may be configured to include an ultraviolet sensor and photograph ultraviolet images for inspecting scratches or spots on the surface of an object. In addition, the photographing device 40 may also be configured to include an X-ray sensor and photograph X-ray images, or may be configured to include an ultrasonic sensor and photograph ultrasonic images. In addition, the photographing device 40 may also be a three-dimensional measuring machine such as a stereo camera.

[0053] The cartons 50 are placed in a densely stacked state on the pallet 60. As described above, the workpiece is not limited to the cartons 50, and any workpiece that can be held by the hand 31 mounted on the arm of the robot 30 may be used, and its shape and the like are not particularly limited.

[0054] <Visual Inspection Device 10>

[0055] FIG. 2 is a functional block diagram showing an example of a functional configuration of the visual inspection device 10 .

[0056] The visual inspection device 10 is a computer known to those skilled in the art, and includes a control unit 100 and a storage unit 200 as shown in FIG2 . In addition, the control unit 100 includes an acquisition unit 110 and a detection unit 120 .

[0057] <Storage Unit 200>

[0058] The storage unit 200 is an SSD (Solid State Drive) or an HDD (Hard Disk Drive), etc. In the storage unit 200, an operating system and application programs executed by the control unit 100 are stored together with an image captured by the camera 40 and acquired by the acquisition unit 110 described later.

[0059] <Control Unit 100>

[0060] The control unit 100 has a CPU (Central Processing Unit), ROM, RAM (Random Access Memory), CMOS (Complementary Metal-Oxide-Semiconductor) memory, etc., which are structures known to those skilled in the art that are capable of communicating with each other via a bus.

[0061] The CPU is a processor that controls the visual inspection device 10 as a whole. The CPU reads the system program and application program stored in the ROM via the bus, and controls the visual inspection device 10 as a whole according to the system program and application program. Thus, as shown in FIG2 , the control unit 100 is configured to realize the functions of the acquisition unit 110 and the detection unit 120. Various data such as temporary calculation data or display data are stored in the RAM. In addition, the CMOS memory is configured as a non-volatile memory, which is backed up by a battery not shown in the figure, and the storage state will be maintained even if the power of the visual inspection device 10 is disconnected.

[0062] <Acquisition Unit 110>

[0063] The acquisition unit 110 acquires, for example, an image of the existence area in which the plurality of cartons 50 are captured from the imaging device 40. The acquisition unit 110 stores the acquired image in the storage unit 200.

[0064] Furthermore, the acquisition unit 110 acquires an image from the imaging device 40 , but may also acquire three-dimensional point group data, range image data, or the like.

[0065] <Detection Unit 120>

[0066] The detection unit 120 detects multiple cartons 50 by, for example, using an image (first image) of an area where multiple cartons 50 exist captured by the camera 40 before the robot control device 20 causes the robot 30 to move to change the position of the carton 50, and an image (second image) of an area where multiple cartons 50 exist captured by the camera 40 after the position has changed, and outputs a detection result including position, posture and appearance information of the multiple cartons 50 to the robot control device 20 at least based on the amount of change between the first image data including the detection result and the second image data including the detection result.

[0067] Specifically, the detection unit 120 reads out an image (first image) of an area where multiple cartons 50 exist, which is captured by the camera 40 before the robot control device 20 causes the robot 30 to move to change the position of the carton 50, from the storage unit 200, and performs visual detection on the read image.

[0068] Fig. 3 is a diagram showing an example of an image of an area where a plurality of cartons 50 exist before the positions of the cartons 50 are changed. The image of Fig. 3 is an image of a plurality of cartons 50 captured by the imaging device 40 from above, for example, in the Z-axis direction.

[0069] The detection unit 120 detects three cartons 50A1, 50A2, and 50A3 represented by thick rectangles that are densely stacked with each other (a diagram when viewed from the TOP surface of the carton 50) by visual detection of the image of FIG. 3, and obtains the positions (positions of plus signs) of the detected cartons 50A1, 50A2, and 50A3. A1 , X A2 , X A3 , Posture P A1 , P A2 , P A3 and shape information (e.g. the width W of the thick rectangle A1 , W A2 , W A3 and depth D A1 , D A2 , D A3 ).

[0070] Next, the detection unit 120 reads an image (a second image) from the storage unit 200 and performs visual detection on the read image. The image (the second image) is an image captured by the camera device 40 after the positions of the multiple cartons 50 are changed, when the robot control device 20 uses the hand 31 installed on the robot 30 to perform an action (a first action, an auxiliary action) of applying force to the three cartons 50A1, 50A2, and 50A3 in the direction of the arrow (Y-axis direction) shown in FIG. 3 .

[0071] FIG. 4 is a diagram showing an example of an image captured after the position of the carton 50 of FIG. 3 has been changed.

[0072] As shown in FIG4 , the detection unit 120 detects four cartons 50B1, 50B2, 50B3, and 50B4 represented by the bold rectangles through visual detection of the image, and obtains the positions (positions of the plus signs) X of the detected cartons 50B1, 50B2, 50B3, and 50B4. B1 , X B2 , X B3 , X B4 , Posture P B1 , P B2 , P B3 , P B4 and shape information (width W B1 , W B2 , W B3 , W B4 and depth D B1 , D B2 , D B3 , D B4 ). That is, the detection unit 120 mistakenly detects the carton 50A1 in FIG. 3 as one carton, but by using the result of the auxiliary action applied by the robot 30, the gap between the two cartons 50B1 and 50B2 can be detected, thereby correctly detecting the carton 50B1 and 50B2.

[0073] Furthermore, the detection unit 120 reads an image (the third image) from the storage unit 200 and performs visual detection on the read image. The image (the third image) is an image captured by the camera device 40 after the positions of the multiple cartons 50 are changed, when the robot control device 20 uses the hand 31 installed on the robot 30 to perform an auxiliary action (the second action) of applying force to the four cartons 50B1, 50B2, 50B3, and 50B4 in the direction of the arrow shown in Figure 4 (X-axis direction).

[0074] FIG. 5 is a diagram showing an example of an image captured after the position of the carton 50 of FIG. 4 has been changed.

[0075] The detection unit 120 detects three cartons 50C1, 50C2, and 50C3 represented by the bold rectangles by visually detecting the image of FIG. 5, and obtains the positions (positions of the plus signs) of the detected cartons 50C1, 50C2, and 50C3. C1 , X C2 , X C3 , Posture P C1 , P C2 , P C3 , and shape information (width W C1 , W C2 , W C3and depth D C1 , D C2 , D C3 ). In addition, the detection unit 120 detects cartons 50B3 and 50B4 in FIG. 4 (cartons 50A2 and 50A3 in FIG. 3) as two cartons, but due to the auxiliary action of the robot 30, the gap between cartons 50B3 and 50B4 cannot be detected, so cartons 50B3 and 50B4 are mistakenly detected as one carton 50C3. In this way, by applying the auxiliary action, it is also possible to have the opposite effect of causing the visual detection that could have been correct to be mistakenly detected.

[0076] The detection unit 120 integrates and calculates the first image data, the second image data, and the third image data including the visual detection results before and after applying the multiple auxiliary actions shown in Figures 3 to 5. For example, if the area is calculated based on the width and depth of the detected shape of the carton 50, and the difference in the area of ​​the visual detection results of Figures 3 and 4 is calculated for comparison, it can be determined that two cartons 50B1 and 50B2 are mistakenly detected as one carton 50A1. As a result, the detection unit 120 can output the correct final detection result including the position or posture and shape information of each carton 50D1~50D4 to the image as shown in Figure 6, and can be stored in the storage unit 200 as a file. The detection unit 120 outputs the detection result to the robot control device 20.

[0077] The above describes the situation where the final detection results of multiple workpieces are output based on the changes in the position, posture and appearance information of the workpieces as the detection results, but it is not limited to this. For example, by calculating the changes (differences) of only two images shown in Figures 3 and 4 taken before and after the execution of the first action, it can be seen that in the area of ​​carton 50B1 and carton 50B2 on the image after the action, which is equivalent to the area of ​​carton 50A1 on the image before the action, due to the movement of the carton, there is a large difference in the pixel values ​​in the same area before and after the action. If carton 50A1 is not a state of two carton boxes close together but one carton, even if an auxiliary action is applied, the difference in the image in the area of ​​carton 50A1 before and after the action should not be large. From the situation where a large difference appears, it can be seen that the two cartons are separated from the close state by applying the auxiliary action. Therefore, it can be seen that the two close cartons are mistakenly detected as one.

[0078] <Detection Processing by Visual System 1>

[0079] Next, the flow of the detection process of the visual system 1 will be described with reference to FIG. 7 .

[0080] Fig. 7 is a flowchart for explaining the detection process of the visual system 1. The process shown here is repeatedly executed every time a plurality of cartons 50 are imaged by the image pickup device 40.

[0081] In step S11 , the imaging device 40 captures an image of an area where a plurality of cartons 50 exist.

[0082] In step S12 , the visual inspection device 10 (the acquisition unit 110 ) acquires the image captured in step S11 , and stores the acquired image in the storage unit 200 .

[0083] In step S13, the robot control device 20 determines whether the robot 30 has performed a predetermined number of auxiliary actions (e.g., 2 times, etc.) using the hand 31 for a plurality of cartons 50. When the robot 30 has performed the predetermined number of auxiliary actions using the hand 31, the process proceeds to step S16. On the other hand, when the robot 30 has not performed the predetermined number of auxiliary actions using the hand 31, the process proceeds to step S14.

[0084] In step S14 , the robot control device 20 causes the robot 30 to perform one assist operation in a predetermined direction shown in FIG. 3 or 4 .

[0085] In step S15, the robot controller 20 determines whether the auxiliary operation performed in step S14 has been completed. If the auxiliary operation has been completed, the process returns to step S11. On the other hand, if the auxiliary operation has not been completed, the process waits until the auxiliary operation has been completed.

[0086] In step S16 , the visual inspection device 10 (detection unit 120 ) reads the image captured in step S11 from the storage unit 200 , and performs visual inspection of the read image.

[0087] In step S17, the visual inspection device 10 (detection unit 120) calculates the difference in position, posture or appearance information between the image data containing the detection results of the visual inspection, and calculates the position, posture and appearance information of each carton 50 based on the calculated difference (change amount).

[0088] In step S18 , the visual inspection device 10 (inspection unit 120 ) calculates a final inspection result based on the calculation result in step S17 , and outputs the final inspection result to the robot control device 20 .

[0089] In step S19 , the robot control device 20 controls the robot 30 based on the detection result of step S18 so as to cause the robot 30 to take out one carton 50 .

[0090] Thus, the visual system 1 of the first embodiment is that the robot 30 performs auxiliary actions on a plurality of densely stacked cartons 50, so that even if a plurality of cartons 50 are densely stacked, the position, posture, and shape information of each cartons 50 can be accurately visually detected from the image. Thus, the visual system 1 can obtain the removal position of the workpiece to be removed by the robot with high accuracy based on the detection results of the visual detection of the image.

[0091] In addition, the visual system 1 captures multiple images before and after the auxiliary action in which the robot 30 contacts the workpiece (carton 50) and uses them in an integrated manner. Thus, even if an auxiliary action is performed that results in an undesirable state in which the two workpieces are in close contact, the difference between the detected position and the outer dimensions of the workpiece on the images before and after the auxiliary action can still be calculated, and the correct position or outer dimensions of the workpiece can be calculated.

[0092] The first embodiment has been described above.

[0093] <Second Embodiment>

[0094] Next, the second embodiment is described. As described above, in the first embodiment, in a state where a plurality of workpieces are densely stacked, the robot is caused to perform the first action of changing the positions of the plurality of workpieces, a second image of the region where the plurality of workpieces exist after the first action is performed is obtained, the first image and the second image are visually inspected, and the inspection results of the plurality of workpieces are output based on the amount of change between the first image data containing the inspection results and the second image data containing the inspection results. In contrast, in the second embodiment, the difference from the first embodiment is that in the first image, when there is an area of ​​the first image where the three-dimensional position cannot be obtained by visual inspection (for example, when halo occurs or the workpiece is transparent), the robot is caused to perform the first action of touching the workpiece for the failed area, contact information indicating the three-dimensional position of the area is obtained, and the three-dimensional position of the failed area is supplemented based on the obtained contact information to correct the inspection result.

[0095] Thus, the vision system 1 can obtain the removal position of the workpiece to be removed by the robot with high accuracy based on the detection result of the visual detection of the image.

[0096] The second embodiment will be described below.

[0097] Fig. 8 is a diagram showing an example of the configuration of a visual system according to the second embodiment. Elements having the same functions as those of the visual system 1 of Fig. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0098] Here, the workpiece is exemplified by a glossy workpiece such as stainless steel, or a transparent workpiece such as plastic. In addition, the present invention can also be applied to any workpiece that generates halation other than a glossy workpiece or a transparent workpiece.

[0099] As shown in FIG. 8 , the vision system 1 includes a visual inspection device 10 a , a robot control device 20 , a robot 30 , an imaging device 40 , a workpiece 50 a , and a pallet 60 .

[0100] The robot control device 20 , the robot 30 , and the pallet 60 have functions equivalent to those of the robot control device 20 , the robot 30 , and the pallet 60 of the first embodiment.

[0101] Furthermore, a force sensor 32 serving as a force measuring unit is disposed at the distal end of the hand of the robot 30 .

[0102] The workpiece 50 a is a glossy workpiece or a transparent workpiece, and is placed on a pallet 60 .

[0103] <Visual Inspection Device 10a>

[0104] Fig. 9 is a functional block diagram showing a functional configuration example of the visual inspection device 10a. Elements having the same functions as those of the visual inspection device 10 in Fig. 2 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0105] Similar to the visual inspection device 10 of the first embodiment, the visual inspection device 10a includes a control unit 100a and a storage unit 200. In addition, the control unit 100a includes an acquisition unit 110 and a detection unit 120a.

[0106] The storage unit 200 has the same function as that of the storage unit 200 of the first embodiment.

[0107] <Control Unit 100a>

[0108] The control unit 100 a includes a CPU, a ROM, a RAM, a CMOS memory, and the like, and these are structures known to those skilled in the art and are configured to be able to communicate with each other via a bus.

[0109] The CPU is a processor that controls the visual inspection device 10a as a whole. The CPU reads the system program and application program stored in the ROM via the bus, and controls the visual inspection device 10a as a whole according to the system program and application program. Thus, as shown in FIG. 9 , the control unit 100a is configured to realize the functions of the acquisition unit 110 and the detection unit 120a.

[0110] The acquisition unit 110 has the same function as that of the acquisition unit 110 of the first embodiment.

[0111] <Detection Unit 120a>

[0112] The detection unit 120a, for example, performs visual detection on the image of the area where the workpiece 50a exists captured by the imaging device 40, calculates the position, posture, and shape information of the workpiece 50a based on the amount of change between the image data including the detection result, and outputs the detection result to the robot control device 20. However, when the workpiece 50a is made of stainless steel or the like and halation occurs due to gloss, or when the workpiece 50a is a transparent workpiece such as plastic, the detection unit 120a may fail to obtain three-dimensional data of the position, posture, and shape information of the workpiece.

[0113] In this case, the detection unit 120a outputs a signal indicating that the three-dimensional data acquisition has failed to the robot control device 20, for example. The robot control device 20 causes the hand 31 of the robot 30 to perform an auxiliary action (first action) of touching and contacting the workpiece 50a, and detects the contact with the workpiece 50a through the force sensor 32 installed at the end of the hand of the robot 30. The detection unit 120a calculates and completes the shape of the workpiece 50a based on the contact information (force data) measured by the force sensor 32 of the entire workpiece 50a or the area where the measurement result of the three-dimensional position of the workpiece 50a is missing due to halo, so as to correct the three-dimensional position of the workpiece 50a. In addition, the contact information preferably includes information related to the presence or absence of contact and information related to the contact position. Then, the detection unit 120a outputs the corrected detection result to the robot control device 20.

[0114] Furthermore, the detection unit 120 a may correct the image (the first image, the second image) captured by the imaging device 40 based on the contact information.

[0115] <Detection Processing by Visual System 1>

[0116] Next, the flow of the detection process of the visual system 1 will be described with reference to FIG. 10 .

[0117] Fig. 10 is a flowchart for explaining the detection process of the vision system 1. The flow shown here is repeatedly executed every time the image capturing device 40 captures an image of the workpiece 50a.

[0118] In addition, the processes of step S31 , step S32 , step S39 , and step S40 are the same as the processes of step S11 , step S12 , step S18 , and step S19 in the first embodiment, and the description thereof is omitted.

[0119] In step S33 , the visual inspection device 10 a (detection unit 120 a ) performs visual inspection of the image captured in step S11 .

[0120] In step S34, the visual inspection device 10a (detection unit 120a) determines whether the detection result of the image is insufficient (for example, halo occurs, or the workpiece 50a is transparent). When the detection result of the image is insufficient (for example, halo occurs, or the workpiece 50a is transparent), the detection unit 120a outputs a signal indicating failure to obtain three-dimensional data to the robot control device 20, and the process proceeds to step S35. On the other hand, when the detection result of the image is sufficient (for example, halo does not occur, and the workpiece 50a is not transparent), the process proceeds to step S39.

[0121] In step S35 , the robot control device 20 causes the robot 30 to perform an assist operation of causing the hand 31 to touch and contact the workpiece 50 a .

[0122] In step S36, the robot control device 20 determines whether the hand 31 contacts the workpiece 50a based on the force data from the force sensor 32. When the hand 31 contacts the workpiece 50a, the process proceeds to step S37. On the other hand, when the hand 31 does not contact the workpiece 50a, the process returns to step S35 and waits until the hand 31 contacts the workpiece 50a.

[0123] In step S37 , the force sensor 32 measures contact information (force data) of the entire workpiece 50 a or a region where the measurement result of the three-dimensional position of the workpiece 50 a is missing due to halation, based on the assist operation in step S35 .

[0124] In step S38 , the visual inspection device 10 a (inspection unit 120 a ) corrects the inspection result of the workpiece 50 a based on the contact information measured by the force sensor 32 for the entire workpiece 50 a or a region where the measurement result of the three-dimensional position of the workpiece 50 a is missing due to halation.

[0125] Thus, when there is an area of ​​the first image in which the three-dimensional position cannot be obtained by visual detection in the image captured by the camera 40 (for example, when halo occurs or the workpiece 50a is transparent), the visual system 1 of the second embodiment causes the hand 31 of the robot 30 to perform an auxiliary action of touching and contacting the workpiece 50a, and causes the force sensor 32 installed at the end of the hand to measure the entire workpiece 50a or the area where the measurement result of the three-dimensional position of the workpiece 50a is missing due to halo, and calculates and complements the shape of the workpiece 50a based on the measured contact information, and corrects the detection result of the workpiece 50a. Thus, the visual system 1 can obtain the removal position of the workpiece to be removed by the robot with high accuracy based on the detection result of the visual detection of the image.

[0126] That is, the visual system 1 calculates the change in force level and extracts features from the contact information including the contact force or contact torque when the robot 30 contacts the workpiece 50a, and feeds back the features of the force information reflecting the actual contact position of the hand 31 with the workpiece 50a in the visual detection results. In addition to providing information that cannot be obtained visually, it will also correct the detection results that are often erroneous due to visual means, thereby obtaining high-precision detection results.

[0127] The second embodiment has been described above.

[0128] <Modification of the Second Embodiment>

[0129] In the second embodiment, one workpiece 50a is placed on the pallet 60, but it is not limited to this. For example, a plurality of workpieces 50a may be densely stacked, similarly to the case of the carton 50 of the first embodiment. In this case, the visual system 1 is sometimes unable to determine which workpiece is located above and which workpiece is located below by visual detection. Therefore, the robot control device 20 may also perform an auxiliary action on the robot 30 to touch the workpiece 50a with the hand 31, so that the force sensor 32 measures the contact information. The visual detection device 10a may also determine the relative position relationship of the workpiece 50a based on the measured contact information, and correct the detection result of the visual detection.

[0130] <Third Embodiment>

[0131] Next, the third embodiment is described. As described above, in the first embodiment, in a state where a plurality of workpieces are densely stacked, the robot is caused to execute the first action of changing the positions of the plurality of workpieces, a second image of the region where the plurality of workpieces exist after the execution of the first action is obtained, the first image and the second image are visually inspected, and the inspection results of the plurality of workpieces are output based on the amount of change between the first image data containing the inspection results and the second image data containing the inspection results. In addition, the second embodiment is different from the first embodiment in that, in the first image, when there is an area where the three-dimensional position fails to be obtained by visual inspection (for example, when there is an area where halo occurs, or when the workpiece is transparent), for the failed area, the robot is caused to execute the first action of touching the workpiece, obtain contact information indicating the three-dimensional position of the area, and the three-dimensional position of the failed area is supplemented based on the obtained contact information to correct the inspection result. In addition, the third embodiment is different from the first and second embodiments in that when the surface of a workpiece is not in a uniform state, but has features that separate areas of the surface (for example, different areas of the surface have different textures or materials), and is thereby mistakenly determined to be a plurality of workpieces, the robot executes the first action of touching the surface of the workpiece, identifies changes in the texture on the workpiece to identify the area, and corrects the detection result of the workpiece.

[0132] Thus, the vision system 1 can obtain the removal position of the workpiece to be removed by the robot with high accuracy based on the detection result of the visual detection of the image.

[0133] The third embodiment will be described below.

[0134] Fig. 11 is a diagram showing an example of the configuration of a visual system according to the third embodiment. Elements having the same functions as those of the visual system 1 of Fig. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0135] Here, the workpiece is exemplified as a carton to which a tape and a label such as a destination label are affixed together. In addition, the present invention can also be applied to any workpiece other than a carton to which a tape and a label are affixed.

[0136] As shown in FIG. 11 , the vision system 1 includes a visual inspection device 10 b , a robot control device 20 , a robot 30 , an imaging device 40 , a workpiece 50 b , and a pallet 60 .

[0137] The robot control device 20 , the robot 30 , and the pallet 60 have functions equivalent to those of the robot control device 20 , the robot 30 , and the pallet 60 of the first embodiment.

[0138] Furthermore, a tactile / force sensor 33 as a force measuring unit is disposed at the distal end of the hand 31 .

[0139] The workpiece 50b is a carton on which labels such as tape and a destination label are affixed, and is placed on a pallet 60. Hereinafter, it is also referred to as the carton 50b unless otherwise specified.

[0140] <Visual Inspection Device 10b>

[0141] Fig. 12 is a functional block diagram showing a functional configuration example of a visual inspection device 10b. Elements having the same functions as those of the visual inspection device 10 in Fig. 2 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0142] Similar to the visual inspection device 10 of the first embodiment, the visual inspection device 10b includes a control unit 100b and a storage unit 200. In addition, the control unit 100b includes an acquisition unit 110 and a detection unit 120b.

[0143] The storage unit 200 has the same function as that of the storage unit 200 of the first embodiment.

[0144] <Control Unit 100b>

[0145] The control unit 100 b includes a CPU, a ROM, a RAM, a CMOS memory, and the like, and these are structures known to those skilled in the art and are configured to be able to communicate with each other via a bus.

[0146] The CPU is a processor that controls the visual inspection device 10b as a whole. The CPU reads the system program and application program stored in the ROM via the bus, and controls the visual inspection device 10b as a whole according to the system program and application program. Thus, as shown in FIG. 12 , the control unit 100b is configured to realize the functions of the acquisition unit 110 and the detection unit 120b.

[0147] The acquisition unit 110 has the same function as that of the acquisition unit 110 of the first embodiment.

[0148] <Detection Unit 120b>

[0149] The detection unit 120b, for example, performs visual detection on the image of the area where a carton 50b exists captured by the camera 40, similarly to the detection unit 120 of the first embodiment, calculates the position, posture, and shape information of the carton 50b based on the amount of change between the image data including the detection result, and outputs the detection result to the robot control device 20. However, since the carton 50b is affixed with labels such as tape and a recipient label, the detection unit 120b may sometimes mistakenly detect the following three workpieces with different textures: the portion of the tape (hereinafter also referred to as the "tape area"), the portion of the label (hereinafter also referred to as the "label area"), and the area of ​​the carton (hereinafter also referred to as the "unpatterned area").

[0150] In this case, the detection unit 120b outputs a signal to the robot control device 20, for example, requesting confirmation of whether the three different textures are the texture of the same workpiece. The robot control device 20 performs the following auxiliary action (first action): the hand 31 of the robot 30 touches and contacts the carton 50b, and the hand 31 maintains the contact state with the carton 50b with a certain force (such as 5N, etc.), and moves along the surface of the carton 50b. At this time, the tactile / force sensor 33 installed at the end of the hand 31 measures the waveform of the contact force caused by contact with the surface of the carton 50b.

[0151] FIG13 is a diagram showing an example of a waveform of a contact force caused by contact with the surface of a carton 50b measured by the tactile / force sensor 33. The upper portion of FIG13 shows an example of a tape area, a label area, and a non-patterned area on the surface of a carton 50b. The middle portion of FIG13 is a diagram showing an example of an auxiliary action. The lower portion of FIG13 is a diagram showing an example of a graph of a waveform of a contact force measured by the tactile / force sensor 33.

[0152] As shown in FIG. 13 , even when the force waveform is pushed with the same predetermined force, the change pattern (amplitude, vibration frequency, etc.) of the contact force will still be different due to the difference in texture of the materials of the tape area, label area, and non-pattern area. The detection unit 120b calculates the amplitude or vibration frequency of the waveform from the measured contact force waveform (contact information) to respectively determine the tape area, label area, and non-pattern area. The detection unit 120b corrects the detection result by eliminating the adverse effects of the tape or label, and can obtain the correct visual detection result of a carton 50b with a tape and a label affixed. Then, the detection unit 120b outputs the corrected detection result to the robot control device 20.

[0153] In addition, the contact information may also include a force waveform which is information related to the change in the magnitude, direction, and distribution of the contact force, as well as information related to the presence or absence of contact, information related to the contact position, information related to the magnitude and direction of the contact torque, and information related to sliding.

[0154] <Detection Processing by Visual System 1>

[0155] Next, the flow of the detection process of the visual system 1 will be described with reference to FIG. 14 .

[0156] Fig. 14 is a flowchart for explaining the detection process of the visual system 1. The process shown here is repeatedly executed every time the imaging device 40 images the carton 50b.

[0157] In addition, the processes of step S51 to step S53, step S55 to step S57, and step S61 are the same as the processes of step S31 to step S33, step S35 to step S37, and step S40 in the second embodiment, and the description thereof is omitted.

[0158] In step S54, the visual inspection device 10b (detection unit 120b) determines whether a plurality of different textures are detected by the visual inspection in step S53. When a plurality of different textures are detected, the visual inspection device 10b (detection unit 120b) outputs a signal to the robot control device 20 requesting confirmation of whether the plurality of different textures are the textures of the same workpiece, and the process proceeds to step S55. On the other hand, when a plurality of different textures are not detected, the process proceeds to step S60.

[0159] In step S58, the visual detection device 10b (detection unit 120b) contacts the upper surface of the carton 50b with a certain force through the hand 31 of the robot 30, and calculates the texture characteristics of the upper surface of the carton 50b based on the force waveform (contact information) measured by the tactile / force sensor 33, and determines the contact areas as the tape area, label area, and non-pattern area, respectively.

[0160] In step S59, the visual inspection device 10b (inspection unit 120b) calculates the final inspection result of treating the carton 50b as one workpiece based on the contact area determined in step S58.

[0161] In step S60 , the visual inspection device 10 b (inspection unit 120 b ) outputs the final inspection result to the robot control device 20 .

[0162] Therefore, the visual system 1 of the third embodiment is that when labels such as recipient labels and tapes are attached to the carton 50b together, the hand 31 of the robot 30 performs an auxiliary action of touching and contacting the carton 50b with a certain force, and the tactile / force sensor 33 installed at the end of the hand 31 measures the waveform of the force on the upper surface of the carton 50b, and calculates the texture characteristics of the carton 50b based on the contact information of the measured force waveform, and determines the contact area as the tape area, the label area, and the non-pattern area, and corrects the detection result of the carton 50b that treats the carton 50b as one workpiece. Therefore, the visual system 1 can obtain the removal position of the workpiece to be removed by the robot with high accuracy based on the detection result of the visual detection of the image.

[0163] That is, the visual system 1 causes the robot 30 to perform an auxiliary action of contacting the carton 50b, and based on the measurement information of the tactile / force sensor 33 in the contact area when the robot 30 contacts the carton 50b, calculates the force change pattern and extracts the texture features, and segment the contact area based on the extracted features, thereby segmenting the tape area, the label area, and the non-pattern area. In this way, the visual system 1 can obtain the correct visual detection result that has eliminated the adverse effects of the tape or label.

[0164] The third embodiment has been described above.

[0165] <Variation of the Third Embodiment>

[0166] In the third embodiment, a carton 50b to which labels such as tape and a recipient label are affixed is used as a workpiece, but it is not limited to this. For example, the workpiece may also be a workpiece having a groove portion or the like formed on the upper surface. In this case, the detection unit 120b may also cause the hand 31 of the robot 30 to perform the following auxiliary action (first action) with respect to the robot control device 20: touch and contact the upper surface of the workpiece having the groove portion or the like formed thereon, so that the hand 31 maintains contact with the workpiece with a certain force (for example, 5N, etc.), and moves along the upper surface. At this time, the tactile / force sensor 33 installed at the end of the hand 31 measures the waveform of the force caused by contact with the upper surface of the workpiece.

[0167] Fig. 15 is a diagram showing an example of a waveform of a force caused by contact with the upper surface of a workpiece measured by the tactile / force sensor 33. The upper portion of Fig. 15 is a diagram showing an example of an auxiliary action. The lower portion of Fig. 15 is a diagram showing an example of a waveform of a force measured by the tactile / force sensor 33.

[0168] As shown in FIG15 , when the hand 31 of the robot 30 moves to the groove of the workpiece, the magnitude of the force measured by the tactile / force sensor 33 decreases, and after passing through the groove, the magnitude of the force returns to the original magnitude. The detection unit 120b can determine whether there is a groove at a specific position on the surface of the workpiece according to the change point of the force based on the contact information of the force waveform of FIG15 .

[0169] That is, the visual system 1 calculates multiple change points of force based on the contact information, extracts features such as grooves or edges, steps, holes, protrusions, detects gaps between workpieces, and calculates detection positions that do not include features that may cause air leakage to obtain correct detection results.

[0170] The first embodiment, the second embodiment, and the third embodiment have been described above, but the visual system 1 is not limited to the above-described embodiments, and includes modifications, improvements, and the like within a range that can achieve the purpose.

[0171] <Modification>

[0172] In the first, second, and third embodiments, the visual inspection device 10 is a device different from the robot control device 20, but the present invention is not limited thereto. For example, the visual inspection device 10 may be installed inside the robot control device 20 and integrated with the robot control device 20.

[0173] In addition, each function included in the visual system 1 in the first embodiment, the second embodiment, and the third embodiment can be realized by hardware, software, or a combination of these. Here, realization by software means that the computer reads and executes the program to realize.

[0174] The program can be stored and supplied to the computer using various types of non-transitory computer readable media (Non-transitory computer readable medium). Non-transitory computer readable media include various types of tangible recording media (Tangible storage medium). Examples of non-transitory computer readable media include magnetic recording media (such as floppy disks, magnetic tapes, hard disk drives), optical magnetic recording media (such as magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, semiconductor memory (such as mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM). In addition, the program can also be supplied to the computer through various types of transitory computer readable media (Transitory computer readable medium). Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable medium can supply the program to the computer via wired communication paths such as wires and optical fibers, or via wireless communication paths.

[0175] Furthermore, the steps describing the program recorded in the recording medium naturally include processing performed in time series according to the order, but are not necessarily processed in time series and also include processing executed in parallel or individually.

[0176] If the above is expressed in other words, the visual system and visual inspection method of the present disclosure can take various embodiments having the following structures.

[0177] (1) The visual system 1 disclosed in the present invention comprises: an acquisition unit 110, which acquires a first image of an area where a plurality of workpieces 50 exist; a detection unit 120, which outputs detection results of the plurality of workpieces 50; and a robot 30, which performs an action to change the position of at least one of the plurality of workpieces 50. The robot 30 performs the first action to change the positions of the plurality of workpieces 50, the acquisition unit 110 acquires a second image of an area where a plurality of workpieces 50 exist after the first action is performed, and the detection unit 120 detects the first image and the second image, and outputs at least one of the positions, postures, and appearance information of the plurality of workpieces 50 as a detection result based at least on the amount of change between the first image data including the detection result and the second image data including the detection result.

[0178] According to the vision system 1 , the removal position of the workpiece to be removed by the robot can be obtained with high accuracy based on the detection result of the visual detection of the image.

[0179] (2) In the vision system 1 described in (1), the detection unit 120 may output detection results of the plurality of workpieces 50 based on a difference between at least one of position, posture, and shape information of the plurality of workpieces 50 included in the first image data and the second image data.

[0180] (3) In the visual system 1 described in (1) or (2), the robot 30 may perform a second action different from the first action of changing the positions of the plurality of workpieces 50, the acquisition unit 110 may acquire a third image of the plurality of workpieces 50 after the second action is performed, the detection unit 120 may detect the third image, and output the detection result based on the amount of change between at least any two of the first image data, the second image data, and the third image data including the detection result.

[0181] (4) In the visual system 1 described in any one of (1) to (3), the robot 30 may include: a force sensor 32 (tactile / force sensor 33) that measures contact information when it contacts multiple workpieces 50; the robot 30 performs a contact action with the surfaces of the multiple workpieces 50 based on the detection result of at least one of the first image and the second image, and the force sensor 32 (tactile / force sensor 33) measures the contact information of the surface.

[0182] (5) In the vision system 1 described in (4), the robot 30 may perform at least the first operation on the plurality of workpieces 50 a based on the contact information.

[0183] (6) In the visual system 1 described in (4) or (5), the contact information may include at least one of information related to the presence or absence of contact, information related to the contact position, information related to the magnitude, direction and distribution of the contact force, information related to the magnitude and direction of the contact torque, and information related to sliding.

[0184] (7) In the visual system 1 described in any one of (4) to (6), the detection units 120a and 120b may correct the detection result based on at least information related to the contact position in the contact information.

[0185] (8) In the visual system 1 described in (6), the detection unit 120a may correct at least one of the first image, the second image, and the detection result based on information related to the magnitude and direction of the contact force or the amount of change in distribution.

[0186] (9) The visual system 1 described in any one of (1) to (8) may further include a display unit 21 that displays the detection result.

[0187] (10) In the visual system 1 described in (9), the display unit 21 may display the first image or the second image acquired by the acquisition unit 110 together with the detection result.

[0188] (11) The display unit may be, for example, a display that can be operated by a user through a touch panel. A user interface may be displayed on the display unit, and based on the first image or the second image displayed on the user interface, the user may specify the position, the magnitude and direction of the force of the action, the number of times, and the conditions for the robot to perform the first action or the second action. In addition, the user may specify, through the user interface, the position of the contact action performed by the robot, the magnitude and direction of the contact force, the path to be moved while contacting, or the area to be contacted.

[0189] The user interface may also include a setting screen for setting parameters related to the robot's operating conditions or the shooting conditions of the shooting device. In addition, the user interface may also display the first image and the second image (and the third image) in an overlapping or aligned manner. In addition, the user interface may also display the detection result together with the first image, etc. In addition, the user interface may also display the detection information detected by the force measurement unit when the robot performs the contact action.

[0190] (12) The visual inspection method disclosed in the present invention comprises: an acquisition step of acquiring a first image of an area where a plurality of workpieces 50 exist; a detection step of outputting the detection results of the plurality of workpieces 50; an execution step of causing the robot 30 to execute an action of changing the position of at least one of the plurality of workpieces 50, wherein the execution step causes the robot 30 to execute the first action of changing the positions of the plurality of workpieces 50, the acquisition step of acquiring a second image of an area where a plurality of workpieces 50 exist after executing the first action, and the detection step of detecting the first image and the second image, and outputting at least one of the positions, postures and appearance information of the plurality of workpieces 50 as the detection result based at least on the amount of change between the first image data including the detection result and the second image data including the detection result.

[0191] According to this visual inspection method, the same effect as (1) can be achieved.

[0192] Description of Reference Numerals

[0193] 1. Vision System

[0194] 10, 10a, 10b Visual inspection device

[0195] 100, 100a, 100b control unit

[0196] 110 Acquisition Department

[0197] 120, 120a, 120b detection unit

[0198] 200 Storage

[0199] 20 Robot control device

[0200] 21 Display

[0201] 25 Teaching operation panel

[0202] 30 Robot

[0203] 31 Remove the hand

[0204] 32 Force Sensor

[0205] 33 Tactile / Force Sensors

[0206] 40 Camera

[0207] 50 Cartons

[0208] 60 Pallets.

Claims

1. A visual system, characterized in that: have: an acquisition unit that acquires a first image in which a plurality of workpieces are present; A detection unit, which outputs detection results of a plurality of the workpieces; a robot that performs an action to change the position of at least one of the plurality of workpieces, The robot performs a first action of changing the positions of the plurality of workpieces, The acquisition unit acquires a second image of the region where the plurality of workpieces exist after the first operation is performed, The detection unit detects the first image and the second image, and outputs at least one of the positions, postures and shape information of the plurality of workpieces as the detection result based on at least an amount of change between the first image data including the detection result and the second image data including the detection result.

2. The visual system according to claim 1, characterized in that The detection unit outputs the detection result of the plurality of workpieces based on a difference between at least one of position, posture, and shape information of the plurality of workpieces included in the first image data and the second image data.

3. The visual system according to claim 1 or 2, characterized in that: The robot performs a second action different from the first action of changing the positions of the plurality of workpieces. The acquisition unit acquires a third image of the plurality of workpieces after the second operation is performed. The detection unit detects the third image and outputs the detection result based on a change amount between at least any two of the first image data, the second image data, and the third image data including the detection result.

4. The visual system according to any one of claims 1 to 3, characterized in that: The robot comprises: a force measuring unit for measuring contact information when the robot contacts the plurality of workpieces; The robot performs a contact operation with the surfaces of the plurality of workpieces based on the detection result of at least one of the first image and the second image. The force measuring section measures the contact information of the surface.

5. The visual system according to claim 4, characterized in that: The robot performs at least the first operation on the plurality of workpieces based on the contact information.

6. The visual system according to claim 4 or 5, characterized in that: The contact information includes at least one of information related to the presence or absence of contact, information related to the contact position, information related to the magnitude, direction and distribution of the contact force, information related to the magnitude and direction of the contact torque, and information related to sliding.

7. The visual system according to any one of claims 4 to 6, characterized in that: The detection unit corrects the detection result based on at least information related to a contact position among the contact information.

8. The visual system according to claim 6, characterized in that: The detection unit corrects at least one of the first image, the second image, and the detection result based on information on a change amount of a magnitude and a direction or a distribution of the contact force.

9. The visual system according to any one of claims 1 to 8, characterized in that: The visual system further includes a display unit that displays the detection result.

10. The visual system according to claim 9, characterized in that: The display unit displays the first image or the second image acquired by the acquisition unit together with the detection result.

11. A visual inspection method, characterized in that: have: An acquisition step of acquiring a first image of an area where a plurality of workpieces exist; A detection step, outputting detection results of a plurality of the workpieces; an execution step, causing the robot to execute an action of changing the position of at least one of the plurality of workpieces, The execution step causes the robot to execute a first action of changing the positions of the plurality of workpieces. The acquiring step acquires a second image of the region where the plurality of workpieces exist after the first operation is performed. The detection step detects the first image and the second image, and outputs at least one of the positions, postures and shape information of the multiple workpieces as the detection result based on at least the change between the first image data containing the detection result and the second image data containing the detection result.

Images