Workpiece detection device and workpiece detection method

Abstract

To provide a workpiece detection device and a workpiece detection method capable of detecting even a workpiece stored in a container in which a mapping sensor cannot be used, eliminating the need for storing a large amount of reference data, and surely detecting a deflection state of the workpiece in a short time from an image of the workpiece photographed by an imaging means.SOLUTION: A workpiece detection method of the present invention includes a step of photographing a workpiece by a camera unit, a step of generating a secondary image by extracting an edge from an image obtained by the photographing step using a determination model, a step of determining a contour of the workpiece from the secondary image, and a step of fitting the contour of the workpiece to a deflection function.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a detection device and a detection method for detecting the state of a thin plate-like workpiece stored in a container. [Background technology]

[0002] In the semiconductor manufacturing process, to prevent foreign matter from adhering to the fine electrical circuits on wafers, wafers are placed on the top surfaces of multiple shelves called teeth formed inside a clean container called a FOUP (Front-Opening Unified Pod) and transported between processes in a sealed state. Once transported to the desired process, the FOUP is fixed onto a stage called a load port installed in the processing equipment, and the door that hermetically closes the FOUP is opened. At this time, a mapping sensor installed in the load port detects the storage status of the wafers stored in the FOUP.

[0003] A typical mapping sensor includes a roughly U-shaped frame housing a light emitter and a light receiver of a transmitted light sensor, an advance / retract mechanism for moving the frame toward and away from the opening of the FOUP, and an elevation mechanism for raising and lowering the frame from the top to the bottom of the FOUP while the frame is inside the FOUP. The light emitter and receiver are fixed to the frame so that they sandwich the wafers stored in the FOUP from the left and right. As the frame moves from the FOUP state to the bottom, the mapping sensor detects whether the wafers are properly placed on each tooth in the FOUP. When a FOUP stores disc-shaped workpieces such as wafers, conventional mapping sensors such as those described above can detect the presence or absence of the wafers by inserting the light emitter and receiver of the transmitted light sensor into the space between the outer periphery of the wafer and the wall of the FOUP.

[0004] Meanwhile, in recent years, panel-level packaging (PLP) technology has become popular, reducing manufacturing costs by collectively manufacturing multiple semiconductor packages by densely arranging numerous semiconductor chips on a panel. PLP technology uses rectangular panels, for example, 600 mm x 600 mm, which are housed in rectangular PLP containers and transported between processes. The specifications for PLP containers and PLP panels are defined in SEMI (Semiconductor Equipment and Materials International) Standard E181, an international standard for semiconductor manufacturing equipment and materials. However, when detecting the storage status of PLP panels housed as workpieces in PLP containers conforming to this standard, the narrow gap between the wall of the PLP container and the PLP panel makes it difficult to use a mapping sensor equipped with a light emitter and a light receiver, as described above.

[0005] Furthermore, as disclosed in Patent Document 1, workpiece detection devices have been devised that are equipped with an imaging means such as a CCD camera instead of the mapping sensor that conventionally detects using detection light, and that determine the presence or absence of workpieces and the storage state from images captured by the imaging means. Patent Document 1 discloses a method for determining whether the storage state of a workpiece is normal or abnormal by comparing image data obtained after performing necessary preprocessing such as known binarization processing on image signals captured by a CCD camera with reference data that serves as a standard for determining whether the storage state is normal or abnormal. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 11-354609 Summary of the Invention [Problem to be solved by the invention]

[0007] When detecting workpieces from image data, a method has been used in which a large number of template images created and registered pixel by pixel in advance are saved, and the similarity between the captured image and the template images is measured to find the image with the highest similarity, thereby identifying the state of the workpiece. This method requires the image processing device to store a large amount of comparison data in order to determine the storage state of workpieces such as wafers. Furthermore, when comparing images, processing takes time, which ultimately has the problem of adversely affecting the throughput of the device.

[0008] Furthermore, there is a major obstacle to applying this image-comparison workpiece detection device to PLP panels. PLP panels are larger and heavier than standard 300mm diameter wafers, which causes significant bending when stored in a PLP container. As a result, a large amount of reference data is required to accurately determine the placement status of the PLP panel from the captured image. Also, capturing the entire PLP panel, which is large, requires taking multiple images, which increases the amount of data required for image detection.

[0009] The present invention has been made in consideration of the above problems, and aims to provide a workpiece detection method that can detect workpieces even when they are stored in a container in which a mapping sensor cannot be used.Furthermore, it is an object of the present invention to provide a workpiece detection device and a workpiece detection method that can reliably detect the deflection state of a workpiece in a short time from an image of the workpiece captured by an imaging means, without the need to store a large amount of reference data. [Means for solving the problem]

[0010] The work detection device of the present invention, devised to solve the above problems, is a work detection device that detects the storage state of work stored in a container with multiple shelves arranged vertically, and is equipped with a camera unit that photographs the work, and a processing device that calculates the deflection state of the work from the image of the work photographed by the camera unit, and the processing device is equipped with at least a communication unit that communicates with the camera unit, a memory unit that stores a program for detecting the work and a judgment model for judging the work, and an image processing unit that detects the deflection state of the work from the image of the work, and is characterized in that it uses the judgment model to detect the work and the background from the image of the work to generate a secondary image of the work, and fits a deflection function that matches the deflection of the work from the secondary image.

[0011] In addition, in the workpiece detection device of the present invention, the arithmetic processing device further includes an AI learning unit, which generates a judgment model by machine learning using training data. Here, the AI ​​learning unit may use the training data to generate a judgment model that distinguishes between the edge of the workpiece and the background of the container in which the workpiece is placed.

[0012] In addition, the processing device provided in the workpiece detection device of the present invention further includes a movement amount calculation unit, which is characterized by calculating the position of the workpiece from the position information of the camera unit and a deflection function fitted to the secondary image.

[0013] Furthermore, the operation amount calculation unit is characterized by calculating correction data for correcting a previously taught position of the transport robot that transports the workpiece.

[0014] The deflection function used in the workpiece detection device of the present invention is characterized by being a quadratic or higher even-order function. [Effects of the Invention]

[0015] With the above configuration, even if the workpieces are stored in a container where a mapping sensor cannot be used, the state and position of the workpieces stored in the container can be detected with high accuracy. Furthermore, since a judgment model generated by machine learning is used, there is no need to store a large amount of reference data, and the deflection state of the workpiece can be detected in a short time from the image of the workpiece captured by the imaging means. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing a PLP container of this embodiment. [Figure 2] FIG. 2 is a diagram showing the load port of this embodiment. [Figure 3] FIG. 3 is a cross-sectional view showing an outline of the transport device of this embodiment from above. [Figure 4] FIG. 4 is a cross-sectional side view showing an outline of the transport device of this embodiment. [Figure 5] FIG. 5 is a diagram showing the workpiece transport robot of this embodiment. [Figure 6] FIG. 6 is a diagram showing the state of the workpieces housed in the PLP container. [Figure 7] FIG. 7 is a diagram showing a workpiece in which deflection has occurred. [Figure 8] FIG. 8 is a diagram showing a binarized workpiece image. [Figure 9] FIG. 9 is a diagram showing a state in which a line segment passing through the middle position of the workpiece has been fitted. [Figure 10] FIG. 10 is a diagram showing the detected deflection amount of the workpiece. [Figure 11] FIG. 11 is a diagram showing a workpiece in which deflection occurs. [Figure 12] FIG. 12 is a diagram showing a workpiece that is tilted and deflected. [Figure 13] FIG. 13 is a diagram showing a method for calculating the thickness of a workpiece. [Figure 14]FIG. 14 is a diagram showing the procedure for fitting to the deflection function to derive a mathematical formula according to the state of the workpiece. [Figure 15] FIG. 15 is a block diagram showing the configuration of a workpiece detection device. DETAILED DESCRIPTION OF THE INVENTION

[0017] An embodiment of the present invention will be described below with reference to the drawings. Note that the embodiment described below is for illustrative purposes only and does not limit the scope of the present invention. Therefore, a person skilled in the art may adopt an embodiment in which each or all of the elements are replaced with equivalents, and these embodiments are also included in the scope of the present invention.

[0018] As an embodiment to which the present invention is applied, a transfer device 5 will be described, which includes a load port 3 that places a PLP container 2 containing a PLP panel 1, which is a workpiece to be detected, on the PLP container 2 and opens and closes the lid 9 of the PLP container 2, and a transfer robot 4 that holds the workpiece 1 and transfers it to a predetermined destination. Fig. 1 is a diagram showing the PLP container 2 that contains the workpiece 1, and Fig. 2 is a diagram showing the load port 3 that opens and closes the PLP container 2 of this embodiment. Also, Figs. 3 and 4 are diagrams showing an outline of the transfer device 5, and Fig. 5 is a diagram showing the workpiece transfer robot 4 of this embodiment.

[0019] The PLP container 2 of this embodiment is composed of a box-shaped container body 7 that stores the workpiece 1, and a lid 9 that detachably closes an opening 8 formed on one side of the container body 7. A pair of left and right teeth 10 for horizontally placing the workpiece 1 are provided vertically at a fixed interval inside the container body 7. Note that the PLP container 2 of this embodiment has 12 rows of teeth 10 formed inside the container body 7. The bottom surface of the container body 7 is provided with an engagement hole (not shown) for engaging with a hook member 12 provided on the stage 11 of the load port 3, and a positioning member (not shown) for placing the container at a predetermined position relative to the stage 11. Furthermore, the top surface of the container body 7 is provided with an automation flange 13 for gripping the container 2 by an OHT (Overhead Hoist Transport), a device that automatically transports the container 2 between processes in a semiconductor manufacturing factory. The lid 9 is a member that seals the container body 7, and the outer surface of the lid 9 is formed with a latch hole 14 for engaging with a latch key provided on the door of the load port 3, and a positioning hole 15 for positioning relative to the load port door.

[0020] The workpiece 1 stored in the container 2 of this embodiment is a PLP panel 1 applied to PLP (Panel Level Packaging) technology, and is a thin rectangular substrate with a side length of 600 mm. Various materials such as glass, stainless steel, and copper are used for the PLP panel 1, and for illustrative purposes, this embodiment uses a glass PLP panel 1. In this embodiment, the PLP panel 1 used is 1 mm thick.

[0021] Next, the load port 3 of this embodiment will be described. The load port 3 of this embodiment is a device for placing a PLP container 2 that stores PLP panels 1 on and opening and closing the lid 9 of the PLP container 2, and is equipped with a base plate fixed to the frame 6 of the transfer device 5, a stage 11 that fixes the container 2 in a predetermined position and moves it back and forth, an opening 16 formed in the base plate through which the workpiece 1 stored in the container 2 passes, a door 17 that can close this opening 16 and is integrated with the lid 9 of the container 2 to lock and unlock the lid 9 and the container body 7, a stage advance / retract mechanism (not shown) that moves the stage 11 back and forth, and a door lift / lowering mechanism (not shown) that moves the door 17 up and down.

[0022] A cylindrical positioning pin called a kinematic pin is erected at a predetermined position on the stage 11. This positioning pin has a hemispherical top, and when the container 2 is placed on this positioning pin, the top of the positioning pin and a positioning member formed on the bottom of the container 2 are guided by an automatic centripetal action, supporting the container 2 so that it is in a horizontal position. The container 2 is also accurately positioned in the left-right and front-back directions. The stage 11 is also provided with a hook member 12, which engages with an engaging hole formed on the bottom of the positioned container 2 to secure the container 2 to the stage 11. A locking mechanism (not shown) is provided inside the door 17 for locking and unlocking the lid 9 to the container 2.

[0023] The stage advancement / retraction mechanism includes a guide mechanism that guides the stage 11 horizontally and a motor that moves the stage 11 horizontally. Rotation of the motor advances and retracts the stage 11 to any desired position. The door lifting / lowering mechanism includes a guide mechanism that guides the door 17 vertically and a motor that moves the door 17 vertically. Rotation of the motor lifts and lowers the door 17 to any desired position. The motors included in the load port 3 of this embodiment are preferably motors capable of controlling the angle of their output shafts, such as stepping motors. Each motor is preferably equipped with an encoder that detects the rotation angle of its output shaft. The operation of the stage advancement / retraction mechanism, door lifting / lowering mechanism, and mechanism for locking and unlocking the lid 9 and container body 7 are controlled by a load port controller 18 included in the load port 3. The load port controller 18 is electrically connected to the transport device 5 and operates in response to control signals from the transport device 5.

[0024] Next, the transfer robot 4 and the X-axis table 19 will be described. The transfer robot 4 of this embodiment is a horizontal articulated SCARA robot, and the base 26 of this transfer robot 4 is fixed via a bracket 20 to a slider of the X-axis table 19, which moves the transfer robot 4 linearly in the X direction within a horizontal plane. The X-axis table 19 includes a linear guide mechanism that guides the slider in the X direction within the horizontal plane, and a travel drive motor 21 that rotates the linear guide mechanism. In this embodiment, a motor capable of controlling the angle of its output shaft, such as a stepping motor, is used as the travel drive motor 21. The travel drive motor 21 is also equipped with an encoder that detects the rotation angle of the output shaft. The travel drive motor 21 and the encoder are electrically connected to a robot controller 33 (not shown). With this configuration, the drive shaft of the travel drive motor 21 of the X-axis table 19 is rotated forward or backward by a predetermined rotation angle, allowing the transfer robot 4 to move to a predetermined position. The X-axis table 19 is also provided with a posture adjustment mechanism (not shown), and the X-axis table 19 is adjusted so as to move the transfer robot 4 within a horizontal plane.

[0025] The transport robot 4 of this embodiment includes at least a finger 22 that holds the workpiece 1, an arm body 23 that supports the finger 22 and performs an extension / retraction motion to move the finger 22 to a predetermined position in a horizontal plane, a body 24 that rotates and moves the arm body 23 in the horizontal plane, a robot lifting mechanism 25 that moves the body 24 up and down in the vertical direction, and a base 26 to which the robot lifting mechanism 25 is fixed.

[0026] A robot lifting mechanism 25 that supports the body 24 and an elevation drive motor that drives the robot lifting mechanism 25 are attached to the base 26. Operation of this elevation drive motor causes the body 24 to move up and down in the vertical direction. The body 24 is attached to the robot lifting mechanism 25 via bearings to enable rotational movement within a horizontal plane, and the body 24 is further equipped with a rotary drive motor and a reducer that rotate the body 24. When the output shaft of the rotary drive motor rotates forward and backward, the body 24 rotates clockwise and counterclockwise within a horizontal plane around a central axis C1 extending in the vertical direction. The base 26 is also equipped with an attitude adjustment mechanism that can adjust the base 26 to maintain a horizontal attitude. The robot lifting mechanism 25 is also equipped with an attitude adjustment mechanism that can adjust the robot lifting mechanism 25 to move up and down in the vertical direction.

[0027] An arm 23 is rotatably attached to the upper part of the body 24. In this embodiment, the arm 23 is composed of a lower arm 23a, the base end of which is rotatably attached to the body 24 via a bearing, and an upper arm 23b, the base end of which is rotatably attached to the tip of the lower arm 23a via a bearing. A finger 22 is rotatably attached to the tip of the upper arm 23b via a bearing. The lower arm 23a is rotatable in a horizontal plane around a central axis C1 extending vertically. The upper arm 23b is rotatable in a horizontal plane around a central axis C2 extending vertically. The finger 22 is rotatable in a horizontal plane around a central axis C3 extending vertically. The body 24 is also provided with an attitude adjustment mechanism, which allows the body 24 to be adjusted so that it is horizontal and rotatable in a horizontal plane.

[0028] An arm drive motor disposed inside the body 24 is connected to the base end of the lower arm 23a via a reducer, and as the output shaft of the arm drive motor rotates forward or backward, the lower arm 23a rotates clockwise or counterclockwise in a horizontal plane with respect to the body 24, with the central axis C1 as the center of rotation, at a predetermined rotation ratio. Pulleys and belts are provided inside the lower arm 23a and the upper arm 23b to transmit the driving force of the arm drive motor to the upper arm 23b and the holder 27. The rotation ratio of the pulleys disposed at each of the first joint 29, which is the connection portion between the body 24 and the lower arm 23a, the second joint 30, which is the connection portion between the lower arm 23a and the upper arm 23b, and the third joint 31, which is the connection portion between the upper arm 23b and the holder 27, is adjusted by the diameter ratio of these pulleys, and the rotation ratio of the pulleys at the first joint 29, the second joint 30, and the third joint 31 is adjusted to be 1:2:1. With the above configuration, the arm body 23 bends and stretches in a horizontal plane as the output shaft of the arm drive motor rotates forward and backward, and the holding part 27 attached to the tip of the arm body 23 moves forward and backward in a straight line in the horizontal plane without changing its posture. The motors included in the transport robot 4 of this embodiment are preferably motors capable of controlling the angle of their output shafts, such as stepping motors. Each motor is preferably equipped with an encoder that detects the rotation angle of the output shaft. Each motor and each encoder are electrically connected to a robot controller 33 included in the transport robot 4, which controls the operation of each motor and driving mechanism in accordance with a pre-stored operation program, position data, and speed data. The lower arm 23a and the upper arm 23b are equipped with posture adjustment mechanisms that adjust the lower arm 23a and the upper arm 23b to a horizontal posture and to be rotatable in a horizontal plane. The robot controller 33 is also electrically connected to the transport device 5 and operates in response to control signals from the transport device 5.

[0029] The finger 22 is composed of a holding portion 27 that holds the workpiece 1 and a wrist portion 28 that supports the base end of the holding portion 27. The finger 22 in this embodiment is a suction finger that secures the workpiece 1 on the finger 22 by vacuuming the workpiece 1 while it is lifted from below, and is connected to a vacuum source provided in the clean room via piping members (not shown). The holding portion 27 in this embodiment is made by bonding together an upper member and a lower member that are formed into a bifurcated V shape, and a workpiece suction pad 32 is provided at a predetermined position on the upper member to come into contact with the underside of the workpiece 1 and suck and hold the workpiece 1. The finger 22 is also provided with an attitude adjustment mechanism that adjusts the finger 22 to a horizontal position.

[0030] With the above configuration, a container 2 carried from a previous process by a transport device installed in a semiconductor manufacturing factory is placed on the stage 11 of the load port 3 provided in the transport device 5, and the lid 9 of the container 2 is removed by the load port 3, making the workpiece 1 stored in the container 2 accessible to the transport robot 4. Once the workpiece 1 is accessible by the transport robot 4, the transport robot 4 operates the arm body 23, the body 24, and the lifting mechanism to insert the fingers 22 below the target workpiece 1, lift and hold the workpiece 1, and then move the workpiece 1 through the internal space 39 of the transport device 5 to transport it to the processing equipment. Note that this series of operations is controlled by a control device (not shown) provided in the transport device 5.

[0031] The transfer device 5 also includes a space surrounded on all four sides by partition members made up of a frame 6 and a cover, and an FFU (Fun Filter Unit) is mounted on the ceiling. The FFU filters air introduced by the rotation of a fan and supplies the resulting clean air to an internal space 39 of the transfer device 5 in which the transfer robot 4 and other components are located. The downflow of clean air supplied from this FFU exhausts dust generated by the operation of the transfer robot 4 and other components from the internal space 39 of the transfer device 5 to the outside, and the internal space 39 of the transfer device 5 is always maintained in a clean atmosphere.

[0032] In this embodiment, the wrist 28 of the transfer robot 4 is equipped with a camera unit 34, which is one of the units constituting the workpiece detection device 40. This camera unit 34 photographs the workpiece 1 stored inside the container 2 and detects the presence or absence of the workpiece 1 and its storage state from the photographed image. The camera unit 34 includes a camera module 35, an illumination module 36, and a communication module 37. The camera module 35 includes an optical lens and an image sensor. The image sensor is composed of semiconductor elements such as a CMOS (Complementary Metal Oxide Semiconductor). The camera module 35 uses the optical lens to focus an image of the subject on each element (cell) of the image sensor. The images obtained by each element are converted into image data and transmitted to the arithmetic processing unit 38 via the communication module 37. The illumination module 36 includes an LED light as a light source, and the LED light illuminates the area 49 photographed by the camera module 35. See FIG. 6(a). The illumination module 36's LED light is turned on and off based on a light source on / off signal received by the communication module 37.

[0033] In the workpiece detection device 40 of this embodiment, the horizontal arrangement of the elements constituting the image sensor in the horizontal direction (X direction) is the u-axis, and the vertical arrangement perpendicular to the X direction (Z direction) is the v-axis. Each element of the image sensor is individually assigned a u-axis coordinate. The u-axis of the image formed by the image sensor is set parallel to the horizontal direction defined by the X-axis table 19, and the v-axis is set parallel to the vertical direction defined by the transport robot 4 and the load port 3. In this embodiment, image data captured by the camera module 35 is transmitted in real time to the arithmetic processing unit 38 of the transport device 5 via the communication module 37 also provided in the camera unit 34. Video capture by the camera module 35 is initiated upon receiving a capture start signal transmitted from the arithmetic processing unit 38. The camera module 35 used in this embodiment is fixed to the wrist unit 28 with the optical axis L2 of the mounted lens adjusted to be horizontal. Furthermore, the optical axis L2 of the lens is adjusted so that it coincides in the horizontal plane with a straight line L3 that connects the center point of the workpiece 1 and the central axis C1, which is the center of rotation of the body 24 of the transport robot 4, when viewed from above. The positions of the straight line L3 and the optical axis L2 are known values ​​from the design point of view.

[0034] 15 is a block diagram showing the configuration of the workpiece detection device 40 of this embodiment. In the workpiece detection device 40 of this embodiment, the arithmetic processing device 38 determines the area occupied by the workpiece 1 by distinguishing between the workpiece 1 and the background from the image captured by the camera unit 34, and then detects the contour and position of the workpiece 1 by curve fitting this area. The calculation processing device 38 of this embodiment is provided in the work detection device 40, and includes at least a calculation unit 42 that executes a program, a communication unit 43 that communicates with the camera unit 34, robot controller 33, and load port controller 18 mounted on the transport robot 4, a memory unit 44 that stores image data and position teaching programs received by the communication unit 43, position teaching data and various setting data obtained by executing the position teaching program, a display unit 45 that displays images and teaching data captured by the camera unit 34, an image processing unit 46 that detects the deflection state of the work 1 from the image captured by the camera unit 34, an operation amount calculation unit 47 that calculates the operation amount of the transport robot 4 from the deflection state of the work 1 detected by the image processing unit 46, and an input unit 48 into which an operator inputs operation commands, etc.

[0035] Next, the position teaching operation for teaching the transfer robot 4 the transfer position of the workpiece 1 will be described. The worker operates the input unit 48 to execute the position teaching program stored in the arithmetic processing device 38, thereby teaching the position of the transfer robot 4. The position teaching operation is performed to teach the transfer robot 4 and the load port 3 an operating position after the assembly of the transfer device 5 is completed, and by executing this position teaching, the transfer robot 4 can safely transfer the workpiece 1 stored in the container 2 from the container 2 to the processing device. The position teaching operation is performed manually by the worker according to the position teaching program, or is automatically performed by a dedicated automatic teaching program. The position teaching data obtained by executing this position teaching program is stored in the memory unit 44.

[0036] Here, for each container 2 brought in from the previous process, the transfer device 5 needs to know which tooth 10 of the container 2 the stored workpiece 1 is placed on and in what state the placed workpiece 1 is placed. In conventional transfer devices 5, the placement state of the workpiece 1 is detected by a transmitted light sensor mounted on a mapping device provided in the load port 3, but in the container 2 that stores the rectangular workpiece 1 of this embodiment, there is no space to insert a transmitted light sensor, so a transmitted light sensor cannot be used. Furthermore, the workpiece 1 of this embodiment is packaged with many semiconductor chips mounted on the surface of a thin glass substrate, and in some cases is filled with sealing material, which makes it heavier than conventional semiconductor wafers.

[0037] In particular, if the workpiece 1 is a PLP panel 1, the workpiece 1 is large, with a side length of approximately 600 mm, and is heavy, resulting in a significant amount of deflection due to the workpiece's own weight. Furthermore, because the amount of deflection varies depending on the workpiece 1, accurately detecting the amount and state of deflection of the workpiece 1 is extremely important when detecting the workpiece 1. For example, as shown in FIG. 6(a), in the case of a workpiece 1 with relatively small deflection, the fingers 22 can advance to a pre-taught position and hold the workpiece 1. However, as shown in FIG. 6(b), in the case of a workpiece 1a with a large deflection, when the fingers 22 advance to the pre-taught position, the holding portions 27 of the fingers 22 come into contact with the large deflection of the workpiece 1a, damaging the workpiece 1a. Therefore, in the workpiece detection device 40 of this embodiment, the operation amount calculation unit 47 detects the deflection state of each workpiece 1, 1a. If the amount of deflection of the large deflection of the workpiece 1a is greater than the specified amount of deflection, a signal indicating that transport is not possible is sent to the control device, and transport is halted. Furthermore, the operation amount calculation unit 47 detects the deflection state of each workpiece 1, 1a from the captured image, and if it is determined that even if the workpiece 1a is significantly deflected, it is within the range that can be transported by the transport robot 4, it calculates a correction amount according to the deflection state of the workpiece 1a from the previously taught position and calculates a new workpiece holding position.

[0038] Next, the workpiece detection method of this embodiment will be described. The workpiece detection method of this embodiment detects the deflection of the workpiece 1, 1a from a photographed image of the workpiece 1, 1a, and fits the deflected state to a deflection function expressed as an even-order polynomial, such as a quadratic or quartic polynomial, to derive a mathematical formula corresponding to the state of the workpiece 1, 1a. In this embodiment, the program that determines the contour of the workpiece 1, 1a from the photographed image of the workpiece 1, 1a improves the contour detection accuracy by machine learning a large amount of training data. Furthermore, the program that fits the deflection function to the determined contour uses a known curve fitting program.

[0039] The image processing unit 46 uses a determination model input in advance by the operator into the memory unit 44 provided in the arithmetic processing device 38 to determine the area occupied by the workpiece 1, 1a and the area occupied by the background in the area detection step from a primary image of the workpiece 1, 1a. This determination model is strengthened to a high accuracy by the AI ​​(Artificial Interigence) processing unit 41 provided in the arithmetic processing device 38 performing machine learning using training data. This improves the accuracy of edge extraction of the workpiece 1, 1a. The AI ​​learning unit 41 of this embodiment improves the detection accuracy of the captured image by performing machine learning in advance using multiple images taken of the workpiece 1, 1a in a state where it is placed on the teeth 10 and deflected as training data.

[0040] In the workpiece detection method of this embodiment, training data is prepared using a primary image of the workpiece 1, 1a and the background where the deflection has occurred, and a secondary image in which the area corresponding to the workpiece 1, 1a is colored with a predetermined color to define the outline of the workpiece 1, 1a. The AI ​​learning unit 41 learns from these images. A large number of these primary and secondary images are prepared by the operator and stored as training data in the memory unit 44 of the arithmetic processing device 38. The stored training data is then loaded into the AI ​​learning unit 41 for learning. Here, the primary image contains an area depicting the workpiece 1, 1a and an area depicting the background. The background area reflects the interior of the container body 7, and the depiction of the interior of the container body 7 varies significantly depending on the environment in which the container body 7 is placed. For example, if illumination reaches the interior of the container body 7, it is depicted as a relatively bright area, whereas if illumination does not reach the interior of the container body 7, it is depicted as a relatively dark area. Furthermore, these areas are out of focus on the camera 34, resulting in a blurred depiction, and the background contains a variety of patterns. Furthermore, the workpieces 1 and 1a also have various patterns of image shading and brightness depending on their placement positions. Therefore, by loading various training data into the AI ​​learning unit 41, the areas of the workpieces 1 and 1a and the background areas can be accurately detected. Note that, although the AI ​​learning unit 41 in this embodiment is implemented in the arithmetic processing device 38, the present invention is not limited to this. For example, it is entirely possible for an operator to train the AI ​​learning unit 41 in advance before implementing it in the arithmetic processing device 38. Furthermore, the AI ​​learning unit 41 may be implemented in the transport robot 4 as a unit separate from the arithmetic processing device 38.

[0041] The data generated by the AI ​​learning unit 41 through machine learning is stored in the storage unit 44 as a judgment model to be used by the image processing unit 46. Note that it is desirable to prepare primary images that the AI ​​learning unit 41 uses in machine learning, which are taken under various conditions, such as lighting intensity, amount of bending, and image clarity.

[0042] Furthermore, for example, as shown in FIG. 8(a), if the difference between the Z coordinates of both ends of the workpieces 1 and 1a is greater than a predetermined threshold, the image processing unit 46 fits the deflection state of the workpieces 1 and 1a to a deflection function based on the judgment model generated by the AI ​​learning unit 41, detects that the workpieces 1 and 1a are placed on teeth 10 at different heights on the left and right, and determines that they cannot be transported. Furthermore, as shown in FIG. 8(b), if a colored dot 50 is found in an area other than the outline of the workpieces 1 and 1a, the dot 50 is determined to be unrelated to the workpieces 1 and 1a, and the drawing data for the dot 50 is deleted. Furthermore, as shown in FIG. 8(c), if there is an area 51 in which the edges of the workpieces 1 and 1a are drawn as if they are separated due to lighting, and the area 51 is determined to be part of the workpieces 1 and 1a, the image data is converted so that the area 51 is included in the area of ​​the workpieces 1 and 1a.

[0043] 14 is a block diagram showing the procedure for fitting a deflection function to an image of the workpiece 1, 1a in which deflection has occurred, and deriving the function as a function corresponding to the state of the workpiece 1, 1a, in the workpiece detection method of the present invention. The method for deriving the function includes a step of photographing the workpiece 1, 1a (STEP 1), an area detection step (STEP 2) that detects the area of ​​the workpiece 1, 1a and the area of ​​the background from the image of the workpiece 1, 1a, a deflection function fitting step (STEP 3) that fits a function representing the deflection to the detected contour data, and a workpiece thickness calculation step (STEP 4) that calculates the thickness of the workpiece 1, 1a. Furthermore, a teaching data correction step (STEP 5) is also performed in which the position information of the workpiece 1, 1a calculated in the above steps is reflected in the position teaching data of the transfer robot 4 to calculate corrected position data.

[0044] Detection of the workpieces 1, 1a begins with a step (STEP 1) in which the camera unit 34 provided on the transfer robot 4 photographs the workpieces 1, 1a stored in the container 2. The photographing is performed when the transfer robot 4 has moved to a position where the optical axis of the camera module 35 directly faces the opening 8 of the container 2 placed on the load port 3. FIG. 7 is a secondary image showing the workpieces 1, 1a stored in the container 2 placed on the stage 11. This secondary image shows the workpieces 1, 1a in a state where they are deflected. Here, in the workpiece detection method of this embodiment, the camera unit 34 does not need to photograph the entire workpieces 1, 1a; as shown in FIG. 6(a), it is sufficient to photograph only the area 49 surrounded by the dashed line. Note that the camera unit 34 of this embodiment uses a telephoto lens with a relatively long focal length. By using a telephoto lens with a shallow depth of field, the background other than the edge portions of the workpieces 1 and 1a that are in focus is blurred, and the boundary between the edge portions and the background is made clear, making it easier to determine the edge portions.

[0045] The image captured by the camera unit 34 in this embodiment is a video, and the camera unit 34 moves upward as the robot lifting mechanism 25 of the transfer robot 4 moves upward, capturing images by scanning the area 49 surrounded by the dashed line from below to above. The distance between the fingers 22 of the transfer robot 4 and the container 2 placed on the stage 11 is a known value, and the horizontal (X-direction) and vertical (Z-direction) positions of each tooth 10 formed on the container 2 are also known values ​​by design. Furthermore, since the lifting speed of the robot lifting mechanism 25 that lifts the camera unit 34 is also known, the position (coordinates) of each pixel in the captured image is also a calculable value. Furthermore, since the rotation angle of the motor that lifts the transfer robot 4 can be controlled by the robot controller 33, the position information of each pixel in the X-direction and Z-direction in the captured image can be calculated from the rotation angle information of each motor.

[0046] Next, the image processing unit 46 executes a step (STEP 2) of detecting the area of ​​the workpiece 1, 1a and the area of ​​the background from the image of the workpiece 1, 1a described above, and generates a secondary image of the workpiece 1, 1a.

[0047] Next, the image processing unit 46 executes a step (STEP 3) in which the deflection state of the workpieces 1, 1a is fitted to a deflection function from the secondary image of the workpieces 1, 1a. In this embodiment, a curve LB passing through the midpoint of the photographed workpieces 1, 1a is fitted to the entire area of ​​the workpiece from the binarized secondary image of the workpieces 1, 1a. Furthermore, the position of the workpiece 1 when placed on the screen without deflection is defined by a straight line LA, and the difference from this reference line LA is calculated as the detected deflection amount D1 of the workpieces 1, 1a. Furthermore, the image processing unit 46 converts the number of pixels into a millimeter value for the detected deflection amount D1, referring to a preset image resolution. See FIG. 10.

[0048] The procedure for determining the deflection state of the workpieces 1, 1a is as follows: First, a straight line that is the center line that vertically bisects the workpieces 1, 1a, which are supported by a pair of left and right teeth 10 and are not deflected, is defined as a reference line LA. Next, a curve LB that vertically bisects the workpieces 1, 1a, which are supported by a pair of left and right teeth 10 and are deflected, is defined. The distance D1 in the v direction on the screen between the curve LB and the reference line LA at the position where the workpieces 1, 1a are deflected most downward is determined, and this distance D1 is determined as the amount of deflection D1 specific to each workpiece 1, 1a.

[0049] Next, a function that represents the deflection state of the workpieces 1 and 1a will be described. First, the formulation of a function that approximates the annotation and deflection curve from the captured image will be described. The function that approximates the deflection in this embodiment is (Number 1) Y=f(x)g(x)+rx+yo Here, Y is the distance in the v direction, f(x) is a function that represents the deflection curve of the workpieces 1 and 1a in which deflection occurs symmetrically, g(x) is a function that reads the asymmetry when deflection occurs asymmetrically, rx is a term that represents the slope of the deflection that occurs, and yo is a term that represents the amount of positional deviation when a positional deviation occurs in the vertical direction (Y direction) of the workpieces 1 and 1a. The method for deriving the above functions will be explained in detail below.

[0050] First, let us consider the horizontal coordinate u and the vertical coordinate v of the image of the workpiece 1, 1a. It is desirable that the image of the substrate does not capture the entire substrate, but rather captures the approximate center of the substrate. When the camera unit 34 is mounted on the transport robot 4, capturing the entire workpiece 1, 1a requires a wide-angle lens capable of capturing a wide range. However, with such a wide-angle lens, the larger the capture range, the fewer pixels in the portion depicting the workpiece 1, 1a, making the image unclear and making it impossible to accurately calculate its position. Therefore, in this embodiment, a telephoto lens with a relatively long focal length is used to capture only the approximate center of the substrate. In this embodiment, the number of pixels arranged in each column of the horizontal axis u and vertical axis v on the screen is 256. See Figure 10.

[0051] To normalize the captured image data, the horizontal axis X and vertical axis Y must be redefined using the following equations: (Number 2) X=u / max(u)-1 / 2 (Number 3) Y=v / max(v) Using the above formula, X is scaled as a value ranging from a minimum of -1 / 2 to a maximum of 1 / 2, and Y is scaled as a value ranging from a minimum of 0 to a maximum of 1. As a result, the X direction (u direction) coordinate of a specific position displayed on the image will be a value between -1 / 2 (-0.5) and +1 / 2 (0.5), and the Y direction (v direction) coordinate will be a value between 0 and 1. Here, since workpieces 1 and 1a will bend downward due to gravity, the axis of Y (v) is chosen so that the value increases in the downward direction. Note that in a board that is not misaligned, the bending will be nearly symmetrical, and the graph representing the bending will also be symmetrical, so it is desirable to center the X direction at 0.

[0052] Next, we define a function that represents the deflection in the normalized coordinate system for the X and Y directions. If the captured image of the board has an area that is 1 / L times larger than the entire board, and the center position of the workpieces 1 and 1a on the screen is at position Lx0 relative to the screen, the coordinates when the board is used as the reference can be expressed by the following formula. (Number 4) x=X / L-x0 Here, the actual dimensions of the workpieces 1 and 1a are L times the size of the image, and x is the position on the workpieces 1 and 1a, representing the workpieces 1 and 1a including the areas not depicted in the image. Using the above formula, when x = -1 / 2, the coordinate of the left edge of the board can be obtained, and when x = 1 / 2, the coordinate of the right edge of the board can be obtained. See Figure 11(a).

[0053] Here, the workpieces 1, 1a are supported at both ends by the teeth 10 of the container 2 at the same height, and the center area is bent downward symmetrically due to gravity. Therefore, the calculation formula for describing the bending curve of the substrate can be any even-order polynomial function. However, considering the calculation processing speed and the load on the arithmetic processing device 38, it is preferable to use a quadratic or quartic function. Furthermore, for workpieces 1, 1a that are already distorted due to distortion caused by coating the surface of the workpieces 1, 1a, thermal stress caused by heat treatment, or residual stress caused by polishing, the workpieces 1, 1a will bend in a complex manner when supported by the teeth 10, making it difficult to approximate with a quadratic function. When fitting such already distorted workpieces 1, 1a, it is preferable to approximate with a function of fourth or higher order.

[0054] For example, assuming a curve on the Y axis that passes through the coordinates (-1 / 2, 0) and (1 / 2, 0) on both ends of the image, the deflection of the workpieces 1 and 1a can be expressed by the following quadratic function. (Number 5) h(x)=(X+1 / 2)(X-1 / 2) Furthermore, the function representing the deflection of the originally distorted workpieces 1 and 1a can be expressed by the following quartic function. (Number 6) h(x)=(X+1 / 2)(X-1 / 2){P-(1-P)(X+1 / 2)(X-1 / 2P)} Here, P is in the range of 0≦P≦1 and represents a value that expresses the ratio between the quadratic term and the quartic term of x. For example, if the deflection of the workpieces 1 and 1a cannot be adequately approximated by a quadratic function (P=1), the deflection of the workpieces 1 and 1a can be approximated by a quartic function by setting the value of P to a value other than 1. Note that the function that expresses the deflection is not limited to the quadratic and quartic functions described above; for example, a hexagonal or octagonal function is also computationally feasible. However, since the workpieces 1 and 1a are supported at both their left and right ends by the teeth 10, it is desirable that the order of the function be an even number.

[0055] Furthermore, by dividing h(x) by h(0) and multiplying it by k, we can derive a function that changes between 0 and k. (Number 7) f(x)=k(h(x) / h(0)) Here, k is a coefficient that represents the degree of deflection of the curve, and the larger the value of k, the greater the amount of deflection of the curve, and conversely, the smaller the value of k, the smaller the amount of deflection of the curve.The above calculation formula can be used to approximate the deflection of the workpieces 1 and 1a when the deflection of the workpieces 1 and 1a is symmetrical.

[0056] Note that the workpieces 1 and 1a do not necessarily bend symmetrically. For example, to avoid contact with the workpieces 1 and 1a during transport, the container 2 that stores the workpieces 1 and 1a has excess space on the left and right. If the workpieces 1 and 1a move into this excess space and become misaligned within the container 2, the center of the workpieces 1 and 1a may be displaced from the center position on the screen. Therefore, it is necessary to approximate the workpieces 1 and 1a that are misaligned and do not bend symmetrically on the screen. To approximate the deflection of the workpieces 1 and 1a that bend asymmetrically on the image, this function is multiplied by g(x), which depends on x, to create an asymmetric graph. (Number 8) g(x)=1+qx Note that g(x) must be a positive value in the range of -1 / 2 to 1 / 2 for the x coordinate so that the direction of the deflection represented by the function does not reverse, and therefore q must be in the range of -2 to 2.

[0057] Furthermore, due to manufacturing errors in the container 2 containing the workpieces 1 and 1a or errors in the leveling of the stage 11 on which the container 2 is placed, the levelness between the container 2 and the camera module 35 may not fall within the tolerance range. If the container 2 and the camera module 35 are not aligned horizontally, the relative tilt between the container 2 and the camera may occur in both the left-right and front-to-back directions with respect to the camera module 35. Therefore, the relative tilt between the camera module 35 and the workpieces 1 and 1a can be expressed as the tilt in the u direction corresponding to the left-to-right direction by adding the term rx, which represents the left-to-right tilt of the workpieces 1 and 1a, to the function. Furthermore, the tilt in the v direction can be expressed by adding the term yo, which represents the tilt in the front-to-back direction. For example, if the camera is tilted in the front-to-back direction with respect to the workpieces 1 and 1a, the image will be captured displaced upward or downward in the Y direction. Therefore, adding the term yo, which represents the tilt, makes it possible to approximate the image generated at the top or bottom of the screen. The term rx represents a straight line with a slope of r, so adding this term makes it possible to approximate even when the workpieces 1 and 1a are tilted in the left-right direction. See Figure 11(b). Also, yo is a term that represents the offset amount in the Y direction (v direction).

[0058] The need to add the Y-direction offset to the calculation formula is explained below. There are individual differences within the allowable range in the assembly accuracy of the load port 3 and transfer robot 4, and there are also individual differences within the allowable range when attaching them to the frame 6 of the transfer device 5. For example, when three load ports 3 are installed as shown in Figure 3, it is extremely difficult to fix all three load ports 3 at exactly the same height. Therefore, by adding a term representing the Y-direction (v-direction) offset to the formula, an offset amount is added so that the workpieces 1, 1a are drawn in the specified positions on the screen, even when multiple load ports 3 are installed at different positions relative to the transfer robot 4 in the vertical direction.

[0059] The above-described procedure makes it possible to fit a function that approximates the deflected workpieces 1, 1a. This allows the workpiece detection device 40 to detect the placement state and amount of deflection of the workpieces 1, 1a placed on the teeth 10, and therefore to detect the presence or absence of the workpieces 1, 1a and whether the transfer robot 4 can safely transfer the workpieces 1, 1a.

[0060] Furthermore, the distance in the v direction between the curve LB passing through the center of the workpiece 1, 1a and the reference line LA shown in Figure 9 is calculated. This makes it possible to determine how far the workpiece 1, 1a is bent downward from the top surface of the tooth 10 on which it is placed. Next, the number of pixels in the vertical direction on the screen depicting the workpiece 1, 1a on the secondary screen is counted to calculate the thicknesses T1 to T7 of the workpiece 1, 1a at each measurement point (see Figure 13). This makes it possible to accurately determine the position of the area occupied by the workpiece 1, 1a within the space inside the container 2. Furthermore, if the measured thickness of the workpiece 1, 1a exceeds a preset thickness threshold, it is determined that the workpieces 1, 1a are placed overlapping, and an alarm signal is sent.

[0061] Then, the movement amount calculation unit 47 calculates a corrected movement amount that allows the finger 22 to move forward safely without coming into contact with the workpiece 1, 1a from the calculated result, and corrects the pre-taught data. As a result, even if the workpiece 1a is significantly bent, if it is determined that it is within the range that the transport robot 4 can transport, the transport robot 4 can safely transport the significantly bent workpiece 1a by correcting the teaching data of the transport robot 4 to teaching data that corresponds to the significantly bent workpiece 1a.

Claims

1. A work detection device that detects the storage state of works stored in a container having a plurality of shelves arranged vertically, The workpiece detection device is A camera unit that photographs the workpiece; a processing unit that calculates a deflection state of the workpiece from an image of the workpiece photographed by the camera unit, The processing unit includes at least a communication unit that communicates with the camera unit; a storage unit that stores a program for detecting the workpiece and a judgment model for judging the workpiece; an image processing unit that detects a deflection state of the workpiece from an image of the workpiece, the calculation processing device uses the judgment model to detect the workpiece and the background from the image of the workpiece, and generates a secondary image of the workpiece; fitting a deflection function that matches the deflection of the workpiece from the secondary image; A workpiece detection device characterized by:

2. The workpiece detection device according to claim 1, The arithmetic processing device further includes an AI learning unit, The AI ​​learning unit performs machine learning using training data, The method is characterized by generating the judgment model.

3. The workpiece detection device according to claim 2, The AI ​​learning unit is characterized by using the training data to generate the judgment model that distinguishes between the edge of the workpiece and the background of the container in which the workpiece is placed.

4. The workpiece detection device according to claim 1, the arithmetic processing device further includes an operation amount calculation unit, The movement amount calculation unit calculates, from the position information of the camera unit and the deflection function fitted to the secondary image, The position of the workpiece is calculated.

5. The workpiece detection device according to claim 4, The operation amount calculation unit calculates correction data for correcting a previously taught position of the transport robot that transports the workpiece.

6. The workpiece detection device according to claim 1, The deflection function is characterized by being a function of even degree, which is equal to or greater than two.

7. A work detection method for detecting the storage state of works stored in a container having a plurality of shelves arranged vertically, comprising: The workpiece detection method includes: A step of photographing the workpiece by a camera unit; a step of determining edges from the image obtained by the step using a determination model to generate a secondary image; detecting a contour of the workpiece from the secondary image; fitting a deflection function that matches the contour; A workpiece detection method comprising:

8. The workpiece detection method according to claim 7, The determination model is characterized by being generated by machine learning using training data.

9. The workpiece detection method according to claim 8, The judgment model is characterized by using the training data to distinguish between the edge of the workpiece and the background of the container in which the workpiece is placed.

10. 8. The workpiece detection method according to claim 7, further comprising: Obtaining position information of the camera unit, The method is characterized by including a step of calculating the position of the workpiece from the position information and the deflection function fitted to the secondary image.

11. 11. The workpiece detection method according to claim 10, further comprising: The method is characterized by including a step of calculating, from the position of the workpiece, correction data for correcting a pre-taught position of a transport robot that transports the workpiece.

12. The workpiece detection method according to claim 7, The function is characterized by being a quadratic or higher even-order function.

Images