Soldering apparatus and soldering system, and processing apparatus

Abstract

The present invention relates to a soldering apparatus and soldering system that perform soldering by irradiating with processing light, and to a processing apparatus that processes an object by irradiating with processing light. [Solution] A soldering apparatus that irradiates processing light to melt solder placed on a circuit board comprises: a light irradiation device having a galvanometer mirror and irradiating processing light through the galvanometer mirror; a detection device that detects light from the circuit board and generates at least one of image data and shape data; a robot arm provided with the light irradiation device and the detection device and having a drive unit that moves the light irradiation device and the detection device; and a control device that controls the orientation of the galvanometer mirror so that the processing light from the light irradiation device, which is displaced together with the detection device, is irradiated to the same position based on at least one of the data that changes with the displacement of the detection device.

Description

[Technical Field]

[0001] The present invention relates to a soldering apparatus and soldering system that perform soldering by irradiating with processing light, and to a processing apparatus that processes an object by irradiating with processing light. [Background technology]

[0002] One example of this type of device is one that projects laser light from a laser head attached to a robot arm toward the area to be soldered (see Patent Document 1). Another related technology is described in Patent Document 2. A technical challenge of this type of device is, for example, properly soldering to a substrate with a three-dimensional shape (i.e., a 3D substrate). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] U.S. Public Gazette No. 2001 / 0054637 [Patent Document 2] U.S. Public Gazette No. 2015 / 0158176 [Overview of the project]

[0004] According to a first embodiment, a soldering apparatus is provided for irradiating a processing light to melt solder arranged on a circuit board, comprising: a light irradiation device having a galvanometer mirror and irradiating the processing light through the galvanometer mirror; a detection device that detects light from the circuit board and generates at least one of image data and shape data; a robot arm provided with the light irradiation device and the detection device and having a drive unit for moving the light irradiation device and the detection device; and a control device that controls the orientation of the galvanometer mirror so that the processing light from the light irradiation device, which is displaced together with the detection device, is irradiated to the same position based on the at least one of the data that changes with the displacement of the detection device.

[0005] According to a second embodiment, a processing apparatus is provided for irradiating an object with processing light, comprising: a light irradiation device having a scanning unit and irradiating the object with processing light via the scanning unit; a detection device for detecting light from the object; a moving device having a drive unit that moves the light irradiation device and the detection device, and the light irradiation device and the detection device; and a control device that controls the scanning unit based on the detection result of the detection device.

[0006] According to a third aspect, a soldering system for soldering an element to a circuit board is provided, comprising: a first moving device having a solder dispensing device for dispensing solder and a drive unit for moving the solder dispensing device; a second moving device having a holding device capable of holding the element and a drive unit for moving the holding device; a third moving device having a light irradiation device for irradiating processing light to melt solder and a detection device for detecting light from the circuit board, and a drive unit for moving the light irradiation device and the detection device; and a control device that (i) controls the solder dispensing device so that solder is placed on a predetermined portion of the circuit board; (ii) controls the holding device so that the element is placed on the circuit board via the placed solder; and (iii) controls the drive unit of the third moving device to bring the light irradiation device closer to the circuit board based on the detection result of the detection device, and controls the light irradiation device to melt the placed solder.

[0007] According to a fourth aspect, a soldering system for soldering an element to a circuit board is provided, comprising: a soldering dispenser for dispensing solder; a holding device capable of holding the element; a light irradiation device for irradiating processing light to melt the solder; a detection device for detecting light from the circuit board; a moving device having a drive unit for moving the soldering dispenser, the holding device, the light irradiation device and the detection device; and a control device that (i) controls the drive unit so that the soldering dispenser, the holding device, the light irradiation device and the detection device move closer to the circuit board; (ii) controls the soldering dispenser so that solder is placed on a predetermined portion of the circuit board; (iii) controls the holding device so that the element is placed on the circuit board via the placed solder; and (iv) controls the light irradiation device so that the placed solder is melted.

[0008] According to a fifth embodiment, a processing apparatus is provided for irradiating an object with processing light, comprising: a light irradiation device for irradiating the object with processing light; a detection device for detecting light from the object; a moving device having a drive unit for moving the light irradiation device and the detection device, and a control device for controlling the drive unit based on the detection result of the detection device. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a schematic diagram showing the overall structure of the soldering system according to the first embodiment. [Figure 2] Figure 2 is a system configuration diagram showing the configuration of a robot that constitutes part of the soldering system according to the first embodiment. [Figure 3] Figure 3 is a system configuration diagram showing the configuration of a robot that constitutes another part of the soldering system according to the first embodiment. [Figure 4] Figure 4 is a system configuration diagram showing the configuration of a robot that constitutes another part of the soldering system according to the first embodiment. [Figure 5]FIG. 5 is a diagram schematically showing the configuration of one detection device according to the first embodiment. [Figure 6] FIG. 6 is a diagram schematically showing the configuration of another detection device according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of structured light projected by a projector included in another detection device according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing a part of the optical path of the light irradiation device according to the first embodiment. [Figure 9] FIG. 12 is a diagram schematically showing the configuration of the matching processing unit included in the control device according to the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining the concept of the matching process according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of the timing chart of the matching process according to the first embodiment. [Figure 12] FIG. 12 is a diagram schematically showing the configuration of the tracking unit included in the control device according to the first embodiment. [Figure 13] FIG. 13 is a diagram showing an example of the timing chart of the tracking process according to the first embodiment. [Figure 14] FIG. 14 is a flowchart showing the operation of the soldering system according to the first embodiment. [Figure 15] FIG. 15 is a diagram showing an example of the irradiation method of the processing light. [Figure 16] FIG. 16 is a diagram schematically showing the air blow and the smoke absorber. [Figure 17] FIG. 17 is a system configuration diagram showing the configuration of the soldering system according to the second embodiment. [Figure 18] FIG. 18 is a flowchart showing the operation of the soldering system according to the second embodiment. [Figure 19] FIG. 19 is a diagram showing a part of the optical path of the light irradiation device according to the modification. [Figure 20] FIG. 20 is a diagram schematically showing the configuration of the tracking unit according to the modification. [Figure 21] Figure 21 is a schematic diagram showing the overall configuration of the laser welding system according to the third embodiment. [Figure 22] Figure 22 is a system configuration diagram showing the configuration of a robot that constitutes part of the laser welding system according to the third embodiment. [Figure 23] Figure 23 is a system configuration diagram showing the configuration of a robot that constitutes another part of the laser welding system according to the third embodiment. [Figure 24] Figure 24 is a flowchart showing the operation of the laser welding system according to the third embodiment. [Modes for carrying out the invention]

[0010] The embodiments will be described below with reference to the drawings.

[0011] <First Embodiment> The first embodiment will be described with reference to Figures 1 to 14. This embodiment includes a soldering system that includes a robot for performing soldering.

[0012] (overview) An overview of the soldering system according to the first embodiment will be described with reference to Figures 1 to 4. In Figure 1, the soldering system is a soldering system for soldering elements to a circuit board T. The soldering system includes robots 1, 2, and 3.

[0013] The robot 1, which may also be called a processing device or a solder coating device, has a robot arm 110, which may also be called a first moving means, that is equipped with a dispenser 40 (see Figures 2(a) and 2(b)) which dispenses solder and may also be called a solder dispensing device, and has a drive unit 111 (see Figure 2(b)) for moving the dispenser 40.

[0014] The robot 2, which may also be called a processing device or an element installation device, has a robot arm 210, which may also be called a second moving means, which has a holding device 50 (see Figures 3(a) and 3(b)) capable of holding an element, which may also be called a gripping device, and a drive unit 211 (see Figure 3(b)) for moving the holding device 50.

[0015] The robot 3, which may also be called a processing device or soldering device, has a robot arm 310, which may also be called a third moving means, equipped with a light irradiation device 60 (see Figures 4(a) and 4(b)) for irradiating processing light to melt solder, and detection devices 320 and 330 (see Figures 4(a) and 4(b)) for detecting light from the circuit board T, and a drive unit 311 (see Figure 4(b)) for moving the light irradiation device 60 and the detection devices 320 and 330.

[0016] The "circuit board" may be a circuit board having a three-dimensional shape consisting of a circuit film on which a circuit is formed and a substrate (i.e., a 3D circuit board). In other words, the circuit board may be a circuit board manufactured by the IMPC (registered trademark) (In-Mold Printed Circuit) method. Furthermore, the above circuit board is not limited to a circuit board manufactured by the IMPC method, but may be a circuit board having a three-dimensional shape consisting of a circuit film and a substrate, for example, manufactured by another method. Furthermore, the above circuit board is not limited to a circuit board consisting of a circuit film and a substrate, but may be a circuit board having another three-dimensional shape. Furthermore, the above circuit board is not limited to a circuit board having a three-dimensional shape (3D circuit board), but may be a circuit board having a planar shape consisting of a circuit film on which a circuit is formed and a substrate. Furthermore, the above circuit board does not have to be a circuit board having a three-dimensional shape (3D circuit board), but may be a circuit board on which the circuit is formed on the substrate itself. Furthermore, the above circuit board may be a surface mount circuit board or a through-hole mount circuit board.

[0017] The circuit board T may include, for example, markers (e.g., 2D codes such as AR (Augmented Reality) markers or cross marks) and solder pads (lands) that can be used for attitude control and position control of at least one end effector of robots 1, 2, and 3 (i.e., dispenser 40, holding device 50, light irradiation device 60) or the detection device described later. Here, the above markers and solder pads are detectable by the detection device described later (e.g., recognizable by image) for attitude control and position control of at least one end effector of robots 1, 2, and 3 or the detection device described later.

[0018] A "component" is a component that is soldered to a circuit board T by the soldering system, and examples include electronic components and electrical components. Furthermore, the above-mentioned components may be surface-mount components or through-hole components (i.e., leaded components). Such "components" may also be referred to as "parts". Examples of components include LEDs (Light Emitting Diodes) (e.g., well-known components such as chip LEDs), resistors (e.g., well-known components such as chip resistors), capacitors (e.g., well-known components such as chip capacitors), transistors (e.g., well-known components such as chip transistors), and connectors.

[0019] Although the term "robot arm" is used, it is not limited to robot arms (i.e., vertical articulated robots), but can also be applied to various existing forms such as SCARA robots (i.e., horizontal articulated robots), parallel link robots, and Cartesian robots. Furthermore, existing moving mechanisms may be used instead of the robot arm 312, etc., as long as they can move the light irradiation device 60, etc. The robot arms 110, 210 and 310 may be industrial robots or collaborative robots.

[0020] The soldering system includes a control device 1000 (see Figures 2(b), 3(b), and 4(b)) which controls the dispenser 40 as the end effector of robot 1 so that solder is placed on a predetermined portion of the circuit board T, (ii) the holding device 50 as the end effector of robot 2 so that elements are placed on the circuit board T via the placed solder, and (iii) the drive unit 311 of robot arm 310 so that the light irradiation device 60 as the end effector of robot 3 is brought closer to the circuit board T based on the detection result of at least one of the detection devices 320 and 330, and controls the light irradiation device 60 so that the placed solder is melted.

[0021] In Figure 1, the control device 1000 first controls the dispenser 40 of the robot 1 so that solder is placed on a predetermined portion of the circuit board T being transported by the belt conveyor (in other words, so that solder is applied) (solder placement process). Next, the control device 1000 controls the holding device 50 of the robot 2 so that elements are placed on the circuit board T' where the solder has been placed, via the placed solder (element placement process). Next, based on the detection result of at least one of the detection devices 320 and 330, the control device 1000 controls the drive unit 311 of the robot arm 310 to bring the light irradiation device 60 closer to the circuit board T'' where the elements are placed, and controls the light irradiation device 60 to melt the placed solder (soldering process). After that, the control device 1000 may, for example, inspect the soldered solder and elements based on the detection result of the detection device 330 (inspection process).

[0022] By having the three robots 1, 2, and 3 work together and divide the tasks in this way, the soldering of components onto the circuit board T can be made more efficient, thereby improving the throughput of component soldering.

[0023] In Figure 1, the circuit board with solder applied is shown as "Circuit Board T'", and the circuit board with components installed is shown as "Circuit Board T''", but below, to avoid complexity in description, we will refer to them all as "Circuit Board T".

[0024] The solder may be, for example, solder paste, solder wire, solder bars, etc. In other words, the dispenser 40 may place, for example, solder paste, solder wire, solder bars, etc., on the circuit board T.

[0025] With respect to the light irradiation device 60, "to melt the placed solder" includes irradiating a predetermined portion of the circuit board T with processing light to melt the solder. For example, this predetermined portion includes solder placed on the solder pads of the circuit board T. In this case, the processing light from the light irradiation device 60 is directly irradiated onto the solder to melt it. The predetermined portion also includes a part of the solder pads provided on the circuit board T (for example, a part of the solder pad where no solder is placed) or a part of an element placed on the circuit board T (for example, an electrode of an element). In this case, by irradiating a part of the circuit board T other than the solder, the solder is indirectly melted by heat conduction from that part to the solder. The circuit board T may be a planar board or a 3D circuit board having a three-dimensional shape as described above. If the circuit board T is a 3D circuit board, the predetermined portion may be set on an inclined surface on the circuit board T. In this case, the dispenser 40 of the robot 1 may place solder on at least a part of the predetermined portion of the inclined surface (for example, a solder pad). The light irradiation device 60 of the robot 3 may irradiate a predetermined part (for example, a part of a solder pad where no solder is placed) with processing light in order to melt the solder placed on the predetermined part of the inclined surface.

[0026] For each of robots 1, 2, and 3, explanations will be provided with reference to Figures 1 through 4, as well as Figures 5 through 8. Here, we will mainly explain robot 3, and explanations common to robots 1 and 2 will be omitted as appropriate.

[0027] (Robot 3) As described above, robot 3 is a robot that irradiates processing light to melt solder placed on a circuit board T. Robot 3 comprises (i) a light irradiation device 60 having a galvanometer mirror 61 (see Figure 8) and irradiating processing light through the galvanometer mirror 61; (ii) detection devices 320 and 330 that detect light from the circuit board T and generate at least one of image data and shape data; and (iii) a robot arm 310 that is equipped with the light irradiation device 60 and the detection devices 320 and 330 and has a drive unit 311 that moves the light irradiation device 60 and the detection devices 320 and 330.

[0028] As shown in Figure 4(a), the robot arm 310 has arm sections 310a and 310b and a wrist section 310c. The drive unit 311 may have, for example, a motor for rotating the entire robot arm 310, a motor for moving the entire robot arm 310 back and forth, a motor for moving each of the arm sections 310a and 310b up and down, a motor for rotating the arm section 310b and the wrist section 310c, a motor for rotating the wrist section 310c, and a motor for bending the wrist section 310c (none of which are shown). In addition to rotary joints, the robot arm 310 may also have linear joints.

[0029] The drive unit 311 rotates the entire robot arm 310, moves it back and forth, and moves at least one of the arm sections 310a and 310b up and down, thereby moving the wrist section 310c to a position near the circuit board T, for example. The drive unit 311 further rotates the arm section 310b and the wrist section 310c, and rotates and bends the wrist section 310c, thereby moving the light irradiation device 60 or changing the orientation of the light irradiation device 60 so that the processing light that melts the solder placed on the circuit board T can be irradiated to at least a portion of a predetermined area (for example, solder placed on the circuit board T, solder pads provided on the circuit board T, elements placed on the circuit board T, etc.).

[0030] In other words, the drive unit 311 moves the robot arm 310, for example, as described above, causing the detection devices 320 and 330 and the light irradiation device 60 to move toward the circuit board T, for example.

[0031] It is assumed that the robot arm 310 and the detection device 320 (more precisely, the cameras 21 and 22 of the detection device 320) have been calibrated using existing methods. For example, an object with a known precise shape (e.g., a checkerboard) is placed at the precise location in the robot arm 310's three-dimensional coordinate system (the so-called world coordinate system), the object is imaged by the detection device 320, and the correspondence between the coordinate system of the detection device 320 (the so-called camera coordinate system) and the coordinate system of the robot arm is determined by performing well-known calculations (i.e., calibration is performed). Similarly, it is assumed that the robot arm 310 and the detection device 330 (more precisely, the cameras 31 and 32 of the detection device 330) have also been calibrated using existing methods.

[0032] In Figure 4(a), the detection device 320 is located on the arm portion 310b of the robot arm 310, and the detection device 330 is located on the wrist portion 310c of the robot arm 310. However, the arrangement of the detection devices 320 and 330 is not limited to this. Furthermore, the robot 3 may be equipped with only one of the detection devices 320 and 330, or it may be equipped with other detection devices in addition to the detection devices 320 and 330 (i.e., the robot 3 may be equipped with three or more detection devices). Also, the robot 3 may be equipped with at least one detection device other than the detection devices 320 and 330. In other words, as long as the light irradiation device 60 can be brought close to the circuit board T or a predetermined part of the circuit board T (for example, solder pads provided on the circuit board T, or elements and solder placed on the circuit board T) by the drive unit 311 of the robot arm 310 so that the solder placed on the circuit board T can be melted by the processing light, the configuration (for example, the number and specifications of cameras in the detection device, the presence or absence of a projector, etc.), placement, and number of detection devices 320 and 330 do not matter.

[0033] Detection device 320 The detection device 320 will be described with reference to Figure 5. In Figure 5, the detection device 320 is configured to include cameras 21 and 22, which may also be called imaging devices. Cameras 21 and 22 each have an optical component such as a lens and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) or a CCD (Charge Coupled Device). As an example, in the detection device 320, cameras 21 and 22 may be configured as a stereo camera arranged at a predetermined baseline length apart from each other.

[0034] Cameras 21 and 22 are configured to detect light incident on them using their image sensors. Examples of such incident light include light reflected by an object (e.g., at least a part of the circuit board T), light scattered by an object, and light transmitted through an object. In other words, cameras 21 and 22 detect light from objects present within their respective fields of view and capture an image of that object. That is, each of cameras 21 and 22 is configured to detect light incident on them and generate image data (i.e., data representing a two-dimensional image) as a result of the detection. In other words, cameras 21 and 22 are configured to output image data representing the captured image. In other words, each of cameras 21 and 22 is configured to detect objects. Here, "image data" refers to data where each pixel of the image sensor of each of cameras 21 and 22 is associated (in other words, linked) with the pixel value of each pixel, such as the brightness value.

[0035] The detection device 320 is configured to simultaneously image an object (for example, at least a part of a circuit board T) with cameras 21 and 22, and to generate and output shape data (i.e., shape data showing the three-dimensional shape of the object) as a detection result based on the two image data outputs from cameras 21 and 22, respectively. The detection device 320 is configured to output the generated shape data, for example, as shape data used in matching processing and tracking processing described later. Here, as an example, the shape data is three-dimensional point cloud data (hereinafter also simply referred to as point cloud data). In this case, the detection device 320 generates point cloud data by calculating the distance from cameras 21 and 22 to the object using a well-known method, based on the difference (i.e., parallax) between the position of the object on the image captured by camera 21 and the position of the object on the image captured by camera 22, the focal lengths of cameras 21 and 22, and the distance between cameras 21 and 22 (i.e., baseline length). For example, point cloud data is data in which points corresponding to each pixel of cameras 21 and 22 are associated with 3D information (X coordinate, Y coordinate, Z coordinate). Shape data is not limited to point cloud data; it may also be data representing existing 3D information, such as depth image data in which the distance to the object is associated with the brightness value of each pixel. Furthermore, the detection device 320 is configured to output image data generated by at least one of cameras 21 and 22, for use in matching processing and tracking processing, for example, as described later.

[0036] The detection device 320 is configured to detect at least a part of the circuit board T from a wide area in order to approach the circuit board T when the circuit board T, which is the target object, and, for example, the light irradiation device 60 are relatively far apart. In other words, it is configured to image at least a part of the circuit board T and its surroundings, and to generate at least one of wide-area image data and shape data. For this reason, cameras 21 and 22 are used that have a wider field of view than cameras 31 and 32 of the detection device 330, which will be described later. For example, cameras 21 and 22 are used that have a larger angle of view (in other words, a shorter focal length) than cameras 31 and 32, which will be described later. That is, for example, cameras 21 and 22 are used that have a larger angle of view (in other words, a shorter focal length) than the lenses of cameras 31 and 32, which will be described later.

[0037] Furthermore, the detection device 320 can detect at least a portion of the circuit board T from a wide area, not only when the circuit board T as the target object and, for example, the light irradiation device 60 are relatively far apart, but also when the circuit board T as the target object and, for example, the light irradiation device 60 are relatively close together.

[0038] The detection device 320 generates, for example, image data and shape data of at least a portion of the circuit board T as the object. Here, "at least a portion of the circuit board T" (i.e., the object) refers to, for example, at least a portion of the circuit board T itself, a marker formed on the circuit board T (e.g., a two-dimensional code such as an AR marker or a cross mark), or a solder pad (land) formed on the circuit board T. The detection device 120 may also generate, for example, image data and shape data of a marker placed on the circuit board T (e.g., a two-dimensional code such as an AR marker or a cross mark) or an element provided on the circuit board T. The detection device 120 may also generate, for example, image data and shape data of a marker placed in the vicinity of the circuit board T (e.g., a two-dimensional code such as an AR marker or a cross mark).

[0039] The detection device 320 may consist of a single camera instead of cameras 21 and 22. In this case, the single camera generates image data of the object. The detection device 320 may also include another camera in addition to cameras 21 and 22. In this case, for example, shape data may be generated by cameras 21 and 22 (stereo camera), and image data may be generated by the other camera. The detection device 320 may also include a projector in addition to cameras 21 and 22, similar to the detection device 330 described later. In this case, the detection device 320 may be configured to generate and output at least one of the image data and shape data of the object. The detection device 320 may also consist of a single camera and a projector instead of cameras 21 and 22. In this case, the detection device 320 may be configured to generate and output at least one of the image data and shape data of the object. For specific methods of generating shape data in the embodiment in which the detection device 320 includes a projector, please refer to the description of the detection device 330 described later.

[0040] Detection device 330 The detection device 330 will be described with reference to Figure 6. In Figure 6, the detection device 130 is configured to include cameras 31 and 32, which may be called imaging devices, and a projector 33, which may be called a projection device. For example, cameras 31 and 32 each include optical elements such as lenses and image sensors such as CMOCs and CCDs. As an example, cameras 31 and 32 may be configured as stereo cameras arranged at a predetermined baseline length apart from each other. Cameras 31 and 32 are configured to detect light incident on themselves. Examples of such incident light include light reflected by an object (for example, at least a part of the circuit board T), light scattered by an object, and light transmitted through an object. In other words, cameras 31 and 32 detect light from objects present within their respective fields of view and capture images of those objects. In other words, cameras 31 and 32 detect light from objects present within their respective fields of view and capture images of those objects. In other words, cameras 31 and 32 are configured to detect light incident on them and generate image data (i.e., data representing a two-dimensional image) as a result of that detection. To put it another way, cameras 31 and 32 are configured to generate image data representing the captured image. The detection device 330 is configured to output image data generated by at least one of cameras 31 and 32, for use in matching processing, tracking processing, etc., as described later.

[0041] The projector 33 is configured to project structured light having a predetermined intensity distribution (in other words, a predetermined pattern) as shown in Figures 7(a) to 7(c) when the detection device 330 is in operation. Various existing configurations, such as DLP (Digital Light Processing) projectors, can be applied to the projector 33.

[0042] Here, the detection device 330 is configured to project structural light onto the object from the projector 33 and generate image data of the object onto which the structural light has been projected using cameras 31 and 32. By projecting a predetermined pattern of structural light using the projector 33, the detection device 330 can generate highly accurate shape data with minimal influence from disturbances, even when the surface of the object is dark or when there are few feature points on the surface of the object.

[0043] The detection device 330 is configured to simultaneously image an object onto which structural light is projected from the projector 33 using cameras 31 and 32, and to generate and output shape data (i.e., shape data showing the three-dimensional shape of the object) based on the two image data outputs from each camera. Here, as an example, the detection device 320 calculates the distance from cameras 31 and 32 to the object using a well-known method based on the difference (i.e., parallax) between the position of the pattern caused by structural light on the image captured by camera 31 and the position of the pattern caused by structural light on the image captured by camera 32, the focal lengths of cameras 31 and 32, and the distance between cameras 31 and 32 (i.e., baseline length), and generates three-dimensional point cloud data (hereinafter also simply referred to as point cloud data). The detection device 330 is configured to output the generated shape data, for example, as shape data used in matching processing and tracking processing described later.

[0044] Furthermore, the shape data is not limited to point cloud data; it may also be existing 3D information data such as depth image data, where the distance to the object is associated with the brightness value of each pixel. Various existing methods, such as the phase shift method, random dot method, and TOF (Time-of-Flight) method, can be applied to generate the shape data.

[0045] Furthermore, the detection device 330 is configured to generate image data using at least one of the cameras 31 and 32 when no structural light is projected from the projector 33.

[0046] The detection device 330 is configured to detect at least a portion of the circuit board T with high accuracy when the circuit board T, as the object, and, for example, the light irradiation device 60 are relatively close, by getting closer to a part of the circuit board T (for example, solder placed on the circuit board T, solder pads provided on the circuit board T, elements placed on the circuit board T, etc.). In other words, it is configured to image at least a portion of the circuit board T and its surroundings, and to generate high-precision image data and at least one of shape data. For this reason, cameras 31 and 32 have a higher resolution than cameras 21 and 22 in the detection device 320. In this case, for example, cameras 31 and 32 have a narrower field of view (in other words, a longer focal length) than cameras 21 and 22. For example, the lenses of cameras 31 and 32 each have a narrower field of view (in other words, a longer focal length) than the lenses of cameras 21 and 22, respectively. Also, the lenses of cameras 31 and 32 may have a higher magnification ratio than the lenses of cameras 21 and 22, respectively.

[0047] Because the resolution of the objects captured by cameras 31 and 32 is high, the resolution of the image data and the accuracy of the shape data generated by the detection device 230 equipped with cameras 31 and 32 are higher than those of the image data and shape data generated by the detection device 320 equipped with cameras 21 and 22. For this reason, the detection device 330 can detect at least a portion of the circuit board T with higher accuracy than the detection device 320. Therefore, by using the image data and shape data generated by the detection device 330 in the control device 1000, the accuracy of position and orientation estimation by the matching process in the matching processing unit 200 and the tracking process in the tracking unit 300, which will be described later, is improved.

[0048] Furthermore, the detection device 330 can detect at least a portion of the circuit board T with high accuracy not only when the circuit board T and, for example, the light irradiation device 60 are relatively close together, but also when the circuit board T and, for example, the light irradiation device 60 are relatively far apart.

[0049] As a result, the field of view of cameras 31 and 32 of detection device 330 becomes narrower than the field of view of cameras 21 and 22 of detection device 320. Detection device 320 may also be referred to as the first imaging unit, and detection device 330 may be referred to as the second imaging unit.

[0050] The detection device 330 generates, for example, image data and shape data of at least a portion of the circuit board T as the object. Here, "at least a portion of the circuit board T" (i.e., the object) refers to, for example, at least a portion of the circuit board T itself, or markers (e.g., 2D codes such as AR markers or cross marks) or solder pads (lands) formed on the circuit board T. The detection device 330 may also generate, for example, image data and shape data of markers (e.g., 2D codes such as AR markers or cross marks) placed on the circuit board T, or elements or solder provided on the circuit board T. The detection device 330 may also generate, for example, image data and shape data of markers (e.g., 2D codes such as AR markers or cross marks) placed in the vicinity of the circuit board T.

[0051] The detection device 330 may consist of a single camera instead of cameras 31 and 32. In this case, the single camera generates image data of the object. The detection device 330 may also include another camera in addition to cameras 21 and 22. In this case, for example, shape data may be generated by cameras 31 and 32 (stereo cameras) and projector 33, and image data may be generated by the other camera. Alternatively, the detection device 330 may consist of a single camera and projector 33 instead of cameras 31 and 32. In this case, the detection device 330 may be configured to generate and output at least one of the image data and shape data of the object.

[0052] Light irradiation device 60 The light irradiation device 60 will be explained with reference to Figure 8. In Figure 8, the light irradiation device 60 is configured to include a galvanometer mirror 61, which can also be called a scanning unit, and an fθ lens 62. Therefore, the light irradiation device 60 can move the irradiation position of the processing light L onto the target object (for example, at least a part of the circuit board T) along a desired direction (in other words, it can scan the irradiation position of the processing light L onto the target object). The galvanometer mirror 61 is configured to change the orientation of its mirror, and by changing the orientation of its mirror, it changes the emission direction of the processing light L incident from a light source (not shown). The processing light L emitted from the galvanometer mirror 61 is incident on the fθ lens 62. The fθ lens 62 focuses the processing light L incident from the galvanometer mirror 61. In other words, the light irradiation device 60 can change the irradiation position of the processing light L irradiated onto the circuit board T, which is the target object, via the fθ lens 62, according to the orientation of its own mirror in the galvanometer mirror 61 (in other words, the change in the direction of emission of processing light L from the galvanometer mirror 61).

[0053] For example, the galvanometer mirror 61 includes a first scanning mirror 61Y and a second scanning mirror 61X, both equipped with mirrors that can swing or rotate around predetermined axes, and is arranged such that the axes of swinging or rotating of the first scanning mirror 61Y and the second scanning mirror 61X intersect (for example, are orthogonal). In this case, the processing light L incident on the first scanning mirror 61Y is reflected by the first scanning mirror 61Y and incident on the second scanning mirror 61X, and is reflected by the second scanning mirror 61X and incident on the fθ lens 62. The fθ lens 62 focuses the processing light L incident on the second scanning mirror 61X. Because the direction of emission of the processing light L from the second scanning mirror 61X differs depending on the orientation of the first scanning mirror 61Y around its axis and the orientation of the second scanning mirror 61X around its axis (in other words, the incident position of the processing light L at the fθ lens 62 differs), the irradiation position of the processing light on the circuit board T changes depending on the orientation of the first scanning mirror 61Y and the second scanning mirror 61X.

[0054] Furthermore, the light irradiation device 60 can melt solder placed on the circuit board T by irradiating it with processing light L. In addition, it is not limited to directly irradiating the solder with processing light L; for example, the solder can be indirectly melted by irradiating solder pads provided on the circuit board T (for example, parts of solder pads where no solder is placed) with processing light, or by irradiating a part of an element (component) placed on the circuit board T (for example, an electrode) with processing light.

[0055] Furthermore, the galvanometer mirror 61 is not limited to two scanning mirrors (first scanning mirror 61Y and second scanning mirror 61X), but may consist of a single scanning mirror or three or more scanning mirrors. Furthermore, the light irradiation device 60 is not limited to the galvanometer mirror 61, but may also use other existing devices that change the direction of light emission, such as a polygon mirror, DMD (Digital Micromirror Device), or spatial light modulator. Furthermore, the light irradiation device 60 is not limited to a configuration that includes an fθ lens 62, but may be without an fθ lens 62, or may be equipped with one or more other lenses instead of the fθ lens 62.

[0056] The light source (not shown) for the processing light L incident on the galvanometer mirror 61 of the light irradiation device 60 may be located outside the soldering system, included in the soldering system, included in the robot 3, or included in the light irradiation device 60. The light source (not shown) can change the intensity of the processing light L irradiated onto the object. The method for changing the intensity of the processing light L irradiated onto the object is not limited to changing the intensity of the light emitted from the light source; methods using existing light intensity changing members such as ND filters can also be applied. The light from the light source (not shown) is incident on the galvanometer mirror 61 of the light irradiation device 60 using existing methods. The light irradiation device 60 may also be equipped with a focus lens. The focus lens consists of one or more lenses, and by changing the position of at least some of the lenses along the optical axis, the focusing position of the processing light L in the optical axis direction of the light irradiation device 60 (i.e., the focal position of the light irradiation device 60) can be changed. In other words, the spot size of the processing light L irradiated onto the object can be changed. For example, the focusing lens may be positioned in the optical path of the processing light L before it enters the galvanometer mirror 61. Furthermore, the configuration for changing the spot size of the processing light L on the object in the light irradiation device 60 is not limited to a focusing lens; existing configurations can also be applied.

[0057] The control device 1000 may control the orientation of the galvanometer mirror 61 for the robot 3 configured as described above, based on at least one of the image data and shape data, which changes with the displacement of at least one of the detection devices 320 and 330, so that the processing light L from the light irradiation device 60, which is displaced along with the displacement of at least one of the detection devices 320 and 330, is irradiated to the same position.

[0058] The control device 1000 may control the drive unit 311 to stop its operation. The control device 1000 may also control the orientation of the galvanometer mirror 61 so that the processing light L from the light irradiation device 60, which is displaced together with at least one of the detection devices 320 and 330, is irradiated to the same position, based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330 after the drive of the drive unit 311 has stopped.

[0059] The control device 1000 may control the drive unit 311 so that the light irradiation device 60 and the detection devices 320 and 330 are moved, and at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 320 and 330 moved by the robot arm 310, so that the processing light L from the light irradiation device 60 moved by the robot arm 310 is irradiated to the same position.

[0060] The control device 1000 may control the orientation of the galvanometer mirror 61 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330, so that the irradiation position of the processing light L from the light irradiation device 60, which is displaced together with the displacement of at least one of the detection devices 320 and 330, is maintained at a first position, and then maintained at a second position different from the first position.

[0061] The control device 1000 may control the drive unit 311 to stop driving the drive unit 311. After the drive unit 311 stops, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L from the light irradiation device 60, which is displaced along with the displacement of at least one of the detection devices 320 and 330, is maintained at a first position, and then at a second position different from the first position, based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330.

[0062] The control device 1000 may control the drive unit 311 to move the light irradiation device 60 and the detection devices 320 and 330, and based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 320 and 330 moved by the robot arm 310, it may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L from the light irradiation device 60 moved by the robot arm 310 is maintained at a first position, and then maintained at a second position different from the first position.

[0063] The control device 1000 may control the drive unit 311 of the robot arm 310 so that the light irradiation device 60 and the detection devices 320 and 330 approach the circuit board T based on at least one of the image data and shape data. When the light irradiation device 60 and the detection devices 320 and 330 approach the circuit board T to a predetermined distance, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the processing light L from the light irradiation device 60, which is displaced together with at least one of the detection devices 320 and 330, is irradiated to the same position, based on the at least one of the data that changes with the displacement of at least one of the detection devices 320 and 330.

[0064] Furthermore, the light irradiation device 60 does not necessarily need to be equipped with a scanning unit such as a galvanometer mirror 61.

[0065] Here, the control device 1000 may be a different device from the robot 3, which can also be called a soldering device, or it may constitute a part of the robot 3 (in other words, the robot 3 may be equipped with the control device 1000). In the latter case, the control device 1000 may be provided solely by the robot 3, or it may be shared by the robot 3 and at least one of the robots 1 and 2 (i.e., the control device 1000, which constitutes a part of the robot 3, may control the robot 3 as well as at least one of the robots 2 and 3). If the robot 3 is equipped solely with the control device 1000, the robots 1 and 2 may each be equipped solely with a control device 1000 that is different from the control device 1000 provided by the robot 3.

[0066] (Robot 1) As described above, robot 1 is a robot that places solder on a predetermined part of a circuit board T (for example, a solder pad or part of a circuit). In Figures 2(a) and 2(b), robot 1 comprises (i) a dispenser 40 that dispenses solder, (ii) detection devices 120 and 220 that detect light from the circuit board T and generate at least one of image data and shape data, and (iii) a robot arm 110 that is equipped with the dispenser 40 and the detection devices 120 and 130 and has a drive unit 111 that moves the dispenser 40 and the detection devices 120 and 130.

[0067] The robot arm 110, like the robot arm 310, has arm sections 110a and 110b and a wrist section 110c. The detection devices 120 and 130 may be configured in the same way as the detection devices 320 and 330, respectively. The dispenser 40 can change the amount of solder paste dispensed, and the control device 1000 can control the amount of solder dispensed from the dispenser 40.

[0068] In Figure 2(a), the detection device 120 is located on the arm portion 110b of the robot arm 110, and the detection device 130 is located on the wrist portion 110c of the robot arm 110. However, the arrangement of the detection devices 120 and 130 is not limited to this. Furthermore, the robot 1 may be equipped with only one of the detection devices 120 and 130, or it may be equipped with other detection devices in addition to the detection devices 120 and 130 (i.e., the robot 1 may be equipped with three or more detection devices). In other words, as long as the dispenser 40 can be brought close to the circuit board T or a predetermined portion of the circuit board T (e.g., a solder pad or a part of the circuit) by the drive unit 111 of the robot arm 110 so that solder can be placed on that predetermined portion of the circuit board T (e.g., a solder pad or a part of the circuit), the configuration (e.g., the number and specifications of cameras in the detection device, the presence or absence of a projector, etc.), the placement position, and the number of detection devices 120 and 130 are all acceptable.

[0069] Furthermore, detection devices 120 and 130 may have the same configuration as detection devices 320 and 330 described above. However, detection devices 120 and 130 do not have to have the same configuration as detection devices 320 and 330 described above. The configuration and specifications of detection device 120 can be appropriately changed as long as it does not contradict the gist or concept that can be gleaned from the description of detection device 320 described above. Similarly, the configuration and specifications of detection device 130 can be appropriately changed as long as it does not contradict the gist or concept that can be gleaned from the description of detection device 330 described above.

[0070] The control device 1000 may control the drive unit 111 for the robot 1 configured as described above, based on at least one of the image data and shape data which changes with the displacement of at least one of the detection devices 120 and 130, so that the solder dispensed from the dispenser 40, which is displaced together with the displacement of at least one of the detection devices 120 and 130, is placed on a predetermined portion of the circuit board T.

[0071] The control device 1000 may control the drive unit 111 to stop the drive of the drive unit 111. The control device 1000 may also control the drive unit 111 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 120 and 130 after the drive of the drive unit 111 has stopped, so that the solder dispensed from the dispenser 40 which displaces together with at least one of the detection devices 120 and 130 is placed on a predetermined portion of the circuit board T.

[0072] The control device 1000 may control the drive unit 111 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 120 and 130, so that the solder dispensed from the dispenser 40, which is displaced along with the displacement of at least one of the detection devices 120 and 130, is initially placed at a first position on the circuit board T, and then at a second position different from the first position.

[0073] The control device 1000 may control the drive unit 111 to stop the drive of the drive unit 111. The control device 1000 may control the drive unit 111 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 120 and 130 after the drive of the drive unit 111 has stopped, so that the solder dispensed from the dispenser 40, which is displaced together with the displacement of at least one of the detection devices 120 and 130, is initially placed at a first position on the circuit board T, and then at a second position different from the first position.

[0074] The control device 1000 may control the drive unit 111 of the robot arm 110 so that the dispenser 40 and the detection devices 120 and 130 approach the circuit board T based on at least one of the image data and shape data, and when the dispenser 40 and the detection devices 120 and 130 approach the circuit board T to a predetermined distance, the control device 1000 may control the drive unit 111 so that the solder dispensed from the dispenser 40, which is displaced together with at least one of the detection devices 120 and 130, is placed on a predetermined portion of the circuit board T, based on the at least one of the data that changes with the displacement of at least one of the detection devices 120 and 130.

[0075] Here, the control device 1000 may be a different device from the robot 1, which can also be called a solder coating device, or it may constitute a part of the robot 1 (in other words, the robot 1 may be equipped with the control device 1000).

[0076] (Robot 2) As described above, robot 2 is a robot that places elements via solder placed on a circuit board T. In Figures 3(a) and 3(b), robot 2 comprises (i) a holding device 50 for holding elements, (ii) detection devices 220 and 230 for detecting light from the circuit board T and generating at least one of image data and shape data, and (iii) a robot arm 210 to which the holding device 50 and the detection devices 220 and 230 are provided, and which has a drive unit 211 for moving the holding device 50 and the detection devices 220 and 230.

[0077] The robot arm 210, like the robot arm 310, has arm sections 210a and 210b and a wrist section 210c. The detection devices 220 and 230 may be configured similarly to the detection devices 220 and 230, respectively. Furthermore, the holding device 50 can be an existing device that can hold the element, such as a tweezers hand or a suction device.

[0078] The element holding force (gripping force) in the holding device 50 is adjustable, and the control device 1000 can control the element holding force in the holding device 50. For example, if the holding device 50 is a tweezers hand, the holding device 50 can control the force with which it grips the element at the tip of the tweezers. The robot 2 may also include a storage unit (not shown) for storing elements and a supply device (not shown) for supplying desired elements from the storage unit to the holding device 50. Examples of storage units include reels, trays, sticks, etc. Various existing configurations can be applied to the storage unit and supply device, so a detailed explanation is omitted. The control device 1000 may control the supply device to supply desired elements to be placed on a part (predetermined portion) of the circuit board T from the storage unit to the holding device 50, and control the holding device 50 to hold the elements. In this case, the robot 2 can move the holding device 50 closer to a separately provided element supply device (so-called parts feeder) (not shown) to hold the desired element in the circuit board T, thus eliminating the need to hold the element in the holding device 50. This makes the element placement work on the circuit board T more efficient.

[0079] In Figure 3(a), the detection device 220 is located on the arm portion 210b of the robot arm 210, and the detection device 230 is located on the wrist portion 210c of the robot arm 210. However, the arrangement of the detection devices 220 and 230 is not limited to this. Furthermore, the robot 2 may be equipped with only one of the detection devices 220 and 230, or it may be equipped with other detection devices in addition to the detection devices 220 and 230 (i.e., the robot 2 may be equipped with three or more detection devices). In other words, as long as the holding device 50 can be brought close to a predetermined portion of the circuit board T (solder pads, solder placed on the circuit, etc.) by the drive unit 211 of the robot arm 210 so that elements can be placed on that portion of the circuit board T, the configuration (for example, the number and specifications of cameras in the detection device, the presence or absence of a projector, etc.), the placement position, and the number of detection devices 120 and 130 are all acceptable.

[0080] Furthermore, detection devices 220 and 230 may have the same configuration as detection devices 320 and 330 described above. However, detection devices 220 and 230 do not have to have the same configuration as detection devices 320 and 330 described above. The configuration and specifications of detection device 220 can be appropriately changed as long as it does not contradict the gist or concept that can be gleaned from the description of detection device 320 described above. Similarly, the configuration and specifications of detection device 230 can be appropriately changed as long as it does not contradict the gist or concept that can be gleaned from the description of detection device 330 described above.

[0081] The control device 1000 may control the drive unit 211 of the robot 2 configured as described above, based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 220 and 230, so that the elements gripped (held) by the holding device 50, which displaces together with the displacement of at least one of the detection devices 220 and 230, are positioned on a predetermined portion of the circuit board T.

[0082] The control device 1000 may control the drive unit 211 to stop its operation. The control device 1000 may also control the drive unit 211 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230 after the drive unit 211 has stopped, so that the element gripped by the holding device 50, which displaces together with at least one of the detection devices 220 and 230, is positioned on a predetermined portion of the circuit board T.

[0083] The control device 1000 may control the drive unit 211 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230, so that after one element gripped by the holding device 50, which displaces together with the displacement of at least one of the detection devices 220 and 230, is positioned at a first position on the circuit board T, the other element gripped by the holding device 50 is positioned at a second position different from the first position.

[0084] The control device 1000 may control the drive unit 211 to stop driving the drive unit 211. After the drive unit 211 has stopped driving, the control device 1000 may control the drive unit 211 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230, so that after one element gripped by the holding device 50, which displaces with the displacement of at least one of the detection devices 220 and 230, is positioned at a first position on the circuit board T, another element gripped by the holding device 50 is positioned at a second position different from the first position.

[0085] The control device 1000 may control the drive unit 211 of the robot arm 210 so that the holding device 50 and the detection devices 220 and 230 approach the circuit board T based on at least one of the image data and shape data, and when the holding device 50 and the detection devices 220 and 230 approach the circuit board T to a predetermined distance, the control device 1000 may control the drive unit 211 so that the element gripped by the holding device 50, which displaces together with at least one of the detection devices 220 and 230, is positioned on a predetermined portion of the circuit board T, based on at least one of the data that changes with the displacement of at least one of the detection devices 220 and 230.

[0086] Here, the control device 1000 may be a different device from the robot 2, which can also be called an element installation device, or it may constitute a part of the robot 2 (in other words, the robot 2 may be equipped with the control device 1000).

[0087] In Figure 1, the control device 1000 controls the drive unit 311 so that the positional relationship between the circuit board T transported by the belt conveyor and, for example, the light irradiation device 60 of the robot 3 is a desired positional relationship, and / or so that the posture of the light irradiation device 60 is a desired posture (for example, by rotating the entire robot arm 310, moving it back and forth, moving the arm portions 310a and 310b up and down, rotating the arm portion 310b and the wrist portion 310c, rotating the wrist portion 310c, or bending it). In other words, by controlling the drive unit 311, the control device 1000 controls at least one of the position and posture of the light irradiation device 60 so that processing light L from the light irradiation device 60 is irradiated onto a predetermined portion of the circuit board T transported by the belt conveyor. At this time, the belt conveyor is temporarily stopped. That is, the belt conveyor is temporarily stopped after transporting the circuit board T to, for example, within the driveable range of the robot arm 310 of the robot 3. Then, after the processing light L from the light irradiation device 60 is irradiated onto a predetermined portion of the stopped circuit board T, the belt conveyor restarts to transport the circuit board T.

[0088] Similarly, the control device 1000 controls the drive unit 111 so that the positional relationship between the circuit board T transported by the belt conveyor and, for example, the dispenser 40 of the robot 1 is a desired positional relationship, and / or so that the posture of the dispenser 40 is a desired posture. In other words, by controlling the drive unit 111, the control device 1000 controls at least one of the position and posture of the dispenser 40 so that the solder dispensed from the dispenser 40 is placed on a predetermined portion of the circuit board T transported by the belt conveyor. At this time, the belt conveyor is temporarily stopped. That is, the belt conveyor is temporarily stopped after transporting the circuit board T to, for example, the drivable range of the robot arm 110 of the robot 1. Then, after the solder dispensed from the dispenser 40 is placed on the predetermined portion of the stopped circuit board T, the belt conveyor resumes driving to transport the circuit board T.

[0089] Similarly, the control device 1000 controls the drive unit 211 so that the positional relationship between the circuit board T transported by the belt conveyor and, for example, the holding device 50 of the robot 2 is a desired positional relationship, and / or so that the posture of the holding device 50 is a desired posture. In other words, by controlling the drive unit 211, the control device 1000 controls at least one of the position and posture of the holding device 50 so that the elements held by the holding device 50 are positioned on a predetermined portion of the circuit board T transported by the belt conveyor. At this time, the belt conveyor is temporarily stopped. That is, the belt conveyor is temporarily stopped after transporting the circuit board T to, for example, the drivable range of the robot arm 210 of the robot 2. Then, after the elements held by the holding device 50 are positioned on a predetermined portion of the stopped circuit board T, the belt conveyor resumes driving to transport the circuit board T.

[0090] The belt conveyor may be driven continuously without pausing in front of each robot arm. In other words, the control device 1000 may control the drive unit 311 so that processing light L from the light irradiation device 60 is irradiated onto a predetermined part of the circuit board T that is being transported (i.e., moving) by the belt conveyor, or control the drive unit 111 so that solder dispensed from the dispenser 40 is positioned, or control the drive unit 211 so that elements held by the holding device 50 are positioned.

[0091] The transport device for the circuit board T is not limited to a belt conveyor; various existing configurations can be applied as long as they can transport the circuit board T. Furthermore, the circuit board T does not have to be transported by a belt conveyor or the like. For example, the robots 1, 2, and 3 described above may be arranged to surround a stage on which the circuit board T is placed. In this case, the control device 1000 may control the drive unit 311 so that processing light L from the light irradiation device 60 is irradiated onto a predetermined part of the circuit board T placed on the stage, control the drive unit 111 so that solder dispensed from the dispenser 40 is placed, or control the drive unit 211 so that elements held by the holding device 50 are placed. At this time, the loading of the circuit board T onto the stage and the unloading of the circuit board T from the stage may be performed by robots other than robots 1, 2, and 3.

[0092] At least one of the robot arms 110, 210, and 310 may be mounted on an AGV (Automatic Guided Vehicle), for example. In this case, the control device 1000 may control at least one of the drive units of at least one of the robot arms 110, 210, and 310, the end effector of at least one of the robot arms 110, 210, and 310, and the drive unit of the AGV, based on the position and orientation information of the object acquired by the matching process and tracking process described later.

[0093] To enable the control of such drive units 311, etc. (in other words, robot arms 310, etc.), the processing performed by the control device 1000 will be explained with reference to Figures 9 to 13. Here, as an example, processing using the outputs of detection devices 320 and 330 provided by robot 3 will be explained. The control device 1000 may perform similar processing using the outputs of detection devices 120 and 130 provided by robot 1, and the outputs of detection devices 220 and 230 provided by robot 2.

[0094] (Calculation of the object's position and orientation through matching processing) The matching process for the circuit board T, which is the target object, performed in the control device 1000 will be explained with reference to Figures 9 to 11. In order for the position and orientation of the light irradiation device 60 of the robot 3 to be in the desired position and orientation relative to a part of the target object (for example, the part of the circuit board T to be irradiated with processing light L) that has been transported by the belt conveyor, it is necessary to bring the light irradiation device 60 closer to the target object using the robot arm 310. In order to bring the light irradiation device 60 closer to the target object using the robot arm 310, the control device 1000 can perform a matching process to calculate (estimate) the position and orientation of the target object.

[0095] In Figure 9, the control device 1000 includes a robot control unit 100 and a matching processing unit 200, which are either logically implemented processing blocks or physically implemented processing circuits.

[0096] Furthermore, the matching processing unit 200 determines whether to use the output of the detection device 320 (for example, at least one of image data and shape data) or the output of the detection device 330 (for example, at least one of image data and shape data) for controlling the drive unit 311 as the light irradiation device 60, which is located relatively far from the circuit board T, approaches the circuit board T.

[0097] The matching processing unit 200 is configured to include a first matching unit 201, a second matching unit 202, and a comparison unit 203.

[0098] The first matching unit 201 performs matching between the output of the detection device 320 (for example, at least one of image data and shape data) and CAD (Computer-Aided Design) data. As a result of the matching, the first matching unit 201 outputs the position and orientation estimation result of the object, which is the result of (calculating) estimating the position and orientation of the object, and the matching rate. The position and orientation estimation result of the object may be expressed in terms of so-called 6DoF (six degrees of freedom). In this case, the position and orientation estimation result is data representing the X coordinate, Y coordinate, Z coordinate, components around the X axis (θX component), components around the Y axis (θY component), and components around the Z axis (θZ component) in the coordinate system of the robot arm 310 (so-called world coordinate system) for the X, Y, and Z axes. These data may be represented as matrices, their values ​​may be represented as tables, or they may be data in other well-known formats. The X, Y, and Z coordinates are estimated results indicating the position of the object, while the components around the X, Y, and Z axes are estimated results indicating the orientation of the object. The components around the X, Y, and Z axes can also be rephrased as yaw, roll, and pitch. In the first matching unit 201, calculating (estimating) the position and orientation of the feature region (described later) of the object is included in the concept of calculating (estimating) the position and orientation of the object.

[0099] The first matching unit 201 calculates (estimates) the position and orientation of an object by, for example, comparing the feature region of the object in the image data output from the detection device 320 (for example, a part of the object's outline in the image data) with the feature region of the object in the object's CAD data (for example, CAD data corresponding to a part of the object's outline in the image data). At this time, the first matching unit 201 first extracts the feature region of the object in the image data output from the detection device 320 and the feature region of the object in the object's CAD data. Next, the first matching unit 201 calculates the position and orientation of the object in the coordinate system (world coordinate system) of the robot arm 310 by, for example, changing the dimensions or rotating the feature region of the object in the CAD data to establish a correspondence between the feature region of the object in the image data and the feature region of the object in the CAD data. More specifically, the first matching unit 201 first determines the correspondence between the CAD coordinate system (so-called local coordinate system) and the coordinate system of the detection device 320 (camera coordinate system) so that the feature region of the object in the CAD data matches the feature region of the object in the image data. Since the position and orientation of the feature region of the object in the CAD coordinate system (i.e., the feature region of the object in the CAD data) are known, the position and orientation of the feature region of the object in the coordinate system of the detection device 320 can be determined by aligning the CAD coordinate system with the coordinate system of the detection device 320. Next, the first matching unit 201 calculates the position and orientation of the object in the coordinate system of the robot arm 310 based on the correspondence between the CAD coordinate system and the coordinate system of the detection device 320, and the correspondence between the coordinate system of the detection device 320 and the coordinate system of the robot arm 310, which was determined in advance by calibration.

[0100] Furthermore, the image data matching process performed by the first matching unit 201 can utilize various existing methods such as SIFT (Scale-Invariant Feature Transform) and SURF (Speed-Upped Robust Feature). The first matching unit 201 may also perform a matching process between multiple feature regions of an object in the image data and multiple feature regions of the same object in the CAD data to calculate the positions and orientations of these multiple feature regions in the coordinate system of the robot arm 310. In this case, the first matching unit 201 may output the calculated positions and orientations of the multiple feature regions of the object, or it may calculate (estimate) and output the position and orientation of the object's center of gravity based on the positions and orientations of the multiple feature regions.

[0101] Furthermore, the feature region of the object used to calculate its position and orientation is not limited to a part of the object's outline, but may be any region in the image that can be distinguished from its surroundings, such as markers on the object or patterns on the object's surface. For example, the feature region of the circuit board T as the object may be, for example, a marker (e.g., a 2D code such as an AR marker or a cross mark) or solder pads or solder formed on at least a part of the circuit board T, or a marker (e.g., a 2D code such as an AR marker or a cross mark) placed on the circuit board T, or an element placed on the circuit board T, or a marker (e.g., a 2D code such as an AR marker or a cross mark) placed in the vicinity of the circuit board T. Furthermore, the data that the first matching unit 201 matches with the feature region in the image data output from the detection device 320 is not limited to CAD data, but other data may be used. For example, the first matching unit 201 may perform a matching process between the feature region of image data generated by imaging a reference object (e.g., a reference circuit board) in advance and the feature region of the image data of the circuit board T output from the detection device 320. In this case, the reference object may be, for example, a good product (e.g., a good circuit board).

[0102] Furthermore, the matching rate is a value that indicates the degree of agreement between the feature region of the object in the image data and the feature region of the same object in the CAD data (here, the higher the degree of agreement between the two, the higher the matching rate).

[0103] Furthermore, the first matching unit 201 can also calculate (estimate) the position and orientation of an object by comparing, for example, the point cloud data of a feature region of the object (for example, a part of the corner of the object in the point cloud data) in the shape data (for example, point cloud data) output from the detection device 320 with the point cloud data of a feature region of the object in the CAD data of the object (for example, the point cloud data in the CAD data corresponding to a part of the corner of the object in the shape data).

[0104] At this time, the first matching unit 201 first extracts the point cloud data of the feature region of the object in the shape data (e.g., point cloud data) output from the detection device 320 and the point cloud data of the feature region of the object in the CAD data of the object. Next, the first matching unit 201 calculates the position and orientation of the object in the coordinate system of the robot arm 310 by, for example, changing the coordinates of each point and the spacing between each point in the point cloud data of the feature region of the object in the CAD data, or by rotating the point cloud, thereby establishing a correspondence between the point cloud data of the feature region of the object in the shape data and the point cloud data of the feature region of the object in the CAD data. More specifically, the first matching unit 201 first determines the correspondence between the CAD coordinate system and the coordinate system of the detection device 320 so that the point cloud data of the feature region of the object in the CAD data matches the point cloud data of the feature region of the object in the shape data. Since the position and orientation of the point cloud of the feature region of the object in the CAD coordinate system are known, the position and orientation of the point cloud of the feature region of the object in the coordinate system of the detection device 320 can be determined by aligning the CAD coordinate system with the coordinate system of the detection device 320. Next, the first matching unit 201 calculates the position and orientation of the object in the coordinate system of the robot arm 310 based on the correspondence between the CAD coordinate system and the coordinate system of the detection device 320, and the correspondence between the coordinate system of the detection device 320 and the coordinate system of the robot arm 310, which was determined in advance by calibration.

[0105] Furthermore, the shape data matching process performed by the first matching unit 201 can utilize various existing methods such as RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), and ICP (Iterative Closest Point). The first matching unit 201 may also output the calculated position and orientation of the object's feature regions as the object's position and orientation. Alternatively, the first matching unit 201 may calculate the position and orientation of multiple feature regions in the coordinate system of the robot arm 310 by matching the point cloud data of multiple feature regions of the object in the shape data with the point cloud data of the same object in the CAD data. In this case, the first matching unit 201 may output the calculated position and orientation of multiple feature regions in the object as the object's position and orientation, or it may calculate (estimate) the position and orientation of the object's center of gravity based on the position and orientation of multiple feature regions and output it as the object's position and orientation.

[0106] Furthermore, the first matching unit 201 may perform matching processing using depth images as shape data output from the detection device 320, rather than being limited to matching processing using point cloud data.The feature region of the object for which the position and orientation are calculated is not limited to a part of the corner of the object, but may be any region in which a difference in shape from the surroundings can be identified, such as the edges or irregularities of the object.For example, the feature region of the circuit board T as the object may be a marker (e.g., a code or cross mark with a three-dimensional shape), solder pads or solder formed on the circuit board T as at least a part of the circuit board T, or a marker (e.g., a code or cross mark with a three-dimensional shape) placed on the circuit board T, or an element placed on the circuit board T.The data that the first matching unit 201 matches with the feature region in the image data output from the detection device 320 is not limited to CAD data, but other data may be used. For example, the first matching unit 201 may perform a matching process between the point cloud data of a feature region in shape data (e.g., point cloud data) generated by imaging a reference object (e.g., a reference circuit board) in advance, and the point cloud data of a feature region in the shape data of the circuit board T output from the detection device 320. In this case, the reference object may be, for example, a good product (e.g., a good circuit board). Note that both CAD data and image data and shape data obtained by imaging a reference object are reference data in the matching process, and can therefore be referred to as reference data.

[0107] Furthermore, the matching rate is a value that indicates the degree of agreement between the point cloud data of the feature region of the object in the shape data and the point cloud data of the feature region of the same object in the CAD data (here, the higher the degree of agreement between the two, the higher the matching rate).

[0108] Similarly, the second matching unit 202 performs matching between the output of the detection device 330 (for example, at least one of image data and shape data) and the CAD data. As a result of the matching, the second matching unit 202 outputs the position and orientation estimation result of the object and the matching rate. The matching process in the second matching unit 202 (i.e., estimation of the position and orientation of the object and calculation of the matching rate) is the same as that of the first matching unit 201 described above, so its explanation is omitted.

[0109] In other words, the first matching units 201 and 202 each calculate (estimate) the position and orientation of the object and output the calculation result as the position and orientation estimation result. Here, for example, in order to bring the light irradiation device 60 of the robot 3 closer to the area on the circuit board T where the processing light L should be irradiated, the object whose position and orientation are calculated by the first matching unit 201 and the second matching unit 202 each may be the area to be irradiated with the processing light L itself, for example, at least a part of the circuit board T (for example, a solder pad formed on the circuit board T), or an element or solder placed on the circuit board T. Alternatively, the object may be an area whose relative position to the area to be irradiated with the processing light L is known, for example, at least a part of the circuit board T (for example, a two-dimensional code or cross mark such as an AR marker formed on the circuit board T), a marker placed on the circuit board T (for example, a two-dimensional code or cross mark such as an AR marker), or a marker placed near the circuit board T (for example, a two-dimensional code or cross mark such as an AR marker). Furthermore, the object may be located in a location other than the aforementioned area on the circuit board T. In other words, the object whose position and orientation are calculated by the first matching units 201 and 202 may be the characteristic area described above.

[0110] The CAD data and reference object data (image data and shape data) used for matching are assumed to be pre-stored in the control device 1000, for example, in memory (not shown). The first matching unit 201 and the second matching unit 202 may calculate (estimate) both the position and orientation of the object, or they may calculate (estimate) only one of the position and orientation of the object. In other words, the first matching unit 201 and the second matching unit 202 may calculate (estimate) at least one of the position and orientation of the object.

[0111] The comparison unit 203 compares the matching rate output from the first matching unit 201 (hereinafter referred to as the "first matching rate") with the matching rate output from the second matching unit 202 (hereinafter referred to as the "second matching rate").

[0112] The comparison unit 203 outputs the position and orientation estimation result output from the first matching unit 201 if the first matching rate is greater than the second matching rate, in other words, if the second matching rate is less than the first matching rate (first matching rate > second matching rate). On the other hand, the comparison unit 203 outputs the position and orientation estimation result output from the second matching unit 202 if the second matching rate is equal to or greater than the first matching rate, in other words, if the first matching rate is less than or equal to the second matching rate (first matching rate ≤ second matching rate).

[0113] The robot control unit 100, as part of the control device 1000, controls, for example, the drive unit 311 of the robot 3 based on the position and orientation estimation results output from the matching processing unit 200.

[0114] The comparison unit 203 may perform the above-mentioned comparison of matching rates for all matching process results (i.e., calculation results of the position and orientation of the object) output from the first matching unit 201 and the second matching unit 202 at predetermined time intervals. Alternatively, the comparison unit 203 may not perform the above-mentioned comparison of matching rates for all matching process results output from the first matching unit 201 and the second matching unit 202 at predetermined time intervals. In this case, the comparison unit 203 may perform the above-mentioned comparison of matching rates at a predetermined time, and thereafter output the results of the matching process output from the first matching unit 201 or the second matching unit 202 at predetermined time intervals based on the matching rate comparison result performed at the predetermined time.

[0115] Furthermore, the control device 1000 only needs to output at least one of the results of the matching process using image data and the results of the matching process using shape data from at least one of the first matching unit 201 and the second matching unit 202 via the comparison unit 203.

[0116] As mentioned above, each of the detection devices 320 and 330 of the robot 3 is capable of outputting at least one of image data and shape data. Therefore, each of the first matching unit 201 and the second matching unit 202 may perform at least one of two matching processes: matching using image data (i.e., data representing a two-dimensional image) (hereinafter referred to as "2D matching" as appropriate) and matching using shape data (e.g., three-dimensional point cloud data) (hereinafter referred to as "3D matching" as appropriate). 2D matching can calculate the position and orientation of an object faster than 3D matching. On the other hand, 3D matching can calculate (estimate) the position and orientation of an object with higher accuracy than 2D matching. Therefore, depending on the purpose, each of the first matching unit 201 and the second matching unit 202 may perform either 2D matching or 3D matching.

[0117] When performing 3D matching, using all of the shape data (for example, all of the generated point cloud data) can result in a relatively long time required for 3D matching. Therefore, the first matching unit 201 and the second matching unit 202 may perform the following processing to shorten the time required for 3D matching.

[0118] The first matching unit 201 and the second matching unit 202 each first identify the position of an object obj in a two-dimensional image, for example, as shown in Figure 10(a), based on the results of 2D matching. Next, the first matching unit 201 and the second matching unit 202 each determine the range A (see Figure 10(b)) in which 3D matching should be performed based on the identified position of the object obj (in other words, they narrow down the range in which 3D matching should be performed). The object obj whose position is identified by the 2D matching of the first matching unit 201 and the second matching unit 202 may be the feature region of the object as described above. Then, the first matching unit 201 and the second matching unit 202 each perform 3D matching using shape data corresponding to the determined range A (for example, point cloud data included in range A). Therefore, since the first matching unit 201 and the second matching unit 202 each perform 3D matching with the minimum necessary point cloud data, the time required for extracting point cloud data of feature regions can be shortened compared to conventional methods, thus enabling faster 3D matching. Furthermore, 3D matching allows for the highly accurate calculation (estimation) of the position and orientation of the target object (feature region).

[0119] To illustrate this process, let's refer to the timing chart in Figure 11 as an example. In Figure 11, the width of the black band along the time axis represents the length of time required for each process.

[0120] In Figure 11, T 2-1 The results of the first 2D matching are used to determine the area where 3D matching should be performed (see “Area crop” in Figure 11), and then T 3-1 The 3D matching process will be performed for the second time. 3-1 In parallel with the first 3D matching, T2-2 Round and T 2-3 The second - round 2D matching is performed. T 3-1 When the third - round 3D matching ends, T 2-4 Using the result of the second - round 2D matching, the range in which 3D matching should be performed is determined, and then T 3-2 The third - round 3D matching is performed. In this case, the above - mentioned comparison unit 203 (see FIG. 9) sequentially compares the results of 3D matching by each of the first matching unit 201 and the second matching unit 202, and outputs the position - attitude estimation result with a high matching rate to the robot control unit 100 at intervals of a predetermined time. The robot control unit 100 controls, for example, the drive unit 311 of the robot 3 based on the position - attitude estimation result output from the matching processing unit 200 at intervals of a predetermined time.

[0121] In addition, each of the first matching unit 201 and the second matching unit 202 is not used for 3D matching T 2-2 Round and T 2-3 The second - round 2D matching may not be performed. In addition, the comparison unit 203 may compare the results of 2D matching and output the position - attitude estimation result with a high matching rate to the robot control unit 100. Note that it is not limited to the timing chart of FIG. 11. Each of the first matching unit 201 and the second matching unit 202 may perform 2D matching and 3D matching at a predetermined timing (for example, the timing when the tracking process described later starts, etc.), determine the range in which 3D matching should be performed using the result of 2D matching as described above, and perform 3D matching.

[0122] The matching process described above is merely an example and is not limited thereto. The control device 1000 may calculate (estimate) the position and orientation of an object (feature region) using only 3D matching (i.e., matching using shape data), or it may calculate (estimate) the position and orientation of an object (feature region) using only 2D matching (i.e., matching using image data). When calculating (estimate) the position and orientation of an object using only 2D matching, the detection devices 320 and 330 may be equipped with only a single camera. Furthermore, if either the detection device 320 or 330 is installed on the robot 3, the matching processing unit 200 may consist of only either the first matching unit 201 or the second matching unit 202.

[0123] (Calculation of the object's position and orientation through tracking) Due to the transport of the circuit board T by the belt conveyor, and / or due to the movement of the light irradiation device 60 by, for example, the robot arm 310 of robot 3, the relative position between the object and the light irradiation device 60 changes over time.

[0124] To ensure that the position and orientation of the light irradiation device 60 of the robot 3 are in the desired position and orientation relative to a part of the circuit board T (for example, the area to be irradiated with processing light L) that is being transported by the conveyor belt, the robot arm 310 of the robot 3 needs to bring the light irradiation device 60 closer to the part of the circuit board T that is being transported by the conveyor belt. In order to bring the light irradiation device 60 closer to the object with the robot arm 310, the control device 1000 can perform tracking processing to calculate (estimate) the changes in the position and orientation of the object. The object for which the changes in position and orientation are calculated (estimated) may be the area to be irradiated with processing light L itself, as described above, or it may be, for example, at least a part of the circuit board T (for example, a solder pad formed on the circuit board T), or an element or solder placed on the circuit board T. Furthermore, the target object may be a location whose relative position to the area to be irradiated with the processing light L described above is known. For example, it may be at least a part of the circuit board T (for example, a two-dimensional code such as an AR marker or a cross mark formed on the circuit board T as a marker), a marker placed on the circuit board T (for example, a two-dimensional code such as an AR marker or a cross mark), or a marker placed near the circuit board T (for example, a two-dimensional code such as an AR marker or a cross mark). In addition, the target object may be a location on the circuit board T other than the aforementioned location.

[0125] The tracking process performed in the control device 1000 will be explained with reference to Figures 12 and 13.

[0126] In Figure 12, the control device 1000 has a tracking unit 300, which is either a logically implemented processing block or a physically implemented processing circuit. The tracking unit 300 is configured to include a matching unit 301, a 2D tracking unit 302, and a 3D tracking unit 303.

[0127] Image data and shape data are input to the tracking unit 300 at predetermined time intervals. Here, if the position and orientation estimation result output from the first matching unit 201 of the matching processing unit 200 described above is output to the robot control unit 100 (i.e., if the first matching rate > second matching rate), then, for example, the image data and shape data output from the detection device 320 of the robot 3 at predetermined time intervals are input to the tracking unit 300. On the other hand, if the position and orientation estimation result output from the second matching unit 202 of the matching processing unit 200 described above is output to the robot control unit 100 (i.e., if the first matching rate ≤ second matching rate), then, for example, the image data and shape data output from the detection device 330 of the robot 3 at predetermined time intervals are input to the tracking unit 300.

[0128] The matching unit 301 calculates (estimates) the position and orientation of an object by performing a matching process on image data and shape data input at predetermined time intervals. The matching process in the matching unit 301 is the same as that of the first matching unit 201 and the second matching unit 202 described above, so its explanation is omitted. Alternatively, as described above, the matching unit 301 may narrow down the range in which 3D matching should be performed based on the results of 2D matching using the input image data, and then calculate (estimate) the position and orientation of the object by performing 3D matching using the shape data corresponding to that range (see Figure 10).

[0129] Image data is input to the 2D tracking unit 302 at predetermined time intervals. The 2D tracking unit 302 calculates (estimates) the displacement of the object at predetermined time intervals based on two image data sets, for example, a first image data set and a second image data set, which are input at predetermined time intervals. Meanwhile, at least the first image data input to the 2D tracking unit 302 is also input to the matching unit 301. In other words, the image data input to the tracking unit 300 is input to the 2D tracking unit 302 and the matching unit 301 at approximately the same time. The matching unit 301 calculates (estimates) the position and orientation of the object by performing the matching process described above using the input first image data.

[0130] The 2D tracking unit 302 calculates (estimates) the position and orientation of the object at a predetermined time (i.e., the time when the second image data is generated) by adding the calculated (estimated) displacement of the object to the position and orientation of the object calculated (estimated) by the matching unit 301 in the very beginning (hereinafter also referred to as the initial position and orientation). Subsequently, the 2D tracking unit 302 sequentially calculates (estimates) the displacement of the object at predetermined time intervals based on each image data input at predetermined time intervals, and calculates (estimates) the position and orientation of the object at each time by adding the calculated (estimated) displacement of the object to the position and orientation of the object that have been calculated (estimated) each time (i.e., performs tracking processing). The position and orientation of the object calculated (estimated) at each time by the tracking processing will also be expressed as 6DoF values, similar to the position and orientation calculated in the matching processing described above. More specifically, the 2D tracking unit 302 first extracts the feature region of the object in the first image data (for example, a part of the object's outline in the first image data) and the feature region of the object in the second image data (for example, a part of the object's outline in the second image data) at each input timing. Next, the 2D tracking unit 302 associates the feature region of the object in the first image data with the feature region of the object in the second image data and determines the displacement of the feature region of the object in the second image data with respect to the feature region of the object in the first image data in the camera coordinate system. Furthermore, based on the obtained displacement of the feature region of the object in the second image data with respect to the feature region of the object in the first image data in the camera coordinate system and the previously determined correspondence between the camera coordinate system and the coordinate system of the robot arm 310, the 2D tracking unit 302 calculates the displacement of the object in the robot arm 310 with respect to the object at the time the first image data was generated and the object at the time the second image data was generated.

[0131] The 2D tracking unit 302 then calculates the position and orientation of the object at a predetermined time (i.e., the time when the second image data is generated) by adding the aforementioned displacement to the position and orientation (initial position and orientation) of the object's feature region calculated by the matching unit 301. Since the time required for the 2D tracking unit 302 to calculate the position and orientation of the object is shorter than the time required for the 3D tracking unit 303, which will be described later, the position and orientation of the object can be tracked at high speed. Various existing methods can be used for extracting the feature region of the object in the first and second image data, associating the feature region of the object in the first image data with the feature region of the object in the second image data, and calculating the displacement of the feature region of the object in the coordinate system of the robot arm 310.

[0132] Existing methods can be used to extract feature regions of the object in the first and second image data, to associate the feature regions of the object in the first image data with the feature regions of the object in the second image data, and to calculate the displacement of the feature regions of the object in the second image data relative to the feature regions of the object in the first image data in the global coordinate system. The 2D tracking unit 302 may extract multiple feature regions of the object in the first image data and multiple feature regions of the object in the second image data, associate these feature regions with each other, and calculate (estimate) the displacement of each feature region of the object in the second image data relative to each feature region of the object in the first image data. The 2D tracking unit 302 may also use the position and orientation estimation result of the object output from the comparison unit 203 (i.e., the position and orientation estimation result with a high matching rate among the 3D matching results output from the first matching unit 201 and the second matching unit 202, respectively, as referenced in Figures 9 and 10) as the initial position and orientation. The characteristic region of the circuit board T as the object may be, for example, a marker (e.g., a two-dimensional code such as an AR marker or a cross mark) or solder pad or solder formed on the circuit board T as at least a part of the circuit board T, or a marker (e.g., a two-dimensional code such as an AR marker or a cross mark) placed on the circuit board T, or an element placed on the circuit board T, or a marker (e.g., a two-dimensional code such as an AR marker or a cross mark) placed in the vicinity of the circuit board T.

[0133] Shape data is input to the 3D tracking unit 303 at predetermined time intervals. The 3D tracking unit 303 calculates (estimates) the displacement of the object at predetermined time intervals based on two shape data sets, a first shape data set and a second shape data set, which are input at predetermined time intervals. Meanwhile, at least the first shape data input to the 3D tracking unit 303 is also input to the matching unit 301. In other words, the shape data input to the tracking unit 300 is input to the 3D tracking unit 303 and the matching unit 301 at approximately the same time. The matching unit 301 calculates (estimates) the position and orientation of the object by performing the matching process as described above using the input first shape data.

[0134] The 3D tracking unit 303 determines the three-dimensional displacement of the object over a very short time. The 3D tracking unit 303 estimates the current orientation and position of the object by adding the determined displacement to the initial orientation and position detected by the matching unit 301.

[0135] The 3D tracking unit 303 calculates the position and orientation of the object at a predetermined time (i.e., the time when the second shape data is generated) by adding the calculated displacement of the object to the position and orientation of the object (initial position and orientation) calculated by the matching unit 301 in the very beginning. Subsequently, the 3D tracking unit 303 sequentially calculates the displacement of the object at predetermined time intervals based on each shape data input at predetermined time intervals, and calculates the position and orientation of the object at each time by adding the calculated displacement of the object to the calculated position and orientation of the object (i.e., performing tracking processing). More specifically, for example, if the shape data is point cloud data, the 3D tracking unit 303 first extracts the point cloud data of the feature region of the object in the first shape data (e.g., a part of the corner of the object in the first shape data) and the point cloud data of the feature region of the object in the second shape data (e.g., a part of the corner of the object in the second shape data) at each input timing. The 3D tracking unit 303 then associates the point cloud data of the feature region of the object in the first shape data with the point cloud data of the feature region of the object in the second shape data, and determines the displacement in the camera coordinate system of the point cloud data of the feature region of the object in the second shape data with respect to the point cloud data of the feature region of the object in the first shape data. The 3D tracking unit 303 further calculates the displacement in the coordinate system of the robot arm 310 of the object at the time the first shape data was generated and the displacement in the camera coordinate system of the object at the time the second shape data was generated, based on the determined displacement in the camera coordinate system of the point cloud data of the feature region of the object in the second shape data with respect to the point cloud data of the feature region of the object in the first shape data, and the correspondence between the camera coordinate system and the coordinate system of the robot arm 310 that was determined in advance.

[0136] The 3D tracking unit 303 then calculates the position and orientation of the object at a predetermined time (i.e., the time when the second shape data is generated) by adding the aforementioned displacement to the position and orientation (initial position and orientation) of the point cloud data of the feature region of the object calculated by the matching unit 301. When the shape data is point cloud data, the tracking process performed by the 3D tracking unit 303 can utilize various existing methods such as RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), and ICP (Iterative Closest Point).

[0137] Furthermore, for example, if the shape data is depth image data, the 3D tracking unit 303 first extracts the feature region of the object in the first shape data (for example, a part of the corner of the object in the first shape data) and the feature region of the object in the second shape data (for example, a part of the corner of the object in the second shape data) at each input timing. Next, the 3D tracking unit 303 associates the feature region of the object in the first shape data with the feature region of the object in the second shape data and calculates the displacement in the camera coordinate system of the feature region of the object in the second shape data with respect to the feature region of the object in the first shape data. The 3D tracking unit 303 further calculates the displacement in the coordinate system of the robot arm 310 of the object at the time the first shape data was generated and the displacement of the object at the time the second shape data was generated, based on the obtained displacement in the camera coordinate system of the object in the second shape data with respect to the feature region of the object in the first shape data and the correspondence between the camera coordinate system and the coordinate system of the robot arm 310 that was determined in advance. The 3D tracking unit 303 then calculates the position and orientation of the object at a predetermined time (i.e., the time when the second shape data is generated) by adding the aforementioned displacement to the position and orientation of the feature region of the object (initial position and orientation) calculated by the matching unit 301. If the shape data is depth image data, the tracking process by the 3D tracking unit 303 can utilize various existing methods such as DSO (Direct Sparse Odometry). Since the accuracy of the position and orientation of the object calculated by the 3D tracking unit 303 is higher than that of the position and orientation of the object calculated by the 2D tracking unit 302, the position and orientation of the object can be tracked with high accuracy.

[0138] Furthermore, the position and orientation of the object estimated by the 2D tracking unit 302, the position and orientation of the object estimated by the 3D tracking unit, and the position and orientation of the object detected by the matching unit 301 may each be output in a format corresponding to the position and orientation estimation results output from the first matching unit 201 and the second matching unit 202 described above (i.e., in 6DoF format).

[0139] Here, we will explain the tracking process by referring to the timing chart in Figure 13 as an example. In Figure 13, the width of the black band along the time axis represents the length of time required for each process.

[0140] In Figure 13, the initial position and orientation of the object (initial position and orientation) have already been detected by the matching unit 301, and the current position and orientation of the object are estimated by adding the displacement of the object at predetermined time intervals to this initial position and orientation. Furthermore, the position and orientation estimation accuracy of the 3D tracking unit 303 and the matching unit 301 is assumed to be higher than that of the 2D tracking unit 302.

[0141] For example, let x1 be the position and orientation of the object at time t1 (i.e., the position and orientation estimated by the matching unit 301). For example, let Δx be the displacement of the object from time t1 to time t2, as determined by the 2D tracking unit 302. 12 The displacement of the object from time t2 to time t3 is Δx 23 In this case, the position and orientation of the object at time t3, estimated by the 2D tracking unit 302, are "x1 + Δx 12 +Δx 23 It is expressed as ".

[0142] Displacement “Δx 12 " and "Δx 23 Since there is an estimation error in each of these, the error in the position and orientation of the object estimated by the 2D tracking unit 302 increases each time the displacement detected by the 2D tracking unit 302 is cumulatively added to the position and orientation "x1". On the other hand, the processing time required in the 2D tracking unit 302 is shorter than the processing time required in the matching unit 301 and the 3D tracking unit 303.

[0143] Therefore, the position and orientation of the object estimated by the 2D tracking unit 302 can be corrected using at least one of the position and orientation of the object estimated by the matching unit 301 and the position and orientation of the object estimated by the 3D tracking unit 303, thereby achieving both faster tracking and improved estimation accuracy.

[0144] Specifically, the T of the 2D tracking unit 302 2-1 The 3D tracking unit 303 uses the image data and shape data from the same period used in the first processing step. 3-1 The position and orientation of the object estimated by the first processing, and the above T 2-1 The difference between the position and orientation of the object estimated by the first processing is considered as the error in the position and orientation of the object estimated by the 2D tracking unit 302. Then, T 3-1 When the first processing is completed, the processing that is in progress in the 2D tracking unit 302 (for example, T 2-2 The error may be corrected in the second processing step or in the processing step following the said processing step.

[0145] Similarly, the matching unit 301 is T i-1 In the first processing step, the T of the 2D tracking unit 302 2-1 When the position and orientation of an object are calculated (estimated) from the image data used in the first processing step, T i-1 The position and orientation calculated (estimated) by the first processing, and the above T 2-1 The difference between the position and orientation of the object estimated by the first processing is considered as the error in the position and orientation of the object estimated by the 2D tracking unit 302. Then, T i-1 When the first processing is completed, the processing that is in progress in the 2D tracking unit 302 or the processing that will be performed next (for example, T 2-9 The error may be corrected in the (1st) processing step.

[0146] Furthermore, the matching unit 301 is T i-1 In the first processing step, the T of the 3D tracking unit 303 3-1When the position and orientation of an object are calculated (estimated) from the shape data used in the first processing step, T i-1 The position and orientation detected by the first processing, and the above T 3-1 The difference between the position and orientation of the object estimated by the first processing is considered as the error in the position and orientation of the object estimated by the 3D tracking unit 303. Then, T i-1 When the first processing is completed, the error may be corrected in the processing currently underway in the 3D tracking unit 303 or in the processing that follows said processing.

[0147] The position and orientation of the object estimated by the 2D tracking unit 302 are output to the robot control unit 100. The robot control unit 100, as part of the control device 1000, controls, for example, the drive unit 311 of the robot 3 based on the estimated position and orientation of the object.

[0148] The tracking unit 300 may receive either image data or shape data as input, but may not receive the other. The tracking unit 300 may include either a 2D tracking unit 302 or a 3D tracking unit 303, but may not include the other. In this case, either the position and orientation of the object estimated by the 2D tracking unit 302 or the position and orientation of the object estimated by the 3D tracking unit 303 may be output to the robot control unit 100. Furthermore, the position and orientation of the object estimated by the 2D tracking unit 302 may not be corrected using the position and orientation of the object estimated by the 3D tracking unit 303. Alternatively, the position and orientation of the object estimated by the 3D tracking unit 303 may be corrected using the position and orientation of the object estimated by the 2D tracking unit 302. In addition, the tracking unit 300 may select the data to be used for tracking processing regardless of the matching rate comparison result by the matching processing unit 200. In this case, the matching unit 301 of the tracking unit 300 may, for example, compare the matching rate when the position and orientation of the object are calculated (estimated) using image data and shape data output from the detection device 320 with the matching rate when the position and orientation of the object are calculated (estimated) using image data and shape data output from the detection device 330, and select (in other words, switch) the data to be used for tracking processing based on the result of the comparison.

[0149] Alternatively, the tracking unit 300 may always perform tracking processing using image data and shape data output from the detection device 330. Alternatively, the tracking unit 300 may always perform tracking processing using image data and shape data output from the detection device 320. In other words, the tracking processing described above can be performed using image data and shape data output from only one of the detection devices 320 and 330. In this case, for example, the robot 3 may have only one of the detection devices 320 and 330. For example, if the robot 3 has only one of the detection devices 320 and 330, the tracking processing described above may be performed using only the image data output from that one detection device, or using only the shape data output from that one detection device.

[0150] The tracking process described above is merely an example and is not limited thereto. In other words, the control device 1000 may perform a well-known tracking process instead of the tracking process described above. Furthermore, the control device 1000 does not have to perform the tracking process described above. In this case, the control device 1000 does not have to include the tracking unit 300 described above, but only needs to include the matching processing unit 200 described above. The control device 1000 may perform a matching process using at least one of the image data and shape data at predetermined time intervals, and control, for example, the drive unit 311 of the robot 3 based on the calculated (estimated) position and orientation of the object.

[0151] (Operation of each robot) The operation of robots 1, 2, and 3 will be explained with reference to the flowchart in Figure 14. Here, we will mainly explain robot 3, and explanations for robots 1 and 2 that are common to robot 3 will be omitted as appropriate.

[0152] Here, the belt conveyor temporarily stops after transporting the circuit board T within the driveable range of, for example, the robot arm 110 of robot 1. Then, after solder dispensed from the dispenser 40 is placed on a predetermined portion of the stopped circuit board T, the belt conveyor resumes driving to transport the circuit board T. The belt conveyor temporarily stops again after transporting the circuit board T within the driveable range of, for example, the robot arm 210 of robot 2. Then, after elements held by the holding device 50 are placed on a predetermined portion of the stopped circuit board T, the belt conveyor resumes driving to transport the circuit board T. The belt conveyor temporarily stops again after transporting the circuit board T within the driveable range of, for example, the robot arm 310 of robot 3. Then, after processing light L from the light irradiation device 60 is irradiated onto a predetermined portion of the stopped circuit board T, the belt conveyor resumes driving to transport the circuit board T.

[0153] Robot 3 First, the control device 1000 may perform calibration of the light irradiation device 60 before performing the following steps S131 to S138.

[0154] As a prerequisite, the detection device 330 and the light illumination device 60 are mounted on the robot arm 310 in such a positional relationship that a part of the light illumination device 60 (for example, its tip) is within the field of view of the cameras 31 and 32 of the detection device 330, respectively.

[0155] As calibration of the light irradiation device 60, the control device 1000 performs the matching process described above using shape data including a part of the light irradiation device 60 output from the detection device 330 and CAD data of the light irradiation device 60, and pre-calculates the position and orientation of the light irradiation device 60 (for example, the position and orientation of the tip of the light irradiation device 60 included in the field of view of the cameras 31 and 32 of the detection device 330). In other words, the control device 1000 pre-calculates the position and orientation of the light irradiation device 60 in the coordinate system of the robot arm 310 based on the shape data of at least a part of the light irradiation device 60.

[0156] Markers may be provided on a portion of the light illuminator 60 that is included within the field of view of each of the cameras 31 and 32 of the detection device 330. In this case, the control device 1000 may perform the above-described calibration based, for example, on shape data including the markers output from the detection device 330.

[0157] Furthermore, the control device 1000 may perform calibration of the light irradiation device 60 by performing a matching process using not only shape data, but also image data output from the detection device 330 and CAD data of the light irradiation device 60. In addition, as described above, the control device 1000 may use not only CAD data, but also shape data or image data of the light irradiation device 60 that have been acquired in advance in the matching process.

[0158] Furthermore, the control device 1000 may use shape data or image data output from the detection device 320, rather than being limited to using the detection device 330 for calibration of the light irradiation device 60. In this case, it is assumed that the detection device 320 and the light irradiation device 60 are mounted on the robot arm 310 in such a positional relationship that a part of the light irradiation device 60 is within the field of view of the cameras 21 and 22 of the detection device 320, respectively.

[0159] In the following steps S131 to S138, the position and orientation of the light irradiator 60 relative to the detection device 330 may change, for example, by the light irradiator 60 coming into contact with a predetermined object. In this case, the control device 1000 can detect the change in the position and orientation of the light irradiator 60 relative to the detection device 330 based on a partial change in the light irradiator 60 in the image data or shape data output from the detection device 330 (for example, a partial change in the light irradiator 60 in the image). If a change in the position and orientation of the light irradiator 60 relative to the detection device 330 is detected, the control device 1000 may perform calibration.

[0160] The control device 1000, which controls the robot 3, calculates (estimates) the position and orientation of the circuit board T as an example of an object (step S131). In the process of step S131, the control device 1000 calculates the initial position and orientation of the circuit board T (i.e., the initial position and orientation) by the matching process of the matching unit 301. Furthermore, the control device 1000 calculates the position of each solder pad formed on the circuit board T. The control device 1000 calculates the position and orientation of each solder pad on the circuit board T based on the Gerber data of the circuit board T (i.e., the design data of the circuit board T). More specifically, the control device 1000 calculates the position and orientation of each solder pad on the circuit board T from the calculated initial position and orientation of the circuit board T and the positional relationship of each solder pad on the circuit board T. In addition, the Gerber data of the circuit board T contains data regarding the order in which elements are mounted on each solder pad, and the control device 1000 identifies the mounting order for each solder pad based on the Gerber data. Furthermore, the control device 1000 does not need to calculate the initial position and orientation of the circuit board T, and may calculate the position and orientation of any object usable for the matching process described above, such as a cross mark formed on the circuit board T, or an AR marker placed on or near the circuit board T. Furthermore, the control device 1000 does not need to identify the position and orientation of each solder pad or the mounting order based on Gerber data, and may identify them using other design data of the circuit board T (e.g., CAD data), or using information input by the user via an interface not shown. Furthermore, the control device 1000 does not need to calculate the position and orientation of each solder pad on the circuit board T, and may calculate, for example, the position and orientation of the area to be irradiated with processing light L, such as the element itself, the area where the element is placed, or the area where solder is placed, or the area near that area.

[0161] Next, the control device 1000 controls the drive unit 311 to move the robot arm 310 so that the detection devices 320 and 330 (and furthermore, the light irradiation device 60) move closer to the circuit board T (step 132). In step S132, the control device 1000 controls the drive unit 311 of the robot arm 310 so that the element placed on the first solder pad to be mounted (for example, a chip LED with two electrodes) comes into the field of view of at least one of the detection devices 320 and 330. More specifically, the control device 1000 uses the initial position and orientation of the circuit board T and the position and orientation information of the first solder pad to be mounted calculated (estimated) in step S131 to control the drive unit 311 to move the robot arm 310 based on the position and orientation of the circuit board T output from the 2D tracking unit 302 of the tracking unit 300 at predetermined time intervals.

[0162] Next, the control device 1000 determines whether the element placed on the first solder pad to be mounted is within the field of view of at least one of the detection devices 320 and 330 (step S133). In the process of S133, the control device 1000 determines whether the detection devices 320 and 330 are in the desired position and orientation relative to the first solder pad to be mounted, based on the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals and the position and orientation information of the first solder pad to be mounted calculated in step S131. At this time, if the detection devices 320 and 330 are in the desired position and orientation relative to the first solder pad to be mounted, the control device 1000 determines that the element placed on the solder pad is within the field of view of at least one of the detection devices 320 and 330. Furthermore, the control device 1000 is not limited to the above determination method. It may also determine whether the position and orientation information output from the 2D tracking unit 302 at predetermined time intervals includes position and orientation information of the element placed on the first solder pad, or it may determine whether the image data and shape data generated by at least one of the detection devices 320 and 330 include information about the element.

[0163] In step S133, if it is determined that the element placed on the first solder pad to be mounted is not within the field of view of at least one of the detection devices 320 and 330 (step S133: No), the control device 1000 controls the drive unit 311 to continue moving the robot arm 310 based on the position and orientation information of the first solder pad to be mounted calculated (estimated) in step S131 and the position and orientation information of the circuit board T output from the 2D tracking unit 302 of the tracking unit 300 at predetermined time intervals. In other words, the process in step S132 is performed until it is determined that the element placed on the first solder pad to be mounted is within the field of view of at least one of the detection devices 320 and 330.

[0164] On the other hand, in the process of step S133, if it is determined that the element placed on the first solder pad to be mounted has entered the field of view of at least one of the detection device 320 and the detection device 330 (step S133: Yes), the control device 1000 calculates (estimates) the position and orientation of the element placed on the first solder pad to be mounted (step S134). In the process of step S134, the control device 1000 calculates (estimates) the initial position and orientation (initial position and orientation) of the element placed on the first solder pad to be mounted by the matching process of the matching unit 301. Note that the control device 1000 does not have to calculate (estimate) the initial position and orientation of the element, and may instead calculate (estimate) the position and orientation of any object that can be used for the matching process described above, such as solder pads and cross marks formed on the circuit board T, or AR markers and solder placed on the circuit board T.

[0165] Next, the control device 1000 moves the robot arm 310 by controlling the drive unit 311 so that the position and orientation of the light irradiation device 60 are in a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad (step S135). In the process of step S135, the control device 1000 uses the initial position and orientation information of the element calculated (estimated) in the process of step S134 to control the drive unit 311 based on the position and orientation of the element output from the 2D tracking unit 302 of the tracking unit 300 at predetermined time intervals, thereby moving the robot arm 310. In other words, the control device 1000 moves the robot arm 310 by controlling the drive unit 311 so that the light irradiation device 60 (detection devices 320 and 330) approaches the element placed on the first solder pad of the circuit board T.

[0166] Next, the control device 1000 determines whether the position and orientation of the light irradiation device 60 are in a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad (step S136). In the process of step S136, the control device 1000 determines whether the position and orientation of the light irradiation device 60 relative to the element is in a desired position and orientation, for example, based on information on the position and orientation of the element output from the 2D tracking unit 302 at predetermined time intervals. At this time, if the position and orientation of the light irradiation device 60 relative to the element is in a desired position and orientation, the control device 1000 determines that the position and orientation of the light irradiation device 60 are in a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad.

[0167] In step S136, if it is determined that the position and orientation of the light irradiation device 60 are not the desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad (step S136: No), the control device 1000 controls the drive unit 311 to move the robot arm 310 by determining that the position and orientation of the element output from the 2D tracking unit 302 at predetermined time intervals, so that the light irradiation device 60 approaches the element placed on the first solder pad. In other words, the process in step S135 is performed until it is determined that the position and orientation are the desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad.

[0168] On the other hand, if the position and orientation of the light irradiation device 60 are determined to be a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad, the control device 1000 controls the light irradiation device 60 to irradiate the electrodes of the element placed on the solder pad, i.e., the two electrodes of the chip LED, with the processing light L so as to melt the solder placed on the solder pad to be mounted first (step S137). As a result, the solder placed on the solder pad melts and the element is soldered to the circuit board T (the solder pad to be mounted first). Here, two specific embodiments of the process in step S137 can be cited as examples.

[0169] In other words, the first embodiment is such that, as shown in Figures 15(a) and (b), for example, when the robot arm 310 is driven by the drive unit 311, that is, when the light irradiation device 60 and the detection devices 320 and 330 are moved by the robot arm 310, the processing light L is irradiated onto the electrodes of the chip LED. In this embodiment, the control device 1000 controls the drive unit 311 so that the light irradiation device 60 and the detection devices 320 and 330 are moved, and may also control the orientation of the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the light irradiation device 60 moved by the robot arm 310 is maintained on one electrode of the chip LED (for example, so that the processing light L from the light irradiation device 60 is irradiated onto the same position on one electrode of the chip LED). For example, the control device 1000 may control the drive unit 311 so that the light irradiation device 60 and the detection devices 320 and 330 are moved, while also controlling the orientation of the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the light irradiation device 60, which is moved by the robot arm 310, is maintained at the same position on one electrode of the chip LED (in other words, so that the processing light L from the light irradiation device 60 continues to irradiate the same position on one electrode of the chip LED for a predetermined time). Note that "same position" is a concept that also includes the possibility that the irradiation position of the processing light L may fluctuate to an extremely small extent at the location where the processing light L should be irradiated, to the extent that it does not affect the melting of the solder.

[0170] For example, as shown in Figures 15(a) and (b), when the light irradiation device 60 is being moved from right to left on the plane of the paper by the robot arm 310, the control device 1000 may control the orientation of the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the moving light irradiation device 60 is maintained on one electrode of the chip LED.

[0171] In the first embodiment described above, for example, the control device 1000 may control the orientation of the galvanometer mirror 61 so that processing light L from the light irradiation device 60 moved by the robot arm 310 is irradiated onto the work object (for example, so that it is irradiated onto the same position on the work object) based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 320 and 330 moved by the robot arm 310.

[0172] In a second embodiment, the control device 1000 controls the drive unit 311 of the robot arm 310, which was driven in step S135, to stop driving, and after the drive unit 311 stops, processing light L is irradiated onto the electrode of the chip LED. In this embodiment, the control device 1000 controls the drive unit 311 to stop driving. After the drive unit 311 stops driving, the control device 1000 may control the orientation of the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the light irradiation device 60 is maintained on one electrode of the chip LED. For example, the control device 1000 controls the drive unit 311 to stop driving. The control device 1000 may, after the drive unit 311 stops, control the orientation of the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the light irradiation device 60 is maintained at the same position on one electrode of the chip LED (so that the processing light L from the light irradiation device 60 continues to irradiate the same position on one electrode of the chip LED for a predetermined time).

[0173] In the second embodiment described above, for example, the control device 1000 may control the orientation of the galvanometer mirror 61 so that processing light L from the light irradiation device 60, which is displaced together with at least one of the detection devices 320 and 330, is irradiated onto the work object (for example, so that it is irradiated onto the same position on the work object), based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 320 and 330 after the drive unit 311 has stopped moving.

[0174] In the first embodiment, for example, when mounting chip LEDs at high speed to multiple solder pads on a circuit board T, the control device 1000 controls the drive unit 311 of the robot arm 310 and the light irradiation device 60 so as to move the light irradiation device 60 (detection devices 320 and 330) from the first solder pad to be mounted to the second solder pad, and sequentially irradiate the electrodes of the chip LEDs placed on the first solder pad (in other words, the first position) and the electrodes of the chip LEDs placed on the second solder pad (in other words, the second position) with processing light L. In this case, in parallel with step S137, the control device 1000 moves the light irradiation device 60 (detection devices 320 and 330) from the first solder pad to be mounted to the second solder pad based on the position and orientation of the second solder pad to be mounted calculated (estimated) in step S131.

[0175] First, the control device 1000 controls the orientation of the galvano mirror 61 of the light irradiation device 60 (detection devices 320 and 330) so that processing light L is irradiated onto the electrodes of the chip LED placed on the first solder pad to be mounted for a predetermined time from the light irradiation device 60 (detection devices 320 and 330) which moves (displaces) relative to the electrodes of the chip LED. More specifically, as the light irradiation device 60 and detection devices 320 and 330 move by the robot arm 310, the control device 1000 gradually changes the orientation of the galvano mirror 61 so that the irradiation position of the processing light L is maintained on one electrode of the chip LED (for example, the same position on one electrode of the chip LED) based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals. At this time, the control device 1000 can recognize changes in the position and orientation of one electrode of the chip LED relative to the light irradiation device 60 based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, and can control the orientation of the galvano mirror 61 so that the irradiation position of the processing light L is maintained on one electrode of the chip LED (for example, the same position on one electrode of the chip LED). Note that the position and orientation of the chip LED in the image shown by the image data output sequentially from at least one of the detection devices 320 and 330, and the position and orientation of the chip LED in the point cloud shown by the shape data, change over time due to the relative displacement between the chip LED and the light irradiation device 60 (detection devices 320 and 330).

[0176] After the control device 1000 has finished irradiating one electrode of the chip LED with processing light L for a predetermined time, it then gradually changes the orientation of the galvano mirror 61 so that the irradiation position of the processing light L is maintained on the other electrode of the chip LED (for example, the same position on the other electrode of the chip LED) based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, as the light irradiation device 60 and detection devices 320 and 330 continue to move by the robot arm 310. Then, after the control device 1000 has finished irradiating the other electrode of the chip LED with processing light L for a predetermined time, it repeats steps S132 to S137 described above to similarly irradiate the electrodes of the chip LEDs placed on the second solder pad to be mounted with processing light L.

[0177] For example, the control device 1000 may control the orientation of the galvanometer mirror 61 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330, so that the irradiation position of the processing light L from the light irradiation device 60, which is displaced together with the displacement of at least one of the detection devices 320 and 330, is maintained at a first position, and then at a second position different from the first position.

[0178] For example, the control device 1000 may control the drive unit 311 to move the light irradiation device 60 and the detection devices 320 and 330, and based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 320 and 330 moved by the robot arm 310, it may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L from the light irradiation device 60 moved by the robot arm 310 is maintained at a first position, and then maintained at a second position different from the first position.

[0179] In the second embodiment, inertial and elastic forces act on the light irradiation device 60 and detection devices 320 and 330 provided on the robot arm 310. For example, the control device 1000 controls the drive unit 311 of the robot arm 310, which was driven in step S135, to stop driving. After the drive unit 311 is stopped, the relative position between the light irradiation device 60 (detection devices 320 and 330) and the chip LED fluctuates to a greater or lesser extent over time due to displacement of the light irradiation device 60 and detection devices 320 and 330 due to vibration or the like. For this reason, the control device 1000 controls the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the light irradiation device 60 is maintained on one electrode of the chip LED (in other words, the first position) for a predetermined time, even if the light irradiation device 60 (detection devices 320 and 330) is displaced due to vibration or the like. Next, the control device 1000 controls the galvanometer mirror 61 of the light irradiation device 60 so that even if the light irradiation device 60 (detection devices 320 and 330) continues to be displaced due to vibration or the like, the irradiation position of the processing light L from the light irradiation device 60 is maintained on the other electrode (in other words, the second position) of the chip LED for a predetermined time.

[0180] More specifically, the control device 1000 first changes the orientation of the galvano mirror 61 over time based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals (due to displacement such as vibration of the light irradiation device 60), so that the irradiation position of the processing light L is maintained on one electrode of the chip LED (for example, the same position on one electrode of the chip LED). Since the control device 1000 can recognize changes in the position and orientation of one electrode of the chip LED relative to the light irradiation device 60, it can control the orientation of the galvano mirror 61 so that the irradiation position of the processing light L is maintained on one electrode of the chip LED (for example, the same position on one electrode of the chip LED). Note that the position and orientation of the chip LED in the image shown by the image data output sequentially from at least one of the detection devices 320 and 330, and the position and orientation of the chip LED in the point cloud shown by the shape data, change over time due to the relative displacement between the chip LED and the light irradiation device 60 (detection devices 320 and 330).

[0181] After the control device 1000 has finished irradiating one electrode of the chip LED with processing light L for a predetermined time, the control device 1000 then changes the orientation of the galvanometer mirror 61 over time, based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals (due to displacement such as vibration of the light irradiation device 60), so that the irradiation position of the processing light L is maintained on the other electrode of the chip LED (for example, the same position on the other electrode of the chip LED).

[0182] Furthermore, when processing light L is irradiated onto one or the other electrode of the chip LED, the position of the chip LED relative to the solder pad may change over time due to the surface tension of the molten solder, etc. In this case, the control device 1000 gradually changes the orientation of the galvano mirror based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, so that the irradiation position of the processing light L is maintained on one or the other electrode of the chip LED. At this time, the control device 1000 can recognize the change in the position and orientation of one or the other electrode of the chip LED relative to the light irradiation device 60 based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, and can control the orientation of the galvano mirror so that the irradiation position of the processing light L is maintained on one electrode of the chip LED.

[0183] Furthermore, not only when the drive unit 311 of the driven robot arm 310 is stopped, but also when the robot arm 310 accelerates or decelerates, or after acceleration or deceleration, the light irradiation device 60 and detection devices 320 and 330 provided on the robot arm 310 are displaced by vibration, etc., and the relative position between the light irradiation device 60 (detection devices 320 and 330) and the area to be irradiated with processing light L (for example, the electrodes of a chip LED) changes over time. However, even in this case, the control device 1000 can recognize the temporal change in the position and orientation of the area to be irradiated with processing light L relative to the light irradiation device 60 based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, and can control the orientation of the galvanometer mirror so that the irradiation position of the processing light L is maintained at the area to be irradiated with processing light L.

[0184] In step S135, the control device 1000 may control the light irradiation device 60 to change the spot size and intensity of the processing light L when irradiating the area to be irradiated with the processing light L (for example, the electrode of the chip LED). If the intensity of the processing light L is changed by changing the intensity of the light emitted from a light source (not shown), and the light source (not shown) is located outside the light irradiation device 60, then the external light source (not shown) should be controlled.

[0185] For example, the control device 1000 may control the orientation of the galvanometer mirror 61 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330, so that the irradiation position of the processing light L from the light irradiation device 60, which is displaced together with the displacement of at least one of the detection devices 320 and 330, is maintained at a first position, and then at a second position different from the first position.

[0186] For example, the control device 1000 may control the drive unit 311 to stop driving the drive unit 311. The control device 1000 may, based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330 after the drive unit 311 has stopped driving, control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L from the light irradiation device 60, which is displaced together with the displacement of at least one of the detection devices 320 and 330, is maintained at a first position, and then maintained at a second position different from the first position.

[0187] Furthermore, when the light irradiation device 60, etc., is being moved by the robot arm 310 and processing light L is being irradiated, the control device 1000 may control at least one of the posture and position of the light irradiation device 60, or the orientation of the galvanometer mirror 61, etc., based on prediction results that predict the movement of the robot arm 310, in addition to the results of the tracking process.

[0188] In the processing of steps S131 to S137 described above, the control device 1000 may, for example, control the drive unit 311 of the robot arm 310 so that the light irradiation device 60 and the detection devices 320 and 330 approach the circuit board T based on at least one of the image data and shape data generated by at least one of the detection devices 320 and 330, and when the light irradiation device 60 and the detection devices 320 and 330 approach the circuit board T to a predetermined distance, control the orientation of the galvanometer mirror 61 so that the processing light L from the light irradiation device 60, which is displaced together with at least one of the detection devices 320 and 330, is irradiated onto the area where the processing light L should be irradiated as part of the object (for example, a solder pad formed on the circuit board T, or elements and solder placed on the circuit board T) based on the at least one of the data that changes with the displacement of the detection devices 320 and 330 (for example, so that it is irradiated onto the same position on the area where the processing light L should be irradiated).

[0189] After the processing in step S137, the control device 1000 performs a quality inspection of the soldered elements and solder based on at least one of the image data and shape data output from at least one of the detection devices 320 and 330 (step S138). Examples of inspection items include misalignment of the elements relative to the solder pad and lifting of the element electrodes relative to the solder pad (the so-called Manhattan phenomenon where the element electrodes separate from the solder pad). For example, when performing a quality inspection of misalignment of elements relative to the solder pad, the control device 1000 recognizes the solder pad and elements in the image shown in the image data based on the image data output from at least one of the detection devices 320 and 330, and detects the misalignment of the elements relative to the solder pad. For example, the control device 1000 may determine that the element is good (good quality) if at least a part of the element electrodes overlaps the solder pad, and may determine that it is of poor quality if at least a part of the element electrodes do not overlap the solder pad. Furthermore, the control device 1000 may detect misalignment of the element relative to the solder pad not only based on image data, but also based on shape data output from at least one of the detection devices 320 and 330, or based on both image data and shape data. The control device 1000 does not necessarily need to determine whether at least a portion of the element's electrodes overlaps the solder pad as a quality defect. For example, the control device 1000 may determine quality based on the area where the element's electrodes overlap the solder pad.

[0190] The control device 1000 may also display at least one piece of information—image data, shape data, and the results of the soldering quality inspection—used in the soldering quality inspection process in step S138, on a display device (not shown).

[0191] The control device 1000 may perform machine learning using an existing method, with training data obtained by associating at least one piece of information from the image data and shape data used in the processes of steps S131 to S137 described above with the soldering quality determined in the process of step S138 described above. In this case, the control device 1000 may use the results of machine learning to control each device of the robot 3 (for example, position and attitude control of the light irradiation device 60 or control of the light irradiation device 60). Here, the control of the light irradiation device 60 includes setting conditions for the processing light L irradiated from the light irradiation device 60 (for example, at least one of the intensity of the processing light L, the spot size of the processing light L, the irradiation time of the processing light L, and the irradiation range of the processing light L). The control device 1000 may also use the results of machine learning to control at least one of the control of each device of the robot 1 and the control of each device of the robot 2.

[0192] After the processing in step S138, the control device 1000 starts moving the light irradiation device 60 (detection devices 320 and 330) to the second solder pad to be mounted, based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals in the processing of step S137 or step S136, and the position and orientation of the second solder pad to be mounted calculated in step S131, and repeats the processing in steps S133 to S138 described above. Note that when the control device 1000 starts moving the light irradiation device 60 (detection devices 320 and 330) to the second solder pad to be mounted, it may perform the processing in step S131 described above before executing the processing in steps S133 to S138 described above. The control device 1000 repeats the above steps until all elements (e.g., chip LEDs, etc.) placed on each solder pad of the circuit board T have been mounted (i.e., soldered).

[0193] If there is only one element placed on the solder pad of the circuit board T, the control device 1000 may, after processing in step S136, control the drive unit 311, etc., so that the robot arm 310, etc., is in a predetermined initial position.

[0194] As described above in each step (particularly the first and second aspects of step S137), the control device 1000 can recognize the position and orientation of the object at predetermined time intervals through tracking processing. As a result, the control device 1000 can irradiate the processing light L to the desired position of the object (in other words, the location where the processing light L should be irradiated) even if the relative position between the object and the light irradiation device 60 (light detection devices 320 and 330) changes (displaces) over time, by controlling at least one of the drive unit 311 of the robot arm 310 and the galvanometer mirror 61.

[0195] In addition, in the first and second embodiments of step S137, the control device 1000 may control at least one of the drive unit 311 of the robot arm 310 and the galvanometer mirror 61 so that the irradiation position of the processing light L changes over time over a wide area of ​​the area to be irradiated with the processing light L (for example, the entire area to be irradiated with the processing light L). For example, when the control device 1000 irradiates the electrode of an element as the area to be irradiated with the processing light L, it may control the orientation of the mirror of the galvanometer mirror 61 so that the irradiation position of the processing light L changes over time over the entire electrode while recognizing the position and orientation of the element (the electrode) at predetermined time intervals by tracking processing. In other words, the orientation of the mirror of the galvanometer mirror 61 may be controlled so that the processing light L is scanned over the electrode. By such control, localized heat input to the electrode (in other words, localized heat input to the solder) can be suppressed, and damage to the element due to heat, solder melting failure due to localized heating, and damage to the circuit board due to localized heating can be prevented. In this case, as in the first embodiment of step S137, the robot arm 310 may move the light irradiation device 60 and the detection devices 320 and 330, and the orientation of the galvano mirror may be changed so that the irradiation position of the processing light L changes over time across the entire electrode of one of the chip LEDs, based on the position and orientation of the chip LED (electrode of the chip LED) output from the 2D tracking unit 302 at predetermined time intervals. At this time, the control device 1000 can recognize the temporal change in the position and orientation of one electrode of the chip LED relative to the light irradiation device 60 based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, and can control the orientation of the galvano mirror so that the irradiation position of the processing light L changes over time across the entire electrode of one of the chip LEDs.

[0196] Furthermore, the position and orientation of the chip LED in the image shown by the image data sequentially output from at least one of the detection devices 320 and 330, and the position and orientation of the chip LED in the point cloud or depth image shown by the shape data, change over time due to the relative displacement between the chip LED and the light irradiation device 60 (detection devices 320 and 330). After the control device 1000 has finished irradiating one electrode of the chip LED with processing light L for a predetermined time, it then changes the orientation of the galvanometer mirror so that the irradiation position of the processing light L changes over time across the entire other electrode of the chip LED, based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals as the light irradiation device 60 and detection devices 320 and 330 continue to move by the robot arm 310. Then, after the control device 1000 has finished irradiating the other electrode of the chip LED with processing light L for a predetermined time, it repeats steps S132 to S137 described above to similarly irradiate the electrode of the chip LED placed on the second solder pad with processing light L.

[0197] Furthermore, for example, the control device 1000 changes the orientation of the galvano mirror over time based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals (due to displacement such as vibration of the light irradiation device 60), so that the irradiation position of the processing light L changes over time across the entire electrode of one of the chip LEDs. Since the control device 1000 can recognize changes in the position and orientation of one electrode of the chip LED relative to the light irradiation device 60, it can control the orientation of the galvano mirror so that the irradiation position of the processing light L changes over time across the entire electrode of one of the chip LEDs. Note that the position and orientation of the chip LED (electrode of the chip LED) in the image shown by the image data output sequentially from at least one of the detection devices 320 and 330, and the position and orientation of the chip LED in the point cloud shown by the shape data, change over time due to the relative displacement between the chip LED and the light irradiation device 60 (detection devices 320 and 330). After the control device 1000 has finished irradiating one electrode of the chip LED with processing light L for a predetermined time, the control device 1000 then changes the orientation of the galvanometer mirror over time, based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals (due to displacement caused by vibration of the light irradiation device 60, etc.), so that the irradiation position of the processing light L changes over time across the entire other electrode of the chip LED.

[0198] Therefore, the control device 1000 recognizes the temporal changes in the position and orientation of the area to be irradiated with processing light L on the light irradiation device 60, based on the position and orientation of the area to be irradiated with processing light L (for example, the electrodes of the element) output from the 2D tracking unit 302 at predetermined time intervals, and can irradiate the area to be irradiated with processing light L over a desired range (the same position or the entire area).

[0199] As mentioned above, the area to be irradiated with the processing light L as part of the object is not limited to the electrodes of the element, but may also be a solder pad or solder. Even in this case, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L changes over time across the entire area to be irradiated with the processing light L (solder pad or solder). Furthermore, if the area to be irradiated with the processing light L is large, the control device 1000 may drive the drive unit 311 of the robot arm 310 and control the orientation of the galvanometer mirror 61 to scan the area to be irradiated with the processing light L.

[0200] Furthermore, for example, the control device 1000 may alternately irradiate one electrode and the other electrode of an element having multiple electrodes (for example, a chip LED having two electrodes) as the area to be irradiated with processing light L in a time-dependent manner to melt the solder. In this case, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L changes alternately in time between one electrode and the other electrode, while recognizing the position and orientation of the element (one electrode and the other electrode) at predetermined time intervals by tracking processing. In addition, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L changes over time across the entire electrode during the time period when the processing light L is irradiated onto one electrode. Similarly, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L changes over time across the entire electrode during the time period when the processing light L is irradiated onto the other electrode. This type of control can further suppress localized heat input to the electrode (in other words, localized heat input to the solder).

[0201] As mentioned above, the area to be irradiated with the processing light L is not limited to electrodes; it may also be a solder pad or solder. Even in this case, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L alternates over time between the first area to be irradiated with the processing light L (one solder pad or one solder) and the second area (the other solder pad or the other solder). Furthermore, if the first area to be irradiated with the processing light L and the second area are far apart, the control device 1000 may scan the processing light L by controlling the orientation of the galvanometer mirror 61 while driving the drive unit 311 of the robot arm 310.

[0202] In step S134 described above, the control device 1000 may perform CAD matching using the shape data output from the detection device 330 and the CAD data related to the element to measure the position and orientation of the element. At this time, the control device 1000 may perform CAD matching after removing data corresponding to, for example, the substrate surface of the circuit board T from the shape data. By configuring it in this way, the time required for CAD matching (in other words, the time required for the processing in step S134) can be shortened. Note that various existing methods can be applied to remove shape data corresponding to, for example, the substrate surface, so a detailed explanation of these methods will be omitted.

[0203] Furthermore, if the solder placed on the solder pad can be melted and the element mounted, the control device 1000 does not need to perform the processes of steps S131 and S133, or steps S135 and S133, described above.

[0204] The control device 1000 may display at least one of the following information used in the processing of steps S131 to S137 above on a display device (not shown): image data, shape data, information on the position and orientation of the circuit board T, information on the position and orientation of each solder pad, information on the order in which components are mounted on each solder pad, and information on the position and orientation of the elements.

[0205] Furthermore, in the processing of steps S131 to S137 described above, the control device 1000 may detect at least one of the solder state (at least one piece of information regarding the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and orientation of the solder) and the area of ​​the solder pad based on at least one of the image data and shape data output from the detection devices 320 and 330. In other words, the control device 1000 may detect information regarding the state of the area to be irradiated with processing light L. The control device 1000 may then use the information regarding the state of the area to be irradiated with processing light L, as detected as described above, to control the conditions of the processing light L irradiated from the light irradiation device 60 (for example, at least one of the intensity of the processing light L, the spot size of the processing light L, the irradiation time of the processing light L, and the irradiation range of the processing light L). In other words, the control device 1000 may determine the conditions of the processing light L based on the detected information regarding the state of the area to be irradiated with processing light L. In addition to the information described above, the information regarding the state of the area to be irradiated with processing light L may also include information regarding elements, solder, and solder pads that can be detected as areas to be irradiated with processing light L, based on at least one of the image data and shape data output from at least one of the detection devices 320 and 330.

[0206] As shown in Figure 16, an air blower and a fume extractor 70 may be placed near the light irradiation device 60, which serves as the end effector of the robot arm 310 (not shown). Alternatively, either the air blower or the fume extractor may be placed, while the other is omitted.

[0207] With this configuration, it is possible to effectively prevent smoke generated when the solder is heated by the processing light L emitted from the light irradiation device 60 from adhering to the optical system of the light irradiation device 60.

[0208] Robot 1 The control device 1000 that controls the robot 1 performs the processes of steps S111 and S112, which correspond to the processes of steps S131 and S132 described above. However, the control device 1000 performs the processes of steps S111 and S112 using the output of at least one of the detection devices 120 and 130 provided by the robot 1.

[0209] The control device 1000 may perform calibration of the dispenser 40 before performing the following steps S111 to S117.

[0210] As a prerequisite, it is assumed that the detection device 130 and the dispenser 40 are mounted on the robot arm 110 in such a positional relationship that a part of the dispenser 40 (for example, its tip) is within the field of view of the cameras 31 and 32 of the detection device 130, which have a configuration similar to that of the detection device 330. For the sake of explanation, the cameras 31 and 32 of the detection device 130 will be referred to as the cameras 31 and 32 of the detection device 130, as an example of a detection device 130 having cameras 31 and 32 similar to those of the detection device 330.

[0211] As a calibration of the dispenser 40, the control device 1000 performs the matching process described above using the shape data including the dispenser 40 output from the detection device 230 and the CAD data of the dispenser 40, and pre-calculates the position and orientation of the dispenser 40 (for example, the position and orientation of the tip of the dispenser 40 included in the field of view of the cameras 31 and 32 of the detection device 130). In other words, the control device 1000 pre-calculates the position and orientation of the dispenser 40 in the coordinate system of the robot arm 110 based on the shape data of at least a part of the dispenser 40.

[0212] Furthermore, markers may be provided on a portion of the dispenser 40 that is included within the field of view of each of the cameras 31 and 32 of the detection device 130. In this case, the control device 1000 may perform the above-described calibration based, for example, on shape data including the markers output from the detection device 130.

[0213] Furthermore, the control device 1000 may perform calibration of the dispenser 40 by executing a matching process using not only shape data, but also image data output from the detection device 130 and CAD data of the dispenser 40. In addition, as described above, the control device 1000 may use shape data or image data of the dispenser 40 that have been acquired in advance, not limited to CAD data, in the matching process.

[0214] Furthermore, the control device 1000 may use shape data or image data output from the detection device 120, rather than being limited to using the detection device 130 for the calibration of the dispenser 40. In this case, it is assumed that the detection device 120 and the dispenser 40 are mounted on the robot arm 110 in such a positional relationship that a part of the dispenser 40 is within the field of view of the cameras 21 and 22 of the detection device 120, which has the same configuration as the detection device 320.

[0215] In the following steps S111 to S117, the dispenser 40 may come into contact with a predetermined object, causing a change in the position and orientation of the dispenser 40 relative to the detection device 130. In this case, the control device 1000 can detect a change in the position and orientation of the dispenser 40 relative to the detection device 130 based on a partial change in the dispenser 40 in the image data or shape data output from the detection device 130 (for example, a partial change in the dispenser 40 in the image). If a change in the position and orientation of the dispenser 40 relative to the detection device 130 is detected, the control device 1000 may perform calibration.

[0216] The control device 1000 that controls the robot 1 calculates the position and orientation of the circuit board T as an example of an object (step S111). Similar to step S131 described above, in the process of step S111, the control device 1000 calculates the initial position and orientation of the circuit board T (i.e., the initial position and orientation) by the matching process of the matching unit 301. Furthermore, the control device 1000 calculates the position of each solder pad formed on the circuit board T. For example, the control device 1000 calculates the position and orientation of each solder pad on the circuit board T based on the Gerber data of the circuit board T (i.e., the design data of the circuit board T). Based on the Gerber data, the control device 1000 identifies the order in which to mount (in this case, place solder) each solder pad.

[0217] Next, the control device 1000 controls the drive unit 111 to move the robot arm 110 so that the dispenser 40 (detection devices 120 and 130) approaches the circuit board T (step 112). In the process of step S112, similar to step S132 described above, the control device 1000 controls the drive unit 111 of the robot arm 110 so that the first solder pad to be mounted comes into the field of view of at least one of the detection devices 120 and 130.

[0218] Next, the control device 1000 determines whether the solder pad to be first mounted is within the field of view of at least one of the detection devices 120 and 130 (step S113). Similar to step S133 described above, in the process of S113, the control device 1000 determines whether the detection devices 120 and 130 are in the desired position and orientation relative to the solder pad to be first mounted, based on the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals and the position and orientation information of the solder pad to be first mounted calculated in step S111. At this time, if the detection devices 120 and 130 are in the desired position and orientation relative to the solder pad, the control device 1000 determines that the solder pad is within the field of view of at least one of the detection devices 120 and 130.

[0219] In step S113, if it is determined that the solder pad is not within the field of view of at least one of the detection devices 120 and 130 (S113: No), the control device 1000 controls the drive unit 111 to continue moving the robot arm 110 based on the position and orientation information of the solder pad calculated in step S111 and the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals.

[0220] On the other hand, in the process of step S113, if it is determined that the solder pad has entered the field of view of at least one of the detection device 120 and the detection device 130 (step S113: Yes), the control device 1000 calculates the position and orientation of the first solder pad to be mounted (step S114). Similar to step S134 described above, in the process of step S114, the control device 1000 calculates the initial position and orientation (initial position and orientation) of the solder pad by the matching process of the matching unit 301. Note that in the process of step S111 described above, the position and orientation of the solder pad are calculated based on Gerber data, but since Gerber data is design data, there is an error between it and the actual position and orientation of the solder pad on the circuit board T. Therefore, the control device 1000 executes the process of step S114.

[0221] Next, the control device 1000 controls the drive unit 111 so that the position and orientation of the dispenser 40 are in a desired position and orientation that allows solder to be dispensed onto the first solder pad to be mounted, and moves the robot arm 110 (step S115). Similar to step S135 described above, in the process of step S115, the control device 1000 uses the initial position and orientation information of the solder pad calculated in the process of step S114 to control the drive unit 111 based on the position and orientation of the solder pad output from the 2D tracking unit 302 at predetermined time intervals, and moves the robot arm 110. In other words, the control device 1000 controls the drive unit 111 so that the dispenser 40 (detection devices 120 and 130) approaches the first solder pad to be mounted on the circuit board T, and moves the robot arm 110.

[0222] Next, the control device 1000 determines whether the position and orientation of the dispenser 40 are in a desired position and orientation that allows solder to be dispensed onto the first solder pad to be mounted (step S116). In the process of step S116, similar to step S136 described above, the control device 1000 determines whether the position and orientation of the dispenser 40 relative to the solder pad is in a desired position and orientation, for example, based on the position and orientation information of the first solder pad to be mounted output from the 2D tracking unit 302 at predetermined time intervals.

[0223] In step S116, if it is determined that the position and orientation of the dispenser 40 are not the desired position and orientation for dispensing solder to the solder pad (step S116: No), the control device 1000 controls the drive unit 111 to continue moving the robot arm 110 so that the dispenser 40 approaches the solder pad, based on the position and orientation of the solder pad output from the 2D tracking unit 302 at predetermined time intervals. On the other hand, if it is determined that the position and orientation of the dispenser 40 are the desired position and orientation for dispensing solder to the solder pad (step S116: Yes), the control device 1000 controls the dispenser 40 so that solder is placed on at least a portion of the solder pad (step S117). Specifically, for example, the control device 1000 controls the dispenser 40 to dispense solder. Here, the control device 1000 may estimate the area of ​​the solder pad based on the position and orientation of the solder pad output from the 2D tracking unit 302 at predetermined time intervals, and control the amount of solder dispensed from the dispenser 40 according to the area of ​​the solder pad. In this case, the control device 1000 may control the dispenser 40 so that the larger the estimated area of ​​the solder pad, the greater the amount of solder dispensed. This makes it possible to place an appropriate amount of solder on the solder pad.

[0224] Furthermore, at this time, the relative position between the dispenser 40 and the workpiece may change due to, for example, vibration of the robot arm 110. Therefore, the position of the solder pad as the workpiece in the image shown by, for example, image data output sequentially from the detection device 130 may also change over time due to the aforementioned change in relative position.

[0225] Therefore, in order to suppress or eliminate the influence of the relative position fluctuations on the processing of step S117, the control device 1000 controls at least one of the attitude and position of the dispenser 40 based on the results of the tracking process described above.

[0226] For example, the control device 1000 may control the drive unit 111 so that the solder dispensed from the dispenser 40, which is displaced along with the displacement of at least one of the detection devices 120 and 130, is placed on the solder pad, which is the target object, based on at least one of the image data and shape data, which changes with the displacement of at least one of the detection devices 120 and 130.

[0227] For example, the control device 1000 may control the drive unit 111 to stop the drive of the drive unit 111. The control device 1000 may also control the drive unit 111 based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 120 and 130 after the drive of the drive unit 111 has stopped, so that the solder dispensed from the dispenser 40 which displaces together with at least one of the detection devices 120 and 130 is placed on the solder pad which is the target object.

[0228] In the process of steps S111 to S117 described above, the control device 1000 may control the drive unit 111 of the robot arm 110 so that the dispenser 40 and the detection devices 120 and 130 approach the circuit board T based on at least one of the image data and shape data generated by at least one of the detection devices 120 and 130. When the dispenser 40 and the detection devices 120 and 130 approach the circuit board T to a predetermined distance, the control device 1000 may control the drive unit 111 so that the solder dispensed from the dispenser 40, which is displaced together with at least one of the detection devices 120 and 130, is placed on the solder pad, which is the target object, based on the at least one of the data that changes with the displacement of at least one of the detection devices 120 and 130.

[0229] After the processing in step S117, the control device 1000 starts moving the dispenser 40 to the second solder pad to be mounted, based on the position and orientation of the first solder pad to be mounted output from the 2D tracking unit 302 at predetermined time intervals during the processing of step S117 or step S116, and the position and orientation of the second solder pad to be mounted calculated in step S111, and repeats the processing in steps S113 to S117 described above. Note that when the control device 1000 starts moving the dispenser 40 to the second solder pad to be mounted, it may perform the processing in step S111 described above before executing the processing in steps S113 to S117 described above. The control device 1000 repeats the above steps until the placement of solder on each solder pad of the circuit board T is completed.

[0230] If there is only one solder pad on the circuit board T, the control device 1000 may, after processing in step S117, control the drive unit 111, etc., so that the robot arm 110, etc., is in a predetermined initial position.

[0231] The control device 1000 may also display at least one of the following information on a display device (not shown): image data, shape data, position and orientation information of the circuit board T, position and orientation information of each solder pad, and information on the order in which components are mounted on each solder pad.

[0232] Furthermore, if solder can be placed on the solder pad, the control device 1000 does not need to perform the processes described in steps S111 and S113 or steps S115 and S116 above.

[0233] Furthermore, in the processing of steps S115 to S117 described above, the control device 1000 may detect (calculate) the area of ​​the solder pad based on at least one of the image data or shape data output from the detection devices 120 and 130.

[0234] Furthermore, after the processing in step S117, the control device 1000 may detect the state of the solder placed on the solder pad based on at least one of the image data and shape data output from at least one of the detection devices 120 and 130, or based on the matching process. Examples of the solder state to be detected include the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, the position and orientation of the solder, etc. The control device 1000 may then determine the quality of the solder based on the detected solder state.

[0235] For example, when detecting the distance between a solder pad and solder, the control device 1000 recognizes the solder pad and solder in the image shown by the image data based on the image data output from at least one of the detection devices 120 and 130, and detects the distance between the solder pad and solder. For example, the control device 1000 may determine whether the solder placement position is good or bad based on the detected distance between the solder pad and solder. For example, the control device 1000 may determine that the solder placement position is bad if the detected distance between the solder pad and solder is greater than or equal to a predetermined threshold (for example, a state where no solder is placed on the solder pad). Also, for example, when detecting the shape of solder, the control device 1000 recognizes the solder in the point cloud shown by the shape data based on the shape data output from at least one of the detection devices 120 and 130, and detects the shape of the solder. For example, the control device 1000 may determine whether the shape of the solder is good or bad based on the detected shape of the solder. For example, the control device 1000 may determine that the shape of the solder is defective if the difference between the detected solder shape and the desired shape is greater than or equal to a threshold. Also, for example, when detecting the volume of solder, the control device 1000 may estimate the volume of solder using an existing method based on the solder shape detected by the method described above. For example, the control device 1000 may determine whether the solder volume is good or bad based on the estimated volume of solder. For example, the control device 1000 may determine that the solder volume is defective if the detected solder volume is outside the threshold (for example, if the solder volume is too large or too small). Also, for example, when detecting the position and orientation of solder, the control device 1000 recognizes the solder in the point cloud indicated by the shape data output from at least one of the detection devices 120 and 130, and detects the position and orientation of the solder. For example, the control device 1000 may determine whether the solder shape is good or bad based on the detected position and orientation of the solder. For example, the control device 1000 may determine that the solder position and orientation are defective if at least one of the detected solder position and orientation is outside a threshold.

[0236] The control device 1000 may perform machine learning using an existing method, with training data obtained by associating at least one piece of information from the image data and shape data used in the above-described steps S111 to S117 with the solder quality detected as described above. In this case, the control device 1000 may use the results of machine learning to control each device of robot 1 (for example, position and attitude control of dispenser 40 or dispensing control of dispenser 40). The control device 1000 may also use the results of machine learning to control at least one of the devices of robot 2 and each device of robot 3.

[0237] Furthermore, the control device 1000 may use the information detected as described above regarding the area of ​​the solder pad and the state of the solder (at least one piece of information regarding the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and orientation of the solder) to control the position and orientation of the holding device 50 on the robot arm 210 of the robot 2. In this case, the elements held by the holding device 50 can be efficiently placed on the solder.

[0238] Furthermore, the control device 1000 may use the information detected as described above regarding the area of ​​the solder pad and the state of the solder (at least one piece of information regarding the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and orientation of the solder) to control at least one of the position and orientation control of the light irradiation device 60 by the robot arm 310 of the robot 3, and the control of the galvanometer mirror 61. In this case, the light irradiation device 60 can efficiently irradiate the processing light L to the areas on the circuit board T where the processing light L should be irradiated (for example, placed elements, placed solder, solder pads, etc.).

[0239] Furthermore, the control device 1000 may use at least one of the information detected as described above—the area of ​​the solder pad and the information regarding the state of the solder (at least one piece of information regarding the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and orientation of the solder)—to control the conditions of the processing light L irradiated from the light irradiation device 60 by the robot arm 310 of the robot 3 (for example, at least one of the intensity of the processing light L, the spot size of the processing light L, the irradiation time of the processing light L, and the irradiation range of the processing light L). The irradiation range of the processing light L includes, for example, at least a part of the area to be irradiated with the processing light L, such as an element, solder, or solder pad. Here, at least one piece of information—the area of ​​the solder pad and the information regarding the state of the solder (at least one piece of information regarding the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and orientation of the solder)—can be rephrased as information regarding the state of the area to be irradiated with the processing light L. For example, the control device 1000 may determine the conditions of the processing light L based on the detected information regarding the state of the area to be irradiated with the processing light L. For example, the control device 1000 may determine the spot size of the processing light L based on the area of ​​the solder pad. In addition to the above-mentioned information, the information regarding the state of the area to be irradiated with the processing light L may also include information regarding elements, solder, and solder pads that can be detected as areas to be irradiated with the processing light L based on at least one of the image data and shape data output from at least one of the detection devices 120 and 130, and matching processing.

[0240] Furthermore, the control device 1000 may use at least one of the following pieces of information calculated in the above steps S111 to S117—the position and orientation information of the circuit board T, the position and orientation information of each solder pad, and the information of the mounting order for each solder pad—to control at least one of the position and orientation control of the holding device 50 by the robot arm 210 of the robot 2 and the holding force control of the holding device 50. In this case, the elements held by the holding device 50 can be efficiently placed on the solder. For example, if the above information is used for position and orientation control of the holding device 50 by the robot arm 210 of the robot 2, the processing in step S121 described later by the control device 1000 can be omitted, and the elements gripped by the holding device 50 can be efficiently placed on the solder pads (solder).

[0241] Furthermore, the control device 1000 may use at least one of the following pieces of information calculated in the processes of steps S111 to S117 above—namely, the position and orientation information of the circuit board T, the position and orientation information of each solder pad, and the information of the mounting order for each solder pad—to control at least one of the following: the control of the position and orientation of the light irradiation device 60 by the robot arm 310 of the robot 3, and the control of the galvanometer mirror 61. In this case, the light irradiation device 60 can efficiently irradiate the processing light L to the areas on the circuit board T where the processing light L should be irradiated. For example, if the above information is used for the position and orientation control of the light irradiation device 60 by the robot arm 310 of the robot 3, the process of step S131 described later by the control device 1000 can be omitted, and the solder can be efficiently melted by the processing light L using the light irradiation device 60.

[0242] The control device 1000 may also display at least one of the following pieces of information—image data, shape data, and solder state detection results—used in the solder state detection process described above, on a display device (not shown).

[0243] For example, the control device 1000 may control the drive unit 111 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 120 and 130, so that the solder dispensed from the dispenser 40, which is displaced together with the displacement of at least one of the detection devices 120 and 130, is initially placed at a first position on the circuit board T, and then at a second position different from the first position.

[0244] For example, the control device 1000 may control the drive unit 111 to stop driving the drive unit 111. The control device 1000 may control the drive unit 111 based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 120 and 130 after the drive unit 111 has stopped driving, so that the solder dispensed from the dispenser 40, which is displaced together with the displacement of at least one of the detection devices 120 and 130, is initially placed at a first position on the circuit board T, and then at a second position different from the first position.

[0245] Robot 2 The control device 1000 that controls the robot 2 performs the processes of steps S122 and S123, which correspond to the processes of steps S131 and S132 described above. However, the control device 1000 performs the processes of steps S121 and S123 using the output of at least one of the detection devices 220 and 230 provided by the robot 2.

[0246] For example, the holding device 50 is equipped with a tweezers hand that can open and close the tips of the tweezers. The holding device 50 may also be equipped with a suction device that can attract and hold the element.

[0247] The control device 1000 may perform calibration of the holding device 50 before performing the following steps S121 to S129.

[0248] As a prerequisite, it is assumed that the detection device 230 and the holding device 50 are mounted on the robot arm 210 in such a positional relationship that the tip of the holding device 50 (i.e., in the case of a tweezers hand, the tip of the tweezers that contacts the element when holding the element) is within the field of view of the cameras 31 and 32 of the detection device 230, which has a configuration similar to that of the detection device 330. For the sake of explanation, the cameras 31 and 32 of the detection device 230 will be referred to as the cameras 31 and 32 of the detection device 230, as an example in which the detection device 230 has cameras 31 and 32 similar to those of the detection device 330.

[0249] As a calibration of the holding device 50, the control device 1000 performs the matching process described above using the shape data output from the detection device 230 when the holding device 50 is not gripping an element and the CAD data of the holding device 50, and pre-calculates the position and orientation of the holding device 50 (for example, the position and orientation of the tip of the tweezers hand included in the field of view of the cameras 31 and 32 of the detection device 230). In other words, the control device 1000 pre-calculates the position and orientation of the holding device 50 in the coordinate system of the robot arm 210 based on the shape data of at least a part of the holding device 50 (for example, the tip of the tweezers hand).

[0250] Furthermore, markers may be provided on a portion of the holding device 50 that is included within the field of view of each of the cameras 31 and 32 of the detection device 230. In this case, the control device 1000 may perform the above-described calibration based, for example, on shape data including the markers output from the detection device 230.

[0251] Furthermore, the control device 1000 may perform calibration of the holding device 50 by executing a matching process using not only shape data, but also image data output from the detection device 230 and CAD data of the holding device 50. In addition, as described above, the control device 1000 may use not only CAD data, but also shape data or image data of the holding device 50 that have been acquired in advance, in the matching process.

[0252] Furthermore, the control device 1000 may use shape data or image data output from the detection device 220, rather than being limited to using the detection device 230 for calibration of the holding device 50. In this case, it is assumed that the detection device 220 and the holding device 50 are mounted on the robot arm 210 in such a positional relationship that the tip of the holding device 50 is within the field of view of the cameras 21 and 22 of the detection device 220, which has a configuration similar to that of the detection device 320.

[0253] In the following steps S121 to S129, the holding device 50 may come into contact with a predetermined object, causing a change in the position and orientation of the holding device 50 relative to the detection device 230. In this case, the control device 1000 can detect a change in the position and orientation of the holding device 50 relative to the detection device 230 based on a partial change in the holding device 50 in the image data or shape data output from the detection device 230 (for example, a partial change in the holding device 50 in the image). If a change in the position and orientation of the holding device 50 relative to the detection device 230 is detected, the control device 1000 may perform calibration.

[0254] The control device 1000 that controls the robot 2 holds the element (step S121). The control device 1000 controls the drive unit 211 of the robot arm 210 and the holding device 50 so that the robot 2's holding device 50 is brought closer to an element supply device (so-called parts feeder) not shown, and the robot arm 210 holds the desired element with the holding device 50. For example, the control device 1000 performs at least one of the matching process and tracking process described above, calculates the position and orientation of the desired element placed on the element supply device not shown, and then brings the holding device 50 closer to the desired element placed on the element supply device not shown, so that the holding device 50 can hold the element. Here, the control device 1000 may determine the holding force (gripping force) of the element in the holding device 50 according to the size of the element calculated in at least one of the matching process and tracking process. This prevents damage to the element caused by the holding of the element by the holding device 50 and prevents the element from falling out of the holding device 50.

[0255] After the processing in step S121, the control device 1000 calculates the position and orientation of the circuit board T as an example of an object (step S122). Similar to steps S111 and S131 described above, in the processing of step S122, the control device 1000 calculates the initial position and orientation of the circuit board T (i.e., the initial position and orientation) by the matching process of the matching unit 301. Furthermore, the control device 1000 calculates the position of each solder pad formed on the circuit board T. The control device 1000 calculates the position of each solder pad on the circuit board T based on the Gerber data of the circuit board T (i.e., the design data of the circuit board T). Based on the Gerber data, the control device 1000 identifies the order in which components are mounted (in this case, placed) on each solder pad.

[0256] Next, the control device 1000 controls the drive unit 211 to move the robot arm 210 so that the holding device 50 (detection devices 220 and 230) approaches the circuit board T (step 123). In the process of step S123, similar to steps S112 and S132 described above, the control device 1000 controls the drive unit 211 of the robot arm 210 so that the first solder pad to be mounted comes into the field of view of at least one of the detection devices 220 and 230.

[0257] Next, the control device 1000 determines whether the marker provided near the first solder pad to be mounted is within the field of view of at least one of the detection devices 220 and 230 (step S124). Similar to steps S113 and S133 described above, in step S124, the control device 1000 determines whether the detection devices 220 and 230 are in the desired position and orientation relative to the first solder pad to be mounted, based on the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals and the position and orientation information of the first solder pad to be mounted calculated in step S121. At this time, if the detection devices 220 and 230 are in the desired position and orientation relative to the first solder pad to be mounted, the control device 1000 determines that the marker provided near the solder pad is within the field of view of at least one of the detection devices 220 and 230.

[0258] In step S124, if it is determined that the marker placed near the first solder pad to be implemented is not within the field of view of at least one of the detection devices 220 and 230 (S124: No), the control device 1000 controls the drive unit 211 to continue moving the robot arm 210 based on the position and orientation information of the first solder pad to be implemented calculated in step S122 and the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals.

[0259] On the other hand, in the process of step S124, when it is determined that the marker provided near the first soldering pad has entered the field of view of at least one of the detection devices 220 and 230 (step S124: Yes), the control device 1000 calculates the position and orientation of the marker provided near the first soldering pad (step S125). In the process of step S125, similar to the above steps S114 and S134, the control device 1000 calculates the initial position and orientation (initial position and orientation) of the marker provided near the first soldering pad by the matching process of the matching unit 301.

[0260] Next, the control device 1000 controls the drive unit 211 so that the position and orientation of the holding device 50 become the desired position and orientation where the solder element can be placed on the first soldering pad to be mounted, and moves the robot arm 210 (step S126). In the process of step S126, similar to the above steps S115 and S135, the control device 1000 uses the information on the initial position and orientation of the marker provided near the soldering pad calculated in the process of step S125, and based on the information on the position and orientation of the marker arranged near the soldering pad output from the 2D tracking unit 302 at a predetermined time interval and the positional relationship between the soldering pad and the marker, controls the drive unit 211 to move the robot arm 210. Note that the positional relationship between the soldering pad and the marker is known.

[0261] In other words, the control device 1000 controls the drive unit 211 so that the holding device 50 (detection devices 220 and 230) approaches the first soldering pad (solder arranged on the soldering pad) of the circuit board T, and moves the robot arm 210. Note that the control device 1000 may control the drive unit 211 to move the robot arm 210 based on the information on the distance between the soldering pad and the solder detected after the solder is arranged on the soldering pad by the dispenser 40 of the robot 1. In this case, the holding device 50 can be brought closer to the soldering pad (solder arranged on the soldering pad) more accurately (more efficiently).

[0262] Next, the control device 1000 determines whether the position and orientation of the holding device 50 are the desired position and orientation where the element can be placed on the solder on the solder pad to be mounted first (step S127). In the process of step S127, similar to the above-described steps S116 and S136, the control device 1000, for example, based on the position and orientation information of the marker arranged near the solder pad to be mounted first, which is output from the 2D tracking unit 302 at predetermined time intervals, and the positional relationship between the solder pad and the marker, determines whether the position and orientation of the holding device 50 with respect to the solder pad (the solder arranged on the solder pad) are the desired position and orientation.

[0263] In the process of step S127, if it is determined that the position and orientation of the holding device 50 are not the desired position and orientation where the element can be placed on the solder on the solder pad to be mounted first (step S127: No), the control device 1000 continues to control the drive unit 211 to move the robot arm 210 based on the position and orientation of the solder pad (the solder) output from the 2D tracking unit 302 at predetermined time intervals so that the holding device 50 approaches the solder pad (the solder).

[0264] On the other hand, if it is determined that the position and orientation of the holding device 50 are the desired position and orientation where the element can be placed on the solder pad (the solder) (step S127: Yes), the control device 1000 calculates the position and orientation of the element held by the holding device 50 (step S128).

[0265] First, as a premise, as described above, the tip of the holding device 50 (that is, in the case of the gripper hand, the tip portion of the gripper that contacts the element when holding the element) is included in the visual fields of the cameras 31 and 32 of the detection device 230. Further, the detection device 230 is provided at a desired position of the robot arm 210 so that at least a part of the element held by the holding device 50 is also included in the visual fields of the cameras 31 and 32.

[0266] The control device 1000 performs the above-described CAD matching process using, for example, the shape data output from the detection device 230 (i.e., data including the shape data of the tip of the holding device 50 and at least a part of the element) and the CAD data of the element while the holding device 50 is holding the element, and calculates the position and orientation of the element. The position and orientation of an element held by the holding device 50 changes each time it is held by the holding device 50, even if the elements are of the same type (i.e., have the same shape). Therefore, by executing the process in step S125, the control device 1000 can recognize the position and orientation of the element, and in step S129 described later, the element can be placed with high precision on the first solder pad (solder) to be mounted.

[0267] Furthermore, the control device 1000 may, for example, recognize the shape data of the tip of the holding device 50 from the shape data output from the detection device 230 while the holding device 50 is holding the element, perform a process to remove the shape data of the tip of the holding device 50 from the shape data output from the detection device 230, and then perform a CAD matching process using the CAD data of the element to calculate the position and orientation of the element. In this case, the shape data of the tip of the holding device 50 can act as noise in the process of calculating the position and orientation of the element, which can suppress a decrease in the accuracy of calculating the position and orientation of the element.

[0268] Furthermore, the control device 1000 may, for example, remove the shape data of the tip of the holding device 50, which was calculated in the matching process during the calibration of the holding device 50 described above, from the shape data output from the detection device 230 while the holding device 50 is holding the element, and then perform a CAD matching process using the CAD data of the element to calculate the position and orientation of the element. In this case as well, a decrease in the accuracy of calculating the position and orientation of the element can be suppressed.

[0269] In addition, for the purpose of calibration of the holding device 50 described above, the detection device 230 is mounted on the robot arm 210 so that the tip of the holding device 50 is included in the field of view of each of the cameras 31 and 32 of the detection device 230. However, if calibration of the holding device 50 is not performed, the detection device 230 may be mounted on the robot arm 210 so that the holding device 50 is not included in the field of view of each of the cameras 31 and 32 of the detection device 230, but at least a portion of the elements held by the holding device 50 is included.

[0270] After the processing in step S128, the control device 1000 controls the holding device 50 so that the element is placed on the first solder pad (solder) to be mounted (step S129). Based on the position and orientation of the element held by the holding device 50 calculated in the processing in step S128 above, and the position and orientation of the solder pad (solder) output from the 2D tracking unit 302 at predetermined time intervals, the control device 1000 controls the drive unit 211 of the robot arm 210 so that the position and orientation of the element held by the holding device 50 becomes a desired position and orientation that allows the element to be placed on the solder pad (solder). Then, the control device 1000 controls the holding device 50 so that the element is released from being held, and the element is placed on the solder pad (solder).

[0271] At this time, similar to the process in step S117 described above, the relative position between the holding device 50 and the solder pad (solder) as the object may fluctuate due to, for example, vibration of the robot arm 210. Therefore, in order to suppress or eliminate the influence of the fluctuation in the relative position on the process in step S129, the control device 1000 controls at least one of the attitude and position of the holding device 50 based on the results of the tracking process described above.

[0272] For example, the control device 1000 may control the drive unit 211 so that the element gripped by the holding device 50, which displaces along with the displacement of at least one of the detection devices 220 and 230, is positioned on the solder pad (solder) based on at least one of the image data and shape data, which changes with the displacement of at least one of the detection devices 220 and 230.

[0273] For example, the control device 1000 may control the drive unit 211 to stop its operation. The control device 1000 may also control the drive unit 211 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230 after the drive unit 211 has stopped, so that the element gripped by the holding device 50, which displaces together with at least one of the detection devices 220 and 230, is positioned on the solder pad (solder).

[0274] In the processing of steps S121 to S129 described above, the control device 1000 may control the drive unit 211 of the robot arm 210 so that the holding device 50 and the detection devices 220 and 230 approach the circuit board T based on at least one of the image data and shape data generated by at least one of the detection devices 220 and 230. When the holding device 50 and the detection devices 220 and 230 approach the circuit board T to a predetermined distance, the control device 1000 may control the drive unit 211 so that the element gripped by the holding device 50, which displaces together with at least one of the detection devices 220 and 230, is positioned on the solder pad (solder) as the target object, based on the at least one of the data that changes with the displacement of at least one of the detection devices 220 and 230.

[0275] After the processing in step S129, the control device 1000 drives the drive unit 211 of the robot arm 210 so that the holding device 50 approaches an element supply device (not shown), and picks up the element to be placed on the second solder pad (solder) to be mounted. The control device 1000 then repeats the processing in steps S122 to S129 described above. The control device 1000 repeats the element picking and the steps described above until the placement of elements on the solder of each solder pad on the circuit board T is complete.

[0276] If there is only one solder pad on the circuit board T (i.e., only one element to be placed on the solder), the control device 1000 may, after processing in step S129, control the drive unit 211, etc., so that the robot arm 210, etc., is in a predetermined initial position.

[0277] The control device 1000 may perform the process in step S128 described above before the process in step S127. Alternatively, the control device 1000 may not perform the process in step S128 described above. In this case, the control device 1000 only needs to control the holding device 50 so that the element is placed on the first solder pad (solder) to be mounted after the process in step S127 (more precisely, step S127: Yes) described above.

[0278] The control device 1000 may also display at least one of the following information on a display device (not shown): image data, shape data, position and orientation information of the circuit board T, position and orientation information of the markers, position and orientation information of each solder pad, and information on the order in which components are mounted on each solder pad.

[0279] Furthermore, if the element can be placed on the solder on the solder pad, the control device 1000 does not need to perform the processes of steps S122 and S124 or steps S126 and S127 described above.

[0280] Furthermore, after the processing in step S129, the control device 1000 may detect the state of the elements placed on the solder pad based on at least one of the image data and shape data output from at least one of the detection devices 220 and 230, or based on the matching process. The elements to be detected include the position and orientation of the elements, the distance between the elements and the solder, and the distance between the elements and the solder pad. The control device 1000 may then determine the quality of the placed elements based on the detected state of the elements.

[0281] For example, when detecting the position and orientation of an element, the control device 1000 calculates the position and orientation of the element based on the matching process described above. For example, the control device 1000 may determine whether the placement of the element is good or bad based on the detected position and orientation of the element. For example, if at least one of the detected position and orientation of the element falls outside a predetermined threshold, the control device 1000 may determine that the solder placement is poor.

[0282] For example, when detecting the distance between an element and solder, the control device 1000 recognizes the element and solder in the point cloud indicated by the shape data output from at least one of the detection devices 220 and 230, and calculates the distance between the element and the solder. For example, the control device 1000 may also determine the quality of the element's placement based on the detected distance between the element and the solder. For example, if the detected distance between the element and the solder is greater than or equal to a predetermined threshold (for example, the element is not placed on the solder), the control device 1000 may determine that the solder placement is poor.

[0283] For example, when detecting the distance between an element and a solder pad, the control device 1000 recognizes the element and the solder pad in the image shown by the image data based on the image data output from at least one of the detection devices 220 and 230, and calculates the distance between the element and the solder pad. For example, the control device 1000 may determine the quality of the placement position of the element based on the detected distance between the element and the solder pad. For example, when the detected distance between the element and the solder pad is greater than or equal to a predetermined threshold value (for example, when the element is not placed on the solder pad), the control device 1000 may determine that the placement position of the element is defective.

[0284] In addition, the control device 1000 may perform machine learning by an existing method using, as teacher data, data in which at least one piece of information of the image data and the shape data used in the above-described processes of steps S122 to S128 is associated with the quality of the element determined as described above. In this case, the control device 1000 may use the result of the machine learning for the control of each device of the robot 2 (for example, the position and posture control of the holding device 50 and the holding control of the holding device 50). In addition, the control device 1000 may use the result of the machine learning for the control of at least one of the control of each device of the robot 1 and the control of each device of the robot 3.

[0285] Further, the control device 1000 may use at least one piece of information of the position and posture of the element, the distance between the element and the solder, and the distance between the element and the solder pad detected as described above for at least one of the position and posture control of the light irradiation device 60 by the robot arm 310 of the robot 3, the control of the galvanometer mirror 61 of the light irradiation device 60, and the change of the intensity and spot size of the processing light L. In this case, the processing light L can be efficiently irradiated to the location (for example, the arranged element, the arranged solder, the solder pad, etc.) on the circuit board T by the light irradiation device 60. In addition, the control device 1000 may use at least one piece of information of the position and posture of the element, the distance between the element and the solder, and the distance between the element and the solder pad detected as described above for the position and posture control of the dispenser 40 by the robot arm 110 of the robot 1.

[0286] Furthermore, in the processing of steps S122 to S129 described above, the control device 1000 may detect at least one of the solder state (at least one piece of information regarding the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and orientation of the solder) and the area of ​​the solder pad based on at least one of the image data and shape data output from the detection devices 120 and 130. In other words, the control device 1000 may detect information regarding the state of the area to be irradiated with processing light L. The control device 1000 may then use the information regarding the state of the area to be irradiated with processing light L, as detected as described above, to control the conditions of the processing light L irradiated from the light irradiation device 60 by the robot arm 310 of the robot 3 (for example, at least one of the intensity of the processing light L, the spot size of the processing light L, the irradiation time of the processing light L, and the irradiation range of the processing light L). In other words, the control device 1000 may determine the conditions of the processing light L based on the detected information regarding the state of the area to be irradiated with processing light L. In addition to the information described above, the information regarding the state of the area to be irradiated with processing light L may also include information regarding elements, solder, and solder pads that can be detected as areas to be irradiated with processing light L, based on at least one of the image data and shape data output from at least one of the detection devices 220 and 230, and a matching process.

[0287] Furthermore, the control device 1000 may use at least one of the information obtained in the processing steps S122 to S129 above—the position and orientation information of the circuit board T, the information on the mounting order for each solder pad, the position and orientation information of the marker, and the position and orientation information of the solder pads (solder)—to control at least one of the following: the position and orientation control of the light irradiation device 60 by the robot arm 310 of the robot 3, the control of the galvanometer mirror 61, and the change of the intensity and spot size of the processing light L. In this case, the processing light L can be efficiently irradiated to the areas on the circuit board T where the processing light L should be irradiated by the light irradiation device 60. The control device 1000 may also use at least one of the information obtained as described above—the position and orientation information of the circuit board T and the position and orientation information of the solder pads (solder)—to control the position and orientation of the dispenser 40 by the robot arm 110 of the robot 1.

[0288] The control device 1000 may also display at least one piece of information—image data, shape data, and the detection result of the element's state—used in the above-described element state detection process, on a display device (not shown).

[0289] For example, the control device 1000 may control the drive unit 211 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230, such that after one element gripped by the holding device 50, which displaces together with the displacement of at least one of the detection devices 220 and 230, is positioned at a first position on the circuit board T, the other element gripped by the holding device 50 is positioned at a second position different from the first position.

[0290] For example, the control device 1000 may control the drive unit 211 to stop driving the drive unit 211. Based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230 after the drive unit 211 has stopped driving, the control device 1000 may control the drive unit 211 so that after one element gripped by the holding device 50, which displaces together with the displacement of at least one of the detection devices 220 and 230, is positioned at a first position on the circuit board T, another element gripped by the holding device 50 is positioned at a second position different from the first position.

[0291] Furthermore, the processes in steps S111 to S117, steps S121 to S129, and steps S131 to S138 described above are performed repeatedly and concurrently.

[0292] The control device 1000 may perform machine learning using an existing method, using as training data data that associates at least one piece of information from the image data and shape data used in at least one of the processes described in steps S131 to S138, steps S111 to S117, and steps S121 to S129 with at least one piece of information from the soldering quality determined in step S138, the solder quality determined as described above, and the element quality determined as described above. In this case, the control device 1000 may use the results of machine learning to control at least one of the devices of robot 1, robot 2, and robot 3.

[0293] (Technical effects) LDS (Lase Direct Structuring) and SMT (Surface Mount Technology) are known technologies for mounting components onto the surface of circuit boards with a three-dimensional shape (i.e., 3D circuit boards). However, LDS requires the use of special heat-resistant molding materials (resins). Furthermore, the 3D circuit board must be placed inside a mounting machine (chip mounter) or reflow oven. For these reasons, LDS and similar technologies can only be applied to relatively high-cost and relatively small products.

[0294] For this reason, for relatively large products, a method is often adopted in which, for example, sub-boards are connected to a main board with cables. However, with this method, the weight of the cables can make the product relatively heavy, and manual labor is required for assembling the wiring components. On the other hand, in fields such as automobiles, there is a demand for technology to mount elements on relatively large 3D circuit boards due to space constraints and weight constraints.

[0295] For relatively large 3D circuit boards, it is desirable to use relatively inexpensive substrate materials from the perspective of cost reduction. Since relatively inexpensive substrate materials have relatively low heat resistance, laser soldering, which allows for pinpoint heat input, is a method for mounting components. However, there is a technical problem in that if the irradiation position of the processing light L, such as laser light, cannot be precisely controlled, the 3D circuit board will suffer thermal damage.

[0296] In contrast, the robot 3 described above controls the orientation of the galvanometer mirror 61, etc., based on at least one of the image data and shape data output from the detection device 330, so that even if the positional relationship between the light irradiation device 60, etc., and the object (e.g., solder pad, element, solder) changes, the irradiation position of the processing light L remains the same. In other words, the robot 3 allows for precise control of the irradiation position of the processing light L.

[0297] Soldering with Robot 3 can suppress thermal damage to the circuit board, thus increasing the range of circuit board materials that can be used. In other words, soldering with Robot 3 allows the use of relatively inexpensive circuit board materials.

[0298] According to robot 3, the robot arm 310 moves the light irradiation device 60, etc. (i.e., while the light irradiation device 60, etc. is moving), and the galvanometer mirror 61 changes the irradiation position of the processing light L so that the processing light L can be irradiated to the desired position. Therefore, one or more elements can be efficiently mounted on relatively large substrates.

[0299] According to robot 3, after the robot arm 310 moves the light irradiation device 60, etc. (in other words, when the robot arm 310 is not being driven by the drive unit 311), the galvanometer mirror 61 can be used to adjust the irradiation position of the processing light L, thereby irradiating the processing light L to the desired position. Therefore, the processing light L can be irradiated without waiting for the vibrations of the light irradiation device 60, etc., moved by the robot arm 310 to subside. Furthermore, even if there is an error in the movement of the light irradiation device 60, etc., by the robot arm 310, this error can be corrected by controlling the orientation of the galvanometer mirror 61, and the processing light L can be irradiated to the desired position.

[0300] Because robot 3 has detection devices 320 and 330, it is possible to perform a quality inspection of the soldering after soldering (after the process in step S137 described above) (see the process in step S138 described above). In other words, because robot 3 has detection devices 320 and 330, it is possible to perform a quality inspection of the soldering on the spot after soldering. To put it another way, robot 3 can perform quality inspections of soldering efficiently.

[0301] Robot 1 can recognize the position and orientation of an object (e.g., a solder pad) at predetermined time intervals through tracking processing by the control device 1000. As a result, the drive unit 111 of the robot arm 110 is controlled so that even if the relative position between the object and the dispenser 40 (detection devices 120 and 130) changes (displaces) over time, solder can be placed at the desired position on the object (in other words, where the solder should be placed).

[0302] Robot 2 can recognize the position and orientation of an object (for example, solder placed on a solder pad) at predetermined time intervals through tracking processing by the control device 1000. As a result, the drive unit 211 of the robot arm 210 is controlled so that even if the relative position between the object and the holding device 50 (detection devices 220 and 230) changes (displaces) over time, the element can be placed at the desired position on the object (in other words, where the element should be placed).

[0303] <Second Embodiment> The second embodiment will be described with reference to Figures 17 and 18. Similar to the first embodiment described above, this embodiment also includes a soldering system that includes a soldering robot. Regarding the second embodiment, explanations that overlap with the first embodiment will be omitted, common parts in the drawings will be denoted by the same reference numerals, and only the fundamentally different points will be explained with reference to Figures 17 and 18.

[0304] (overview) An overview of the soldering system according to the second embodiment will be described with reference to Figures 17(a) and 17(b). In Figure 17, the soldering system is a system for soldering elements to a circuit board T. The soldering system includes a robot 4. The robot 4, which may also be called a processing device, solder coating device, element placement device, or soldering device, is provided with a dispenser 40 for dispensing solder, a holding device 50 capable of holding elements, a light irradiation device 60 for irradiating processing light L to melt the solder, a storage section (not shown) for storing elements, a supply device (not shown) for supplying desired elements from the storage section to the holding device 50, and detection devices 420 and 430 for detecting light from the circuit board T. The robot arm 410 has a drive unit 411 for moving the dispenser 40, the holding device 50, the light irradiation device 60, and the detection devices 420 and 430.

[0305] Here, detection devices 420 and 430 correspond to detection devices 320 and 330 described above, respectively. Detection device 420 may have the same configuration as detection device 320 described above. Detection device 430 may have the same configuration as detection device 330 described above.

[0306] The soldering system includes a control device 1000 that (i) controls a drive unit 411 so that a dispenser 40, a holding device 50, a light irradiation device 60, and detection devices 420 and 430 move closer to the circuit board T; (ii) controls the dispenser 40 so that solder is placed on a predetermined portion of the circuit board T; (iii) controls the holding device 50 so that elements are placed on the circuit board T via the placed solder; and (iv) controls the light irradiation device 60 so that the placed solder melts.

[0307] In Figure 17(a), the control device 1000 controls the dispenser 40, etc., so that solder is placed on a predetermined portion of the circuit board T. The control device 1000 then controls the holding device 50, etc., so that elements are placed via the placed solder. The control device 1000 then controls the light irradiation device 60 to melt the solder. After that, the control device 1000 may perform a quality inspection related to soldering, for example, based on the detection results of the detection device 430. In other words, robot 4 performs the work that was divided among robots 1, 2, and 3 according to the first embodiment described above, all by itself. With this configuration, it is possible to improve productivity, etc., while suppressing the initial investment in introducing robots.

[0308] (Robot 4) In Figures 17(a) and 17(b), the robot 4 comprises a dispenser 40, a holding device 50, a light irradiation device 60, a storage unit (not shown) for storing elements, and a supply device (not shown) for supplying desired elements from the storage unit to the holding device 50. The robot 4 further comprises (i) detection devices 220 and 230 that detect light from the circuit board T and generate data of at least one of image data and shape data, and (ii) a robot arm 410 that is equipped with the dispenser 40, the holding device 50, the light irradiation device 60, and the detection devices 420 and 430, and has a drive unit 411 that moves the holding device 50 and the detection devices 420 and 430. Examples of the storage unit include reels, trays, sticks, etc. Various existing configurations can be applied to the storage unit and the supply device, so a detailed explanation thereof is omitted.

[0309] The robot arm 410, like the robot arm 310 described above, has arm sections 410a and 410b and a wrist section 410c.

[0310] In Figure 17(a), the detection device 420 is located on the arm portion 410b of the robot arm 410, and the detection device 430 is located on the wrist portion 410c of the robot arm 410. However, the arrangement of the detection devices 420 and 430 is not limited to this. Furthermore, the robot 4 may be equipped with only one of the detection devices 420 and 430, or it may be equipped with other detection devices in addition to the detection devices 420 and 430 (i.e., the robot 4 may be equipped with three or more detection devices). Also, the robot 4 may be equipped with at least one detection device other than the detection devices 320 and 330. In other words, as long as the dispenser 40, holding device 50, and light irradiation device 60 can be brought close to the circuit board T or a predetermined part of the circuit board T (for example, solder pads provided on the circuit board T, or elements and solder placed on the circuit board T) by the drive unit 411 of the robot arm 410, the configuration (for example, the number and specifications of cameras in the detection device, the presence or absence of a projector, etc.), placement position, and number of detection devices 420 and 430 do not matter.

[0311] The control device 1000 may, with respect to the robot 4 configured as described above, control the drive unit 411 based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 420 and 430, so that (i) the solder dispensed from the dispenser 40 which displaces with the displacement of at least one of the detection devices 420 and 430 is placed on a predetermined part of the circuit board T; (ii) the drive unit 411 is controlled so that the element gripped (held) by the holding device 50 which displaces with the displacement of at least one of the detection devices 420 and 430 is placed on a predetermined part of the circuit board T; or (iii) the orientation of the galvanometer mirror 61 is controlled so that the processing light L from the light irradiation device 60 which displaces with the displacement of at least one of the detection devices 420 and 430 is irradiated to the same position.

[0312] Here, the control device 1000 may be a separate device from the robot 4, or it may constitute a part of the robot 4 (in other words, the robot 4 may be equipped with the control device 1000).

[0313] The robot arm 410 may be mounted on an AGV (Automatic Guided Vehicle), for example. In this case, the control device 1000 may control at least one of the drive unit of the robot arm 410, the end effector of the robot arm 410, and the drive unit of the AGV based on the position and orientation information of the object acquired by the matching process and tracking process described above and below.

[0314] (Robot movements) First, the control device 1000 may perform calibration of the dispenser 40, the holding device 50, and the light irradiation device 60 before performing the following steps.

[0315] As a prerequisite, the detection device 430, dispenser 40, holding device 50, and light illumination device 60 are mounted on the robot arm 410 in such a positional relationship that a portion of the dispenser 40, a portion of the holding device 50, and a portion of the light illumination device 60 are within the field of view of the cameras 31 and 32 of the detection device 430, respectively.

[0316] The positions and orientations of the dispenser 40, the holding device 50, and the light irradiation device 60 are calculated in advance using a process similar to the calibration process described above.

[0317] Next, the operation of robot 4 will be explained with reference to the flowchart in Figure 18. However, the control device 1000 performs each of the processes described in the flowchart in Figure 18 using the output of at least one of the detection devices 420 and 430 provided by robot 4.

[0318] The control device 1000 that controls the robot 1 calculates the position and orientation of the circuit board T as an example of an object (step S111). In the process of step S111, the control device 1000 calculates the initial position and orientation of the circuit board T (i.e., the initial position and orientation) by the matching process of the matching unit 301. Furthermore, the control device 1000 calculates the position of each solder pad formed on the circuit board T. For example, the control device 1000 calculates the position and orientation of each solder pad on the circuit board T based on the Gerber data of the circuit board T (i.e., the design data of the circuit board T). Based on the Gerber data, the control device 1000 identifies the order in which to mount (in this case, place solder) each solder pad.

[0319] Next, the control device 1000 controls the drive unit 411 to move the robot arm 410 so that the dispenser 40 (detection devices 420 and 430) approaches the circuit board T (step 112). In step S112, the control device 1000 controls the drive unit 411 of the robot arm 410 so that the first solder pad to be mounted is within the field of view of at least one of the detection devices 420 and 430.

[0320] Next, the control device 1000 determines whether the first solder pad to be mounted is within the field of view of at least one of the detection devices 420 and 430 (step S113). In step S113, the control device 1000 determines whether the detection devices 420 and 430 are in the desired position and orientation relative to the first solder pad to be mounted, based on the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals and the position and orientation information of the first solder pad to be mounted calculated in step S111. At this time, if the detection devices 420 and 430 are in the desired position and orientation relative to the solder pad, the control device 1000 determines that the solder pad is within the field of view of at least one of the detection devices 420 and 430.

[0321] In step S113, if it is determined that the solder pad is not within the field of view of at least one of the detection devices 420 and 430 (step S113: No), the control device 1000 controls the drive unit 411 to continue moving the robot arm 410 based on the position and orientation information of the solder pad calculated in step S111 and the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals.

[0322] On the other hand, if, in step S113, it is determined that the solder pad has entered the field of view of at least one of the detection devices 420 and 430 (step S113: Yes), the control device 1000 calculates the position and orientation of the first solder pad to be mounted (step S114). In step S114, the control device 1000 calculates the initial position and orientation (initial position and orientation) of the solder pad by the matching process of the matching unit 301.

[0323] Next, the control device 1000 controls the drive unit 411 so that the position and orientation of the dispenser 40 are in a desired position and orientation that allows solder to be dispensed onto the first solder pad to be mounted, and moves the robot arm 410 (step S115). In the process of step S115, the control device 1000 uses the initial position and orientation information of the solder pad calculated in the process of step S114 to control the drive unit 411 based on the position and orientation of the solder pad output from the 2D tracking unit 302 at predetermined time intervals, and moves the robot arm 410. In other words, the control device 1000 controls the drive unit 411 so that the dispenser 40 (detection devices 420 and 430) approaches the first solder pad to be mounted on the circuit board T, and moves the robot arm 410.

[0324] Next, the control device 1000 determines whether the position and orientation of the dispenser 40 are in a desired position and orientation that allows solder to be dispensed onto the first solder pad to be mounted (step S116). In the process of step S116, the control device 1000 determines whether the position and orientation of the dispenser 40 relative to the solder pad is in a desired position and orientation, for example, based on the position and orientation information of the first solder pad to be mounted output from the 2D tracking unit 302 at predetermined time intervals.

[0325] In step S116, if it is determined that the position and orientation of the dispenser 40 are not the desired position and orientation for dispensing solder to the solder pad (step S116: No), the control device 1000 controls the drive unit 411 to continue moving the robot arm 410 so that the dispenser 40 approaches the solder pad, based on the position and orientation of the solder pad output from the 2D tracking unit 302 at predetermined time intervals. On the other hand, if it is determined that the position and orientation of the dispenser 40 are the desired position and orientation for dispensing solder to the solder pad (step S116: Yes), the control device 1000 controls the dispenser 40 so that solder is placed on at least a portion of the solder pad (step S117). Specifically, for example, the control device 1000 controls the dispenser 40 to dispense solder. Here, the control device 1000 may estimate the area of ​​the solder pad based on the position and orientation of the solder pad output from the 2D tracking unit 302 at predetermined time intervals, and control the amount of solder dispensed from the dispenser 40 according to the area of ​​the solder pad. In this case, the control device 1000 may control the dispenser 40 so that the larger the estimated area of ​​the solder pad, the greater the amount of solder dispensed. This makes it possible to place an appropriate amount of solder on the solder pad.

[0326] At this time, the relative position between the dispenser 40 and the workpiece may change due to, for example, vibration of the robot arm 410. Therefore, the position of the workpiece in the image shown by, for example, image data output sequentially from the detection device 430 may also change over time due to the aforementioned change in relative position.

[0327] Therefore, in order to suppress or eliminate the influence of the relative position fluctuations on the processing of step S117, the control device 1000 controls at least one of the attitude and position of the dispenser 40 based on the results of the tracking process described above.

[0328] The control device 1000 may also display at least one of the following information on a display device (not shown): image data, shape data, position and orientation information of the circuit board T, position and orientation information of each solder pad, and information on the order in which components are mounted on each solder pad.

[0329] Furthermore, if solder can be placed on the solder pad, the control device 1000 does not need to perform the processes described in steps S111 and S113 or steps S115 and S116 above.

[0330] Furthermore, after the processing in step S117, the control device 1000 may detect the state of the solder placed on the solder pad based on at least one of the image data and shape data output from at least one of the detection devices 420 and 430, or based on the matching process. Examples of the solder state to be detected include the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, the position and orientation of the solder, etc. The control device 1000 may then determine the quality of the solder based on the detected solder state.

[0331] In parallel with the processing of steps S111 to S117 described above, or at least before the processing of step S128 described later, the control device 1000 controls a supply device (not shown) provided on the robot arm 410 to supply the desired element from the storage unit (not shown) to the holding device 50, and controls the holding device 50 to hold the supplied element (step S141).

[0332] After the processing in step S117 (step S141), the control device 1000 controls the drive unit 211 to move the robot arm 210 so that the position and orientation of the holding device 50 are in a desired position and orientation that allows the element to be placed on the solder placed on the solder pad to be first mounted (step S126). In the processing of step S126, the control device 1000 controls the drive unit 411 to move the robot arm 410 based on the position and orientation information of the marker placed near the solder pad, which is output from the 2D tracking unit 302 at predetermined time intervals using, for example, the initial position and orientation information of the marker placed near the solder pad calculated in the processing of step S114, and the positional relationship between the solder pad and the marker.

[0333] Next, the control device 1000 determines whether the position and orientation of the holding device 50 are the desired position and orientation that allows the element to be placed on the solder placed on the first solder pad to be mounted (step S127). In the process of step S127, the control device 1000 determines whether the position and orientation of the holding device 50 relative to the solder pad (the solder placed on the solder pad) is the desired position and orientation, for example, based on the position and orientation information of a marker placed near the first solder pad to be mounted, which is output from the 2D tracking unit 302 at predetermined time intervals, and the positional relationship between the solder pad and the marker.

[0334] In step S127, if it is determined that the position and orientation of the holding device 50 are not the desired position and orientation that would allow the element to be placed on the solder placed on the first solder pad to be mounted (step S127: No), the control device 1000 controls the drive unit 411 to continue moving the robot arm 410 based on the position and orientation of the solder pad (solder) output from the 2D tracking unit 302 at predetermined time intervals, so that the holding device 50 approaches the solder pad (solder).

[0335] On the other hand, if it is determined that the position and orientation of the holding device 50 are a desired position and orientation that allows the element to be placed on the solder pad (solder) (step S127: Yes), the control device 1000 calculates the position and orientation of the element held by the holding device 50 (step S128).

[0336] After the processing in step S128, the control device 1000 controls the holding device 50 so that the element is placed on the first solder pad (solder) to be mounted (step S129). Based on the position and orientation of the element held by the holding device 50 calculated in the processing in step S128 above, and the position and orientation of the solder pad (solder) output from the 2D tracking unit 302 at predetermined time intervals, the control device 1000 controls the drive unit 211 of the robot arm 210 so that the position and orientation of the element held by the holding device 50 becomes a desired position and orientation that allows the element to be placed on the solder pad (solder). Then, the control device 1000 controls the holding device 50 so that the element is released from being held, and the element is placed on the solder pad (solder).

[0337] At this time, similar to the process in step S117 described above, the relative position between the holding device 50 and the work object may fluctuate due to, for example, vibration of the robot arm 410. Therefore, in order to suppress or eliminate the influence of the fluctuation in the relative position on the process in step S129, the control device 1000 controls at least one of the attitude and position of the holding device 50 based on the results of the tracking process described above.

[0338] The control device 1000 may also display at least one of the following information used in the processing of steps S126 to S129 above on a display device (not shown): image data, shape data, position and orientation information of the circuit board T, position and orientation information of the markers, position and orientation information of each solder pad, and information on the order in which components are mounted on each solder pad.

[0339] Furthermore, if the element can be placed on the solder pad, the control device 1000 does not need to perform the processes of steps S126 and S127 described above.

[0340] Furthermore, after the processing in step S129, the control device 1000 may detect the state of the elements placed on the solder pad based on at least one of the image data and shape data output from at least one of the detection devices 220 and 230, or based on the matching process. The elements to be detected include the position and orientation of the elements, the distance between the elements and the solder, and the distance between the elements and the solder pad. The control device 1000 may then determine the quality of the placed elements based on the detected state of the elements.

[0341] The control device 1000 may also display at least one piece of information—image data, shape data, and the detection result of the element's state—used in the above-described element state detection process, on a display device (not shown).

[0342] Next, the control device 1000 moves the robot arm 310 by controlling the drive unit 311 so that the position and orientation of the light irradiation device 60 are in a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad (step S135). In the process of step S135, the control device 1000 uses the initial position and orientation information of the element calculated (estimated) in the process of step S134 to control the drive unit 311 based on the position and orientation of the element output from the 2D tracking unit 302 of the tracking unit 300 at predetermined time intervals, thereby moving the robot arm 310. In other words, the control device 1000 moves the robot arm 310 by controlling the drive unit 311 so that the light irradiation device 60 (detection devices 320 and 330) approaches the element placed on the first solder pad of the circuit board T.

[0343] Next, the control device 1000 determines whether the position and orientation of the light irradiation device 60 are in a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad (step S136). In the process of step S136, the control device 1000 determines whether the position and orientation of the light irradiation device 60 relative to the element is in a desired position and orientation, for example, based on information on the position and orientation of the element output from the 2D tracking unit 302 at predetermined time intervals. At this time, if the position and orientation of the light irradiation device 60 relative to the element is in a desired position and orientation, the control device 1000 determines that the position and orientation of the light irradiation device 60 are in a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad.

[0344] In step S136, if it is determined that the position and orientation of the light irradiation device 60 are not the desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad (step S136: No), the control device 1000 controls the drive unit 411 to continue moving the robot arm 410 based on the position and orientation of the element output from the 2D tracking unit 302 at predetermined time intervals, so that the light irradiation device 60 approaches the element placed on the first solder pad. In other words, the process in step S135 is performed until it is determined that the position and orientation are the desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad.

[0345] On the other hand, if the position and orientation of the light irradiation device 60 are determined to be a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad (step S136: Yes), the control device 1000 controls the light irradiation device 60 to irradiate the electrodes of the element placed on the solder pad, i.e., the two electrodes of the chip LED, with the processing light L so as to melt the solder placed on the solder pad to be mounted first (step S137). As a result, the solder placed on the solder pad melts and the element is soldered to the circuit board T (the solder pad to be mounted first).

[0346] After the processing in step S137, the control device 1000 performs a quality inspection of the soldered elements and solder based on at least one of the image data and shape data output from at least one of the detection devices 320 and 330 (step S138). Examples of inspection items include misalignment of elements relative to the solder pads and lifting of the element electrodes relative to the solder pads (the so-called Manhattan phenomenon where the element electrodes separate from the solder pads).

[0347] The control device 1000 may also display at least one piece of information—image data, shape data, and the results of the soldering quality inspection—used in the soldering quality inspection process in step S138, on a display device (not shown).

[0348] Furthermore, if the solder placed on the solder pad can be melted to mount the element, the control device 1000 does not need to perform the processes described in steps S135 and S136 above. The control device 1000 may perform the process in step S128 described above before the process in step S127. Alternatively, the control device 1000 may not perform the process in step S128. In this case, the control device 1000 should control the holding device 50 so that the element is placed on the first solder pad (solder) to be mounted after the process in step S127 described above.

[0349] The control device 1000 may display at least one of the following information used in the processing of steps S135 to S137 above on a display device (not shown): image data, shape data, position and orientation information of the circuit board T, position and orientation information of each solder pad, information on the order in which components are mounted on each solder pad, and position and orientation information of the elements.

[0350] After the processing in step S138, the control device 1000 starts moving the dispenser 40 to the second solder pad to be mounted, based on the position and orientation of the first solder pad to be mounted output from the 2D tracking unit 302 at predetermined time intervals and the position and orientation of the second solder pad to be mounted calculated in step S111, and repeats the processing from step S113 onwards described above. Note that when the control device 1000 starts moving the dispenser 40 to the second solder pad to be mounted, it may perform the processing in step S111 before executing the processing from step S113 onwards described above. The control device 1000 repeats the steps described above until the placement of solder on each solder pad of the circuit board T is complete.

[0351] (Technical effects) As described above, robot 4 performs the solder placement process, element installation process, soldering process, and inspection process with a single robot arm. Therefore, the processes from step S114 onwards in Figure 17 are performed after the drive unit 411 stops driving the robot arm 410.

[0352] However, immediately after the robot arm 410 stops moving, vibrations may occur at the tip of the robot arm 410 (for example, the light irradiation device 60 as an end effector). Also, vibrations may occur due to the movement of the wrist portion 410c of the robot arm 410, such as during solder placement. If it is necessary to wait for the vibration to subside before starting processes such as solder placement each time a vibration occurs, productivity will be significantly reduced.

[0353] In contrast, the robot 4 controls, for example, at least one of the attitude and position of the light irradiation device 60, and the orientation of the galvanometer mirror 61 of the light irradiation device 60, based on the results of tracking processing similar to that of the first embodiment described above. Therefore, even if vibration occurs, the robot 4 can appropriately place solder, appropriately place elements, and appropriately irradiate the workpiece with processing light L based on the results of tracking processing. In other words, the robot 4 can start processing such as solder placement without waiting for the vibration to subside.

[0354] <Variation> Modifications of the first and second embodiments described above will now be explained.

[0355] (Modified version of detection device 320) Here, we will describe a modified version of the detection device 320, but similar modifications can be made to the detection devices 120, 220, and 420.

[0356] (1) The detection device 320 may be equipped with a single camera instead of cameras 21 and 22. In this case, the detection device 320 generates and outputs only image data. Even in this case, the control device 1000 can appropriately bring, for example, the light irradiation device 60 or the like closer to the circuit board T based on the results of the tracking process performed by the 2D tracking unit 302 (see Figure 12) based on the image data.

[0357] (2) The detection device 320 may include a projector in addition to the cameras 21 and 22. In this case, the detection device 320 may be configured to generate and output at least one of the image data and shape data of the object.

[0358] (3) The detection device 320 may include a single camera instead of cameras 21 and 22, and a projector. In this case, the detection device 320 may be configured to project structural light, for example as shown in Figure 7, onto the object from the projector, and generate image data of the object onto which the structural light has been projected using the single camera. The detection device 320 may also be configured to generate shape data in addition to image data. Various existing methods such as the phase shift method, random dot method, and TOF method can be applied to the method of generating shape data.

[0359] (Modified example of detection device 330) Here, we will describe a modified version of the detection device 330, but similar modifications can be made to the detection devices 130, 230, and 430.

[0360] (1) The detection device 330 may be equipped with a single camera instead of cameras 31 and 32 and projector 33. In this case, the detection device 330 generates and outputs only image data. Even in this case, the control device 1000 can appropriately irradiate the work target with processing light L, for example, based on the results of the tracking process performed by the 2D tracking unit 302 (see Figure 12) based on the image data.

[0361] (2) The detection device 330 does not need to be equipped with a projector 33. In this case, the detection device 330 may, for example, image the target object with cameras 31 and 32 at the same time, and generate and output shape data as a detection result based on the two image data output from cameras 31 and 32, respectively.

[0362] (3) The detection device 330 may include a single camera instead of cameras 31 and 32, and a projector 33. In this case, the detection device 330 may be configured to project structural light, for example as shown in Figure 7, onto the object from the projector 33, and generate image data of the object onto which the structural light has been projected using the single camera. The detection device 330 may also be configured to generate shape data in addition to image data. Various existing methods such as the phase shift method, random dot method, and TOF method can be applied to the method of generating shape data.

[0363] (Modified example of the light irradiation device 60) (1) In the robots 3 and 4 described above, the light irradiation device 60 and the detection devices 320, 330, 420, and 430 are each individually provided on the robot arm 310 or 410. In other words, the optical path of the processing light L within the light irradiation device 60 and the optical paths of the respective detection devices 320, 330, 420, and 430 (more precisely, the optical paths of the cameras of the respective detection devices 320, 330, 420, and 430) are different from each other. However, the configuration is not limited to this, and as shown in Figure 19, a part of the optical path of the processing light L within the light irradiation device 60 and a part of the optical path of the detection devices 320, 330, 420, or 430 (more precisely, the optical path of the cameras of the detection devices 320, 330, 420, or 430) may be common optical paths. In other words, the final optical element 63 of the light irradiation device 60 may constitute, for example, a part of the optical system of the camera provided in the detection device 330. In other words, the light irradiation device 60 may be a so-called coaxial laser processing head. The final optical element 63 may have a galvanometer mirror 61 and an fθ lens 62.

[0364] (2) The light irradiation device 60 may have a mechanism that can change the optical path, such as a MEMS (Micro Electro Mechanical System) mirror, a polygon mirror, or a DMD, instead of the galvanometer mirror 61. In this case, the mechanism that can change the optical path may function as a scanning unit that can scan the surface of the object with processing light L.

[0365] (3) The light irradiation device 60 does not need to be equipped with a scanning unit such as a galvanometer mirror 61.

[0366] (Variations of matching and tracking processes) First variation The control device 1000 may include, for example, a tracking unit 300' as shown in Figure 20, instead of the matching processing unit 200 and tracking unit 300 described above. In other words, the matching unit 301 of the tracking unit 300 described above may have a processing block or processing circuit similar to that of the matching processing unit 200 described above.

[0367] In the comparison unit 203 of the matching processing unit 200 of the tracking unit 300', if it is determined that the first matching rate is greater than the second matching rate (i.e., first matching rate > second matching rate), the image data from the image data and shape data output from the detection device 320 at predetermined time intervals is input to the 2D tracking unit 302, and the shape data from the image data and shape data is input to the 3D tracking unit 303.

[0368] Furthermore, the comparison unit 203 outputs to the 2D tracking unit 302 the position and orientation of the object (i.e., position and orientation estimation result) calculated by the first matching unit 201 using 2D matching with image data output from the detection device 320 at predetermined time intervals. The comparison unit 203 also outputs to the 3D tracking unit 303 the position and orientation of the object (i.e., position and orientation estimation result) calculated by the first matching unit 201 using shape data output from the detection device 320 at predetermined time intervals.

[0369] If the comparison unit 203 determines that the second matching rate is equal to or greater than the first matching rate (i.e., the first matching rate ≤ the second matching rate), the image data from the image data and shape data output from the detection device 330 at predetermined time intervals is input to the 2D tracking unit 302, and the shape data from the image data and shape data is input to the 3D tracking unit 303.

[0370] Furthermore, the comparison unit 203 outputs to the 2D tracking unit 302 the position and orientation of the object (i.e., position and orientation estimation result) calculated by the second matching unit 202 using image data output from the detection device 330 at predetermined time intervals. The comparison unit 203 also outputs to the 3D tracking unit 303 the position and orientation of the object (i.e., position and orientation estimation result) calculated by the second matching unit 202 using shape data output from the detection device 330 at predetermined time intervals.

[0371] As described above, the first matching unit 201 and the second matching unit 202 may each narrow down the range in which 3D matching should be performed based on the results of 2D matching, and perform 3D matching using shape data corresponding to that narrowed range (see Figure 10). This method allows the first matching unit 201 and the second matching unit 202 to perform 3D matching at high speed. In this case, the comparison unit 203 (see Figure 9) may sequentially compare the results of 3D matching by the first matching unit 201 and the second matching unit 202, and output the position and orientation estimation result with the higher matching rate to the 2D tracking unit 302 and the 3D tracking unit 303 at predetermined time intervals.

[0372] Furthermore, the control device 1000 may perform matching processing in the first matching unit 201 or the second matching unit 202 at predetermined time intervals according to the comparison result of the matching rate in the comparison unit 203, and based on the result of that matching processing (calculation result of the position and orientation of the object), correct the results of the tracking processing of the 2D tracking unit 302 and the 3D tracking unit 303, respectively (calculation result of the position and orientation of the object), using the result of the matching processing generated in the first matching unit 201 or the second matching unit 202. In other words, the control device 1000 may correct the results of the tracking processing of the 2D tracking unit 302 and the 3D tracking unit 303, respectively, at predetermined time intervals based on the result of the matching processing, similar to the timing chart in Figure 13.

[0373] Furthermore, the control device 1000 may perform matching processing in the first matching unit 201 or the second matching unit 202 at predetermined time intervals according to the comparison result of the matching rate in the comparison unit 203, and correct the result of the tracking processing of the 2D tracking unit 302 or the 3D tracking unit based on the result of the matching processing generated in the first matching unit 201 or the second matching unit 202. In this case, the control device 1000 may output the result of the matching processing from the first matching unit 201 or the second matching unit 202 to at least one of the 2D tracking unit 302 and the 3D tracking unit 303, which correct the result of the tracking processing via the comparison unit 203, based on the comparison result of the matching rate.

[0374] Furthermore, the control device 1000 does not need to correct the results of the tracking processes of the 2D tracking unit 302 and the 3D tracking unit 303 based on the results of the matching process generated by the first matching unit 201 or the second matching unit 202.

[0375] The control device 1000 may also perform the above-mentioned matching rate comparison in the comparison unit 203 for all matching process results output from the first matching unit 201 and the second matching unit 202 at predetermined time intervals. In this case, the control device 1000 may switch the matching process results output from the comparison unit 203 to the 2D tracking unit 302 and the 3D tracking unit 303 based on the matching rate comparison results generated in the comparison unit 203 at predetermined time intervals, using the matching process results from the first matching unit 201 and the matching process results from the second matching unit 202.

[0376] Furthermore, the control device 1000 does not need to perform the above-mentioned comparison of matching rates in the comparison unit 203 for all matching process results output from the first matching unit 201 and the second matching unit 202 at predetermined time intervals. For example, the control device 1000 may perform the above-mentioned comparison of matching rates in the comparison unit 203 based on the matching process results output from the first matching unit 201 and the second matching unit 202 at the timing when the object tracking process is started. Furthermore, the control device 1000 may perform the above-mentioned comparison of matching rates in the comparison unit 203 at a predetermined time, and thereafter output the results of the matching process output from the first matching unit 201 or the second matching unit 202 at predetermined time intervals based on the matching rate comparison result performed at the predetermined time to at least one of the 2D tracking unit 302 and the 3D tracking unit 303.

[0377] Furthermore, the control device 1000 only needs to output at least one of the results of the 2D matching process and the 3D matching process from at least one of the first matching unit 201 and the second matching unit 202 (via the comparison unit 203) to at least one of the 2D tracking unit 302 and the 3D tracking unit 303.

[0378] Second variation Here, we will use detection device 330 as an example of a detection device, but the same can be said for detection devices 130, 230, and 430.

[0379] In the tracking process described above, the result of the tracking process performed by the 3D tracking unit 303 (hereinafter referred to as "3D tracking process") corrects the result of the tracking process performed by the 2D tracking unit 302 (hereinafter referred to as "2D tracking process"), and the result of the 2D tracking process is output to the robot control unit 100 (see Figure 12).

[0380] However, the tracking method is not limited to the method described above (see Figure 12).

[0381] For example, in the control device 1000, the tracking result output to the robot control unit 100 may be selected (or switched) from among the results of the 2D tracking process and the results of the tracking process by the 3D tracking unit 303 (hereinafter referred to as "3D tracking process") based on predetermined determination conditions. In other words, in the control device 1000, the tracking result output to the robot control unit 100 is selected sequentially from the results of the 2D tracking process and the 3D tracking process generated at predetermined time intervals, based on predetermined determination conditions, and output to the robot control unit 100. Here, the predetermined determination conditions include, for example, the number of feature regions of the object extracted by the 2D tracking process, the temporal change in the position and orientation of the object calculated by the 2D tracking process, and the difference between the position and orientation calculated by the 2D tracking process and the position and orientation calculated by the 3D tracking process.

[0382] For example, in the control device 1000, the number of feature regions extracted by the 2D tracking process is detected, and if the number of detected feature regions is less than a predetermined number, the result of the 3D tracking process is output to the robot control unit 100. This is because when the number of feature regions is small, the accuracy of estimating the position and orientation of the object by the 2D tracking process decreases, and the control device 1000 outputs the result of the 3D tracking process, which has a higher accuracy of estimating the position and orientation of the object than the 2D tracking process, to the robot control unit 100.

[0383] Furthermore, for example, the control device 1000 calculates the temporal changes in the position and orientation of the object calculated by the 2D tracking process. The control device 1000 separates the calculated temporal changes in the position and orientation of the object into temporal changes along each axis of the robot arm's coordinate system (a coordinate system defined by the X, Y, and Z axes). If the temporal displacement along the X axis, the temporal change along the Y axis, and the temporal change along the Z axis are all greater than predetermined thresholds, the results of the 3D tracking process are output to the robot control unit 100. This is because when the position and orientation of the object change significantly in three dimensions, the accuracy of the estimation of the object's position and orientation by the 2D tracking process decreases. Therefore, the control device 1000 outputs the results of the 3D tracking process, which has higher accuracy in estimating the position and orientation of the object than the 2D tracking process, to the robot control unit 100.

[0384] Furthermore, for example, in the control device 1000, the difference between the position and orientation calculated by 2D tracking and the position and orientation calculated by 3D tracking is calculated, and if this difference is greater than a predetermined threshold, the results of the 3D tracking are output to the robot control unit 100. This is because if the difference between the position and orientation calculated by 2D tracking and the position and orientation calculated by 3D tracking is large, it is likely that the results of the 2D tracking, which have lower accuracy in estimating the position and orientation of the object by 3D tracking, contain many errors.

[0385] Furthermore, the control device 1000 may select the result to be output to the robot control unit 100 based on at least one of the following predetermined determination conditions: the number of feature regions of the object extracted by the 2D tracking process, the temporal change in the position and orientation of the object calculated by the 2D tracking process, and the difference between the position and orientation calculated by the 2D tracking process and the position and orientation calculated by the 3D tracking process. In addition, if the result to be output to the robot control unit 100 is selected based on multiple predetermined determination conditions, the control device 1000 may, for example, determine that the selection result based on at least one of the multiple determination conditions is a result in the robot control unit 100 outputting the 3D tracking process result, regardless of the selection result based on the other determination conditions. As described above, the control device 1000 selects the result of the tracking process output to the robot control unit 100 based on predetermined judgment conditions. This allows the system to switch between processing that prioritizes processing speed (2D tracking) and processing that prioritizes estimation accuracy (3D tracking) depending on the type, position, and orientation of the object, and to execute appropriate tracking processing.

[0386] Furthermore, since the processing performed by the control device 1000, other than the selection process of the results of the 2D tracking process and the 3D tracking process based on predetermined judgment conditions, is the same as the processing described above, such explanations will be omitted.

[0387] Furthermore, the second modification described above can also be applied to the tracking unit 300' shown in Figure 20. In other words, the tracking unit 300' can also adopt the same configuration as the second modification described above. When the second modification described above is applied to the tracking unit 300' shown in Figure 20, as described above, the first matching unit 201 and the second matching unit 202 may each narrow down the range in which 3D matching should be performed based on the results of 2D matching, and perform 3D matching using the shape data corresponding to that narrowed range (see Figure 10). By this method, the first matching unit 201 and the second matching unit 202 can each perform 3D matching at high speed.

[0388] Third variation In the first and second embodiments described above, the matching processing unit 200 (see Figure 9) determines, for example, whether the output of the detection device 320 or the output of the detection device 330 is input to the tracking unit 300, based on the comparison result of the matching rate. However, the control device 1000 does not necessarily have a matching processing unit 200. In this case, the control device 1000 may perform the following tracking process.

[0389] As described above, the detection device 320 of the robot 3 is configured to detect at least a portion of the circuit board T from a wide range when the circuit board T and the light irradiation device 60 are relatively far apart. The detection device 330 is configured to detect a portion of the circuit board T with high precision so that the light irradiation device 60 can get closer to a portion of the circuit board T (for example, the predetermined portion described above) in order to irradiate that portion of the circuit board T with processing light L when the circuit board T and the light irradiation device 60 are relatively close together.

[0390] For example, the control device 1000 may control the drive unit 311 so that the light irradiation device 60 and the detection devices 320 and 330 approach the circuit board T based on at least one of the image data and shape data generated when the detection device 320, which can also be called the first imaging unit, detects light from the circuit board T. When the light irradiation device 60 and the detection devices 320 and 330 approach the circuit board T to, for example, a predetermined distance, the control device 1000 may control the galvanometer mirror 61 so that the processing light L from the light irradiation device 60, which displaces together with the detection device 330, is irradiated to the same position on the circuit board T, based on at least one of the image data and shape data generated when the detection device 330, which can also be called the second imaging unit, detects light from the circuit board T and changes with the displacement of the detection device 330.

[0391] In other words, the control device 1000 may, when the circuit board T and the light irradiation device 60 are relatively far apart, control the drive unit 311 of the robot arm 310 to move the light irradiation device 60 closer to the circuit board T based on the output of the detection device 320 (e.g., at least one of image data and shape data), and when the circuit board T and the light irradiation device 60 are relatively close together, control the drive unit 311 to move the light irradiation device 60 to a desired position and orientation based on the output of the detection device 330 (e.g., at least one of image data and shape data), and further control the orientation of the galvanometer mirror 61 so that the processing light L irradiated from the light irradiation device 60 irradiates the same position on the circuit board T.

[0392] For example, the desired position and orientation of the robot 3 refers to the relative position and orientation of the light irradiation device 60 with respect to the circuit board T, which allows the processing light L emitted from the light irradiation device 60 to properly melt the solder on the circuit board T. The control device 1000 may control the drive unit 311 to move the light irradiation device 60 closer to the circuit board T based on the output of the detection device 330.

[0393] For example, when the light irradiation device 60 of the robot 3 and the circuit board T are relatively far apart, the control device 1000 performs CAD matching of the object using image data output from the detection device 320 (corresponding to the 2D matching described above) to identify the position of the object in the image shown by the image data. Next, based on the identified position of the object, the control device 1000 determines the range (for example, range A in Figure 10(b)) in which CAD matching of the object using shape data output from the detection device 320 (corresponding to the 3D matching described above) should be performed. Then, the control device 1000 performs CAD matching of the object using the shape data corresponding to the determined range.

[0394] At this time, the control device 1000 calculates the matching rate for CAD matching of the object using image data, and the matching rate for CAD matching of the object using shape data.

[0395] In parallel with the matching described above, the control device 1000 performs tracking processing similar to the tracking processing performed in the tracking unit 300, using the image data and shape data output from the detection device 320. For this reason, the control device 1000 controls the drive unit 311 so that the light irradiation device 60 and the like move closer to the circuit board T, based on the image data and shape data output from the detection device 320.

[0396] As the light irradiation device 60 approaches the circuit board T, the matching rate for CAD matching of the object using the image data and shape data output from the detection device 320 decreases. This is because, for example, the images of the circuit board T captured by the cameras 21 and 22 of the detection device 320 become blurred, or the entire circuit board T no longer fits within the field of view of the cameras 21 and 22, reducing the number of feature regions used for matching.

[0397] If the matching rate for CAD matching of the object using the image data and shape data output from the detection device 320 falls below a threshold, the control device 1000 performs CAD matching of the object using the image data and shape data output from the detection device 330 instead of the image data and shape data output from the detection device 320. In addition, the control device 1000 performs tracking processing similar to the tracking processing performed in the tracking unit 300 described above, using the image data and shape data output from the detection device 330 instead of the image data and shape data output from the detection device 320. As a result, the control device 1000 controls the drive unit 311 so that the light irradiation device 60 etc. move closer to the circuit board T based on the image data and shape data output from the detection device 330 instead of the image data and shape data output from the detection device 320.

[0398] Furthermore, the control device 1000 does not need to determine the range in which CAD matching of the object using shape data should be performed using the results of CAD matching of the object using image data. The control device 1000 may perform the matching and tracking using only one of the image data or shape data. If the control device 1000 performs the matching and tracking using only shape data, the detection devices 320 and 330 do not need to generate image data (in other words, they may generate only shape data). If the control device 1000 performs the matching and tracking using only image data, the detection devices 320 and 330 do not need to generate shape data (in other words, they may generate only image data). In this case, the detection devices 320 and 330 may be equipped with only a single camera.

[0399] The control device 1000 does not need to calculate the matching rate. In this case, the control device 1000 may, for example, perform tracking using image data and shape data output from the detection device 330 instead of image data and shape data output from the detection device 320 when the degree of blurring of the image of the object captured by the cameras 21 and 22 of the detection device 320 exceeds a threshold. Alternatively, the control device 1000 may, for example, perform tracking using image data and shape data output from the detection device 330 instead of image data and shape data output from the detection device 320 when the number of feature points used for matching using image data and shape data output from the detection device 320 falls below a threshold.

[0400] <Third Embodiment> A third embodiment will be described with reference to Figures 21 to 24. This embodiment describes a welding system including a robot that performs laser welding. For the third embodiment, explanations that overlap with the first embodiment will be omitted, and common parts in the drawings will be indicated with the same reference numerals. Only the fundamentally different points will be explained with reference to Figures 21 to 24.

[0401] A laser welding system according to the third embodiment will be described with reference to Figures 21 to 24. In Figure 21, the laser welding system is a system for welding, for example, part T1 and part T2. The laser welding system includes robot 5 and robot 6.

[0402] Here, "welding" is a concept that includes, for example, fusion welding, brazing, and build-up welding. In the case of fusion welding, for example, parts T1 and T2 are metal members (i.e., base material). In the case of brazing, for example, a metal with a lower melting point than parts T1 and T2 (i.e., brazing material / filler material) is placed between parts T1 and T2 (not shown). In the case of build-up welding, the metal for build-up may be supplied from a robot other than robots 5 and 6 (not shown).

[0403] Examples of laser beam irradiation locations (in other words, welding locations) include the boundary between part T1 and part T2 when the ends of part T1 and part T2 are butted together, the boundary between part T1 and part T2 when one of parts T1 and T2 is positioned upright on the other (see Figure 20), and the boundary created when at least a portion of one of parts T1 and T2 is superimposed on at least a portion of the other of parts T1 and T2. Note that parts T1 and T2 do not need to be in contact at these boundaries (in other words, there may be a gap between part T1 and part T2).

[0404] The robot 5, which may also be called a processing device, has a robot arm 510 that includes, for example, a holding device 50 capable of holding a part T2, and detection devices 520 and 530 for detecting light from the part T2, and a drive unit 511 for moving the holding device 50 and the detection devices 520 and 530.

[0405] The robot 6, which may also be called a processing device, is a robot that irradiates a target (in this case, a welding area) with laser light as processing light. The robot 6 has a light irradiation device 60 that irradiates laser light as processing light, detection devices 620 and 630 that detect light from the target, and a robot arm 610 that is equipped with the light irradiation device 60 and the detection devices 620 and 630 and has a drive unit 611 that moves the light irradiation device 60 and the detection devices 620 and 630.

[0406] Here, detection devices 520 and 630 correspond to the detection device 320 described above. Detection devices 520 and 620 may have the same configuration as detection device 320. Also, detection devices 530 and 630 correspond to the detection device 330 described above. Detection devices 530 and 630 may have the same configuration as detection device 330. Note that the light irradiation device 60 does not need to be equipped with a scanning unit such as a galvanometer mirror 61.

[0407] In Figure 21, detection device 520 is located on the arm of robot arm 510, detection device 530 is located on the wrist of robot arm 510, detection device 620 is located on the arm of robot arm 610, and detection device 630 is located on the wrist of robot arm 610. However, the arrangement of detection devices 520, 530, 620, and 630 is not limited to this. Also, robot 5 may be equipped with other detection devices in addition to detection devices 520 and 530 (i.e., robot 5 may be equipped with three or more detection devices). Robot 5 may be equipped with only one of detection devices 520 and 530. Similarly, robot 6 may be equipped with other detection devices in addition to detection devices 620 and 630 (i.e., robot 6 may be equipped with three or more detection devices). Robot 6 may be equipped with only one of detection devices 620 and 630.

[0408] The laser welding system includes a control device 1000 (see Figures 22 and 23) which controls (i) a holding device 50 as an end effector of the robot 5 to hold, for example, part T2 in a predetermined position on part T1, and (ii) a control device 1000 (see Figures 22 and 23) which controls the drive unit 611 of the robot arm 610 to bring the light irradiation device 60 as an end effector of the robot 6 closer to parts T1 and T2 based on the detection result of at least one of the detection devices 620 and 630, and controls the light irradiation device 60 to weld parts T1 and T2.

[0409] The control device 1000 may control the drive unit 511 of the robot 5 so that, based on at least one of the image data and shape data which changes with the displacement of at least one of the detection devices 520 and 530, a part T2, for example, which is held by a holding device 50 that displaces together with the displacement of at least one of the detection devices 520 and 530, maintains a predetermined position on part T1.

[0410] The control device 1000 may control the orientation of the galvanometer mirror 61 for the robot 6 based on at least one of the image data and shape data, which changes with the displacement of at least one of the detection devices 620 and 630, so that the laser light as processing light from the light irradiation device 60, which is displaced together with the displacement of at least one of the detection devices 620 and 630, is irradiated onto the target (in this case, the welding area).

[0411] The control device 1000 may be a separate device from the robot 5, or it may constitute a part of the robot 5 (in other words, the robot 5 may be equipped with the control device 1000). Similarly, the control device 1000 may be a separate device from the robot 6, or it may constitute a part of the robot 6 (in other words, the robot 6 may be equipped with the control device 1000).

[0412] At least one of the robot arms 510 and 610 may be mounted on an AGV (Automatic Guided Vehicle), for example. In this case, the control device 1000 may control at least one of the drive units of at least one of the robot arms 510 and 610, the end effector of at least one of the robot arms 510 and 610, and the drive unit of the AGV, based on the position and orientation information of the object acquired by the matching and tracking processes described above and below.

[0413] (Operation of each robot) The operations of robots 5 and 6 will be explained with reference to the flowchart in Figure 24. In the following explanation, part T2 shown in Figure 21 is assumed to be a flat plate shape extending in the depth direction of the paper. The laser light from the light irradiation device 60, which serves as processing light, is assumed to be irradiated onto at least a portion of the boundary between part T1 and part T2, which are the target objects.

[0414] Robot 5 For example, the holding device 50 is equipped with a gripper that can open and close its tips. The control device 1000 may perform calibration of the holding device 50 before performing the following steps S151 to S155.

[0415] As a prerequisite, it is assumed that the detection device 530 and the holding device 50 are mounted on the robot arm 510 in such a positional relationship that the tip of the holding device 50 (i.e., in the case of a gripper, the tip of the gripper that contacts part T2 when holding part T2) is within the field of view of cameras 31 and 32 of the detection device 530, which has a configuration similar to that of the detection device 330. For the sake of explanation, the cameras 31 and 32 of the detection device 530 will be referred to as the cameras 31 and 32 of the detection device 530, as an example of the detection device 530 having cameras 31 and 32 similar to those of the detection device 530.

[0416] As a calibration of the holding device 50, the control device 1000 performs the matching process described above using the shape data output from the detection device 530 when the holding device 50 is not holding part T2, and the CAD data of the holding device 50, and pre-calculates the position and orientation of the holding device 50 (for example, the position and orientation of the tip of the gripper included in the field of view of the cameras 31 and 32 of the detection device 530). In other words, the control device 1000 pre-calculates the position and orientation of the holding device 50 in the coordinate system of the robot arm 510 based on the shape data of at least a part of the holding device 50 (for example, the tip of the gripper).

[0417] Furthermore, the control device 1000 is not limited to using the detection device 530 for calibration of the holding device 50; it may also use shape data or image data output from the detection device 520. In this case, it is assumed that the detection device 520 and the holding device 50 are mounted on the robot arm 510 in such a positional relationship that the tip of the holding device 50 is within the field of view of the cameras 21 and 22 of the detection device 520, which has the same configuration as the detection device 320.

[0418] In the following steps S151 to S155, the holding device 50 may come into contact with a predetermined object, causing a change in the position and orientation of the holding device 50 relative to the detection device 530. In this case, the control device 1000 can detect a change in the position and orientation of the holding device 50 relative to the detection device 530 based on a partial change in the holding device 50 in the image data or shape data output from the detection device 530 (for example, a partial change in the holding device 50 in the image). If a change in the position and orientation of the holding device 50 relative to the detection device 530 is detected, the control device 1000 may perform calibration.

[0419] Before performing the process of step S152 described later, the control device 1000 moves the holding device 50 of the robot 5 closer to a part storage section (not shown) so that it can hold part T2, for example, and controls the drive unit 511 of the robot arm 510 and the holding device 50 so that part T2 is held by the holding device 50. The control device 1000 may also perform at least one of the matching process and tracking process described above in the picking process of part T2 described above, and hold the desired part T2 with the holding device 50. Here, the control device 1000 may determine the holding force (gripping force) of part T2 in the holding device 50 according to the size of part T2 calculated in at least one of the matching process and tracking process. This prevents damage to part T2 caused by holding part T2 with the holding device 50 and prevents part T2 from falling out of the holding device 50.

[0420] After the processing in step S151, the control device 1000 that controls the robot 5 calculates the position and orientation of part T1 (step S152). In the processing in step S152, the control device 1000 calculates the initial position and orientation of part T1 (i.e., initial position and orientation) by the matching process of the matching unit 301. Furthermore, the control device 1000 calculates the position where part T2 should be placed on part T1 based on the design data of the welded part T1 and part T2.

[0421] Next, the control device 1000 controls the drive unit 511 to move the robot arm 510 so that the holding device 50 (detection devices 520 and 530) approaches the part T1 (step 153). In step S153, the control device 1000 controls the drive unit 511 of the robot arm 510 so that the location on part T1 where part T2 should be placed is within the field of view of at least one of the detection devices 520 and 530.

[0422] Next, the control device 1000 determines whether the location on part T1 where part T2 should be placed is within the field of view of at least one of the detection devices 520 and 530 (step S154). In step S154, the control device 1000 determines whether the detection devices 520 and 530 are in the desired position and orientation relative to the location on part T1 where part T2 should be placed, based on the position and orientation information of part T1 output from the 2D tracking unit 302 at predetermined time intervals and the position and orientation information of the location on part T1 where part T2 should be placed, calculated in step S152. At this time, if the detection devices 520 and 530 are in the desired position and orientation relative to the location on part T1 where part T2 should be placed, the control device 1000 determines that the location on part T1 where part T2 should be placed is within the field of view of at least one of the detection devices 520 and 530.

[0423] In step S124, if it is determined that the location on part T1 where part T2 should be placed is not within the field of view of at least one of the detection devices 520 and 530 (S154: No), the control device 1000 controls the drive unit 511 to continue moving the robot arm 510 ba...

Claims

[Claim 1] A control device for performing data processing for controlling the drive unit of a robot arm that moves an end effector, an imaging device, and a shape measuring device that perform processing on an object, the end effector, the imaging device, and the shape measuring device, The position and orientation of the object are calculated based on the image data acquired by the imaging device during the relative movement between the robot arm and the object as the imaging device and the shape measuring device approach the object, and the shape data representing the three-dimensional shape of the object generated by measuring the object with the shape measuring device. Based on two image data obtained by the imaging device imaging the object at different times during the relative movement, and two shape data representing the three-dimensional shapes of the object at each of the different times, generated by the shape measuring device measuring the object, the displacement of the object between the different times is calculated. The calculated position and orientation of the object are added to the calculated displacement of the object to output information on the position and orientation of the object. A control device equipped with a processing unit.

Images