Soldering apparatus and soldering system, and processing apparatus

Abstract

The present invention provides a soldering apparatus and soldering system that perform soldering by irradiating with processing light, and a processing apparatus that processes an object by irradiating with processing light. [Solution] A soldering apparatus that irradiates processing light to melt solder placed on a circuit board T comprises: a light irradiation device having a galvanometer mirror and irradiating processing light through the galvanometer mirror; a detection device that detects light from the circuit board and generates at least one of image data and shape data; a robot arm provided with the light irradiation device and the detection device and having a drive unit that moves the light irradiation device and the detection device; and a control device that controls the orientation of the galvanometer mirror so that the processing light from the light irradiation device, which is displaced together with the detection device, is irradiated to the same position based on at least one of the data that changes with the displacement of the detection device.

Description

[Technical Field]

[0001] The present invention relates to a soldering apparatus and soldering system that perform soldering by irradiating with processing light, and to a processing apparatus that processes an object by irradiating with processing light. [Background technology]

[0002] One example of this type of device is one that projects laser light from a laser head attached to a robot arm toward the area to be soldered (see Patent Document 1). Another related technology is described in Patent Document 2. A technical challenge of this type of device is, for example, properly soldering to a substrate with a three-dimensional shape (i.e., a 3D substrate). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] U.S. Public Gazette No. 2001 / 0054637 [Patent Document 2] U.S. Public Gazette No. 2015 / 0158176 [Overview of the project]

[0004] According to a first embodiment, a soldering apparatus is provided for irradiating a processing light to melt solder arranged on a circuit board, comprising: a light irradiation device having a galvanometer mirror and irradiating the processing light through the galvanometer mirror; a detection device that detects light from the circuit board and generates at least one of image data and shape data; a robot arm provided with the light irradiation device and the detection device and having a drive unit for moving the light irradiation device and the detection device; and a control device that controls the orientation of the galvanometer mirror so that the processing light from the light irradiation device, which is displaced together with the detection device, is irradiated to the same position based on the at least one of the data that changes with the displacement of the detection device.

[0005] According to a second embodiment, a processing apparatus is provided for irradiating an object with processing light, comprising: a light irradiation device having a scanning unit and irradiating the object with processing light via the scanning unit; a detection device for detecting light from the object; a moving device having a drive unit that moves the light irradiation device and the detection device, and the light irradiation device and the detection device; and a control device that controls the scanning unit based on the detection result of the detection device.

[0006] According to a third aspect, a soldering system for soldering an element to a circuit board is provided, comprising: a first moving device having a solder dispensing device for dispensing solder and a drive unit for moving the solder dispensing device; a second moving device having a holding device capable of holding the element and a drive unit for moving the holding device; a third moving device having a light irradiation device for irradiating processing light to melt solder and a detection device for detecting light from the circuit board, and a drive unit for moving the light irradiation device and the detection device; and a control device that (i) controls the solder dispensing device so that solder is placed on a predetermined portion of the circuit board; (ii) controls the holding device so that the element is placed on the circuit board via the placed solder; and (iii) controls the drive unit of the third moving device to bring the light irradiation device closer to the circuit board based on the detection result of the detection device, and controls the light irradiation device to melt the placed solder.

[0007] According to a fourth aspect, a soldering system for soldering an element to a circuit board is provided, comprising: a soldering dispenser for dispensing solder; a holding device capable of holding the element; a light irradiation device for irradiating processing light to melt the solder; a detection device for detecting light from the circuit board; a moving device having a drive unit for moving the soldering dispenser, the holding device, the light irradiation device and the detection device; and a control device that (i) controls the drive unit so that the soldering dispenser, the holding device, the light irradiation device and the detection device move closer to the circuit board; (ii) controls the soldering dispenser so that solder is placed on a predetermined portion of the circuit board; (iii) controls the holding device so that the element is placed on the circuit board via the placed solder; and (iv) controls the light irradiation device so that the placed solder is melted.

[0008] According to a fifth embodiment, a processing apparatus is provided for irradiating an object with processing light, comprising: a light irradiation device for irradiating the object with processing light; a detection device for detecting light from the object; a moving device having a drive unit for moving the light irradiation device and the detection device, and a control device for controlling the drive unit based on the detection result of the detection device. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a schematic diagram showing the overall structure of the soldering system according to the first embodiment. [Figure 2] Figure 2 is a system configuration diagram showing the configuration of a robot that constitutes part of the soldering system according to the first embodiment. [Figure 3] Figure 3 is a system configuration diagram showing the configuration of a robot that constitutes another part of the soldering system according to the first embodiment. [Figure 4] Figure 4 is a system configuration diagram showing the configuration of a robot that constitutes another part of the soldering system according to the first embodiment. [Figure 5]FIG. 5 is a diagram schematically showing the configuration of one detection device according to the first embodiment. [Figure 6] FIG. 6 is a diagram schematically showing the configuration of another detection device according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of structured light projected by a projector included in another detection device according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing a part of the optical path included in the light irradiation device according to the first embodiment. [Figure 9] FIG. 12 is a diagram schematically showing the configuration of the tracking unit included in the control device according to the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining the concept of the matching process according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a timing chart of the matching process according to the first embodiment. [Figure 12] FIG. 12 is a diagram schematically showing the configuration of the tracking unit included in the control device according to the first embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a timing chart of the tracking process according to the first embodiment. [Figure 14] FIG. 14 is a flowchart showing the operation of the soldering system according to the first embodiment. [Figure 15] FIG. 15 is a diagram showing an example of a method of irradiating processing light. [Figure 16] FIG. 16 is a diagram schematically showing an air blower and a smoking machine. [Figure 17] FIG. 17 is a system configuration diagram showing the configuration of the soldering system according to the second embodiment. [Figure 18] FIG. 18 is a flowchart showing the operation of the soldering system according to the second embodiment. [Figure 19] FIG. 19 is a diagram showing a part of the optical path included in the light irradiation device according to the modification. [Figure 20] FIG. 20 is a diagram schematically showing the configuration of the tracking unit according to the modification. [Figure 21] Figure 21 is a schematic diagram showing the overall configuration of the laser welding system according to the third embodiment. [Figure 22] Figure 22 is a system configuration diagram showing the configuration of a robot that constitutes part of the laser welding system according to the third embodiment. [Figure 23] Figure 23 is a system configuration diagram showing the configuration of a robot that constitutes another part of the laser welding system according to the third embodiment. [Figure 24] Figure 24 is a flowchart showing the operation of the laser welding system according to the third embodiment. [Figure 25] Figure 25 is a flowchart showing the operation of an application example of the robot according to the first embodiment. [Modes for carrying out the invention]

[0010] The embodiments will be described below with reference to the drawings.

[0011] <First Embodiment> The first embodiment will be described with reference to Figures 1 to 14. This embodiment includes a soldering system that includes a robot for performing soldering.

[0012] (overview) An overview of the soldering system according to the first embodiment will be described with reference to Figures 1 to 4. In Figure 1, the soldering system is a soldering system for soldering elements to a circuit board T. The soldering system includes robots 1, 2, and 3.

[0013] The robot 1, which may also be called a processing device or a solder coating device, has a robot arm 110, which may also be called a first moving means, that is equipped with a dispenser 40 (see Figures 2(a) and 2(b)) which dispenses solder and may also be called a solder dispensing device, and has a drive unit 111 (see Figure 2(b)) for moving the dispenser 40.

[0014] The robot 2, which may also be called a processing device or an element installation device, has a robot arm 210, which may also be called a second moving means, which has a holding device 50 (see Figures 3(a) and 3(b)) capable of holding an element, which may also be called a gripping device, and a drive unit 211 (see Figure 3(b)) for moving the holding device 50.

[0015] The robot 3, which may also be called a processing device or soldering device, has a robot arm 310, which may also be called a third moving means, equipped with a light irradiation device 60 (see Figures 4(a) and 4(b)) for irradiating processing light to melt solder, and detection devices 320 and 330 (see Figures 4(a) and 4(b)) for detecting light from the circuit board T, and a drive unit 311 (see Figure 4(b)) for moving the light irradiation device 60 and the detection devices 320 and 330.

[0016] The "circuit board" may be a circuit board having a three-dimensional shape consisting of a circuit film on which a circuit is formed and a substrate (i.e., a 3D circuit board). In other words, the circuit board may be a circuit board manufactured by the IMPC (registered trademark) (In-Mold Printed Circuit) method. Furthermore, the above circuit board is not limited to a circuit board manufactured by the IMPC method, but may be a circuit board having a three-dimensional shape consisting of a circuit film and a substrate, for example, manufactured by another method. Furthermore, the above circuit board is not limited to a circuit board consisting of a circuit film and a substrate, but may be a circuit board having another three-dimensional shape. Furthermore, the above circuit board is not limited to a circuit board having a three-dimensional shape (3D circuit board), but may be a circuit board having a planar shape consisting of a circuit film on which a circuit is formed and a substrate. Furthermore, the above circuit board does not have to be a circuit board having a three-dimensional shape (3D circuit board), but may be a circuit board on which the circuit is formed on the substrate itself. Furthermore, the above circuit board may be a surface mount circuit board or a through-hole mount circuit board.

[0017] The circuit board T may include, for example, markers (e.g., 2D codes such as AR (Augmented Reality) markers or cross marks) and solder pads (lands) that can be used for attitude control and position control of at least one end effector of robots 1, 2, and 3 (i.e., dispenser 40, holding device 50, light irradiation device 60) or the detection device described later. Here, the above markers and solder pads are detectable by the detection device described later (e.g., recognizable by image) for attitude control and position control of at least one end effector of robots 1, 2, and 3 or the detection device described later.

[0018] A "component" is a component that is soldered to a circuit board T by the soldering system, and examples include electronic components and electrical components. Furthermore, the above-mentioned components may be surface-mount components or through-hole components (i.e., leaded components). Such "components" may also be referred to as "parts". Examples of components include LEDs (Light Emitting Diodes) (e.g., well-known components such as chip LEDs), resistors (e.g., well-known components such as chip resistors), capacitors (e.g., well-known components such as chip capacitors), transistors (e.g., well-known components such as chip transistors), and connectors.

[0019] Although the term "robot arm" is used, it is not limited to robot arms (i.e., vertical articulated robots), but can also be applied to various existing forms such as SCARA robots (i.e., horizontal articulated robots), parallel link robots, and Cartesian robots. Furthermore, existing moving mechanisms may be used instead of the robot arm 312, etc., as long as they can move the light irradiation device 60, etc. The robot arms 110, 210 and 310 may be industrial robots or collaborative robots.

[0020] The soldering system includes a control device 1000 (see Figures 2(b), 3(b), and 4(b)) which controls the dispenser 40 as the end effector of robot 1 so that solder is placed on a predetermined portion of the circuit board T, (ii) the holding device 50 as the end effector of robot 2 so that elements are placed on the circuit board T via the placed solder, and (iii) the drive unit 311 of robot arm 310 so that the light irradiation device 60 as the end effector of robot 3 is brought closer to the circuit board T based on the detection result of at least one of the detection devices 320 and 330, and controls the light irradiation device 60 so that the placed solder is melted.

[0021] In Figure 1, the control device 1000 first controls the dispenser 40 of the robot 1 so that solder is placed on a predetermined portion of the circuit board T being transported by the belt conveyor (in other words, so that solder is applied) (solder placement process). Next, the control device 1000 controls the holding device 50 of the robot 2 so that elements are placed on the circuit board T' where the solder has been placed, via the placed solder (element placement process). Next, based on the detection result of at least one of the detection devices 320 and 330, the control device 1000 controls the drive unit 311 of the robot arm 310 to bring the light irradiation device 60 closer to the circuit board T'' where the elements are placed, and controls the light irradiation device 60 to melt the placed solder (soldering process). After that, the control device 1000 may, for example, inspect the soldered solder and elements based on the detection result of the detection device 330 (inspection process).

[0022] By having the three robots 1, 2, and 3 work together and divide the tasks in this way, the soldering of components onto the circuit board T can be made more efficient, thereby improving the throughput of component soldering.

[0023] In Figure 1, the circuit board with solder applied is shown as "Circuit Board T'", and the circuit board with components installed is shown as "Circuit Board T''", but below, to avoid complexity in description, we will refer to them all as "Circuit Board T".

[0024] The solder may be, for example, solder paste, solder wire, solder bars, etc. In other words, the dispenser 40 may place, for example, solder paste, solder wire, solder bars, etc., on the circuit board T.

[0025] With respect to the light irradiation device 60, "to melt the placed solder" includes irradiating a predetermined portion of the circuit board T with processing light to melt the solder. For example, this predetermined portion includes solder placed on the solder pads of the circuit board T. In this case, the processing light from the light irradiation device 60 is directly irradiated onto the solder to melt it. The predetermined portion also includes a part of the solder pads provided on the circuit board T (for example, a part of the solder pad where no solder is placed) or a part of an element placed on the circuit board T (for example, an electrode of an element). In this case, by irradiating a part of the circuit board T other than the solder, the solder is indirectly melted by heat conduction from that part to the solder. The circuit board T may be a planar board or a 3D circuit board having a three-dimensional shape as described above. If the circuit board T is a 3D circuit board, the predetermined portion may be set on an inclined surface on the circuit board T. In this case, the dispenser 40 of the robot 1 may place solder on at least a part of the predetermined portion of the inclined surface (for example, a solder pad). The light irradiation device 60 of the robot 3 may irradiate a predetermined part (for example, a part of a solder pad where no solder is placed) with processing light in order to melt the solder placed on the predetermined part of the inclined surface.

[0026] For each of robots 1, 2, and 3, explanations will be provided with reference to Figures 1 through 4, as well as Figures 5 through 8. Here, we will mainly explain robot 3, and explanations common to robots 1 and 2 will be omitted as appropriate.

[0027] (Robot 3) As described above, robot 3 is a robot that irradiates processing light to melt solder placed on a circuit board T. Robot 3 comprises (i) a light irradiation device 60 having a galvanometer mirror 61 (see Figure 8) and irradiating processing light through the galvanometer mirror 61; (ii) detection devices 320 and 330 that detect light from the circuit board T and generate at least one of image data and shape data; and (iii) a robot arm 310 that is equipped with the light irradiation device 60 and the detection devices 320 and 330 and has a drive unit 311 that moves the light irradiation device 60 and the detection devices 320 and 330.

[0028] As shown in Figure 4(a), the robot arm 310 has arm sections 310a and 310b and a wrist section 310c. The drive unit 311 may have, for example, a motor for rotating the entire robot arm 310, a motor for moving the entire robot arm 310 back and forth, a motor for moving each of the arm sections 310a and 310b up and down, a motor for rotating the arm section 310b and the wrist section 310c, a motor for rotating the wrist section 310c, and a motor for bending the wrist section 310c (none of which are shown). In addition to rotary joints, the robot arm 310 may also have linear joints.

[0029] The drive unit 311 rotates the entire robot arm 310, moves it back and forth, and moves at least one of the arm sections 310a and 310b up and down, thereby moving the wrist section 310c to a position near the circuit board T, for example. The drive unit 311 further rotates the arm section 310b and the wrist section 310c, and rotates and bends the wrist section 310c, thereby moving the light irradiation device 60 or changing the orientation of the light irradiation device 60 so that the processing light that melts the solder placed on the circuit board T can be irradiated to at least a portion of a predetermined area (for example, solder placed on the circuit board T, solder pads provided on the circuit board T, elements placed on the circuit board T, etc.).

[0030] In other words, the drive unit 311 moves the robot arm 310, for example, as described above, causing the detection devices 320 and 330 and the light irradiation device 60 to move toward the circuit board T, for example.

[0031] It is assumed that the robot arm 310 and the detection device 320 (more precisely, the cameras 21 and 22 of the detection device 320) have been calibrated using existing methods. For example, an object with a known precise shape (e.g., a checkerboard) is placed at the precise location in the robot arm 310's three-dimensional coordinate system (the so-called world coordinate system), the object is imaged by the detection device 320, and the correspondence between the coordinate system of the detection device 320 (the so-called camera coordinate system) and the coordinate system of the robot arm is determined by performing well-known calculations (i.e., calibration is performed). Similarly, it is assumed that the robot arm 310 and the detection device 330 (more precisely, the cameras 31 and 32 of the detection device 330) have also been calibrated using existing methods.

[0032] In Figure 4(a), the detection device 320 is located on the arm portion 310b of the robot arm 310, and the detection device 330 is located on the wrist portion 310c of the robot arm 310. However, the placement of the detection devices 320 and 330 is not limited to these. For example, both the detection device 320 and 330 may be located on the wrist portion 310c, on the arm portion 310b, or at a different location on the robot arm 310 from the wrist portion 310c or the arm portion 310b. Furthermore, the robot 3 may be equipped with only one of the detection devices 320 and 330, or it may be equipped with other detection devices in addition to the detection devices 320 and 330 (i.e., the robot 3 may be equipped with three or more detection devices). Also, the robot 3 may be equipped with at least one detection device other than the detection devices 320 and 330. In other words, as long as the light irradiation device 60 can be brought close to the circuit board T or a predetermined part of the circuit board T (for example, solder pads provided on the circuit board T, or elements and solder placed on the circuit board T) by the drive unit 311 of the robot arm 310 so that the solder placed on the circuit board T can be melted by the processing light, the configuration (for example, the number and specifications of cameras in the detection device, the presence or absence of a projector, etc.), placement, and number of detection devices 320 and 330 do not matter.

[0033] Detection device 320 The detection device 320 will be described with reference to Figure 5. In Figure 5, the detection device 320 is configured to include cameras 21 and 22, which may also be called imaging devices. Cameras 21 and 22 each have an optical component such as a lens and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) or a CCD (Charge Coupled Device). As an example, in the detection device 320, cameras 21 and 22 may be configured as a stereo camera arranged at a predetermined baseline length apart from each other.

[0034] Cameras 21 and 22 are configured to detect light incident on them using their image sensors. Examples of such incident light include light reflected by an object (e.g., at least a part of the circuit board T), light scattered by an object, and light transmitted through an object. In other words, cameras 21 and 22 detect light from objects present within their respective fields of view and capture an image of that object. That is, each of cameras 21 and 22 is configured to detect light incident on them and generate image data (i.e., data representing a two-dimensional image) as a result of the detection. In other words, cameras 21 and 22 are configured to output image data representing the captured image. In other words, each of cameras 21 and 22 is configured to detect objects. Here, "image data" refers to data where each pixel of the image sensor of each of cameras 21 and 22 is associated (in other words, linked) with the pixel value of each pixel, such as the brightness value.

[0035] The detection device 320 is configured to simultaneously image an object (for example, at least a part of a circuit board T) with cameras 21 and 22, and to generate and output shape data (i.e., shape data showing the three-dimensional shape of the object) as a detection result based on the two image data outputs from cameras 21 and 22, respectively. The detection device 320 is configured to output the generated shape data, for example, as shape data used in matching processing and tracking processing described later. Here, as an example, the shape data is three-dimensional point cloud data (hereinafter also simply referred to as point cloud data). In this case, the detection device 320 generates point cloud data by calculating the distance from cameras 21 and 22 to the object using a well-known method, based on the difference (i.e., parallax) between the position of the object on the image captured by camera 21 and the position of the object on the image captured by camera 22, the focal lengths of cameras 21 and 22, and the distance between cameras 21 and 22 (i.e., baseline length). For example, point cloud data is data in which points corresponding to each pixel of cameras 21 and 22 are associated with 3D information (X coordinate, Y coordinate, Z coordinate). Note that shape data is not limited to point cloud data; it may also be existing 3D information data such as depth image data in which the distance to the object is associated with the brightness value of each pixel.

[0036] The detection device 320 does not need to image the target object with cameras 21 and 22 and generate shape data based on the two image data outputs from cameras 21 and 22, respectively. In this case, for example, the detection device 320 may be configured to output the two image data outputs from cameras 21 and 22, respectively, as a detection result. The two image data outputs from cameras 21 and 22, respectively, may be input to the control device 1000. In this case as well, the control device 1000 may generate shape data based on the two input image data using a well-known method, similar to that described above. The control device 1000 may perform matching processing and tracking processing, as described later, based on the generated shape data.

[0037] Furthermore, the detection device 320 is configured to output image data generated by at least one of the cameras 21 and 22, for use as image data for matching processing, tracking processing, etc., as described later.

[0038] The detection device 320 is configured to detect at least a part of the circuit board T from a wide area in order to approach the circuit board T when the circuit board T, which is the target object, and, for example, the light irradiation device 60 are relatively far apart. In other words, it is configured to image at least a part of the circuit board T and its surroundings, and to generate at least one of wide-area image data and shape data. For this reason, cameras 21 and 22 are used that have a wider field of view than cameras 31 and 32 of the detection device 330, which will be described later. For example, cameras 21 and 22 are used that have a larger angle of view (in other words, a shorter focal length) than cameras 31 and 32, which will be described later. That is, for example, cameras 21 and 22 are used that have a larger angle of view (in other words, a shorter focal length) than the lenses of cameras 31 and 32, which will be described later.

[0039] Furthermore, the detection device 320 can detect at least a portion of the circuit board T from a wide area, not only when the circuit board T as the target object and, for example, the light irradiation device 60 are relatively far apart, but also when the circuit board T as the target object and, for example, the light irradiation device 60 are relatively close together.

[0040] The detection device 320 generates, for example, image data and shape data of at least a portion of the circuit board T as the object. Here, "at least a portion of the circuit board T" (i.e., the object) refers to, for example, at least a portion of the circuit board T itself, a marker formed on the circuit board T (e.g., a two-dimensional code such as an AR marker or a cross mark), or a solder pad (land) formed on the circuit board T. The detection device 120 may also generate, for example, image data and shape data of a marker placed on the circuit board T (e.g., a two-dimensional code such as an AR marker or a cross mark) or an element provided on the circuit board T. The detection device 120 may also generate, for example, image data and shape data of a marker placed in the vicinity of the circuit board T (e.g., a two-dimensional code such as an AR marker or a cross mark).

[0041] The detection device 320 may consist of a single camera instead of cameras 21 and 22. In this case, the single camera generates image data of the object. The detection device 320 may also include another camera in addition to cameras 21 and 22. In this case, for example, shape data may be generated by cameras 21 and 22 (stereo camera), and image data may be generated by the other camera. Furthermore, the detection device 320 may include a projector in addition to cameras 21 and 22, similar to the detection device 330 described later. In this case, the detection device 320 may be configured to generate and output at least one of the image data and shape data of the object.

[0042] In this case, the detection device 320 does not need to image the object onto which structural light is projected from the projector using cameras 21 and 22 (stereo cameras) and generate shape data based on the two image data outputs from cameras 21 and 22, respectively, as is done with the detection device 330 described later. In this case, for example, the detection device 320 may be configured to output two image data (two image data of the object onto which structural light is projected) output from cameras 21 and 22 (stereo cameras) as the detection result. The two image data outputs from cameras 21 and 22, respectively, may be input to the control device 1000. In this case as well, the control device 1000 may generate shape data based on the two input image data using a well-known method. The control device 1000 may perform matching processing and tracking processing described later based on the generated shape data.

[0043] Furthermore, the detection device 320 may be equipped with a single camera and a projector instead of cameras 21 and 22. In this case, the detection device 320 may be configured to generate and output at least one of image data and shape data of the object.

[0044] In the case of the detection device 320, which includes a single camera and a projector, it is not necessary to image the object onto which structural light is projected from the projector with the single camera and generate shape data based on the image data output from the single camera, similar to the detection device 330 described later. In this case, for example, the detection device 320 may be configured to output image data output from the single camera as a detection result. The image data output from the single camera may be input to the control device 1000. In this case as well, the control device 1000 may generate shape data based on the input image data from the single camera using a well-known method. The control device 1000 may perform matching processing and tracking processing described later based on the generated shape data.

[0045] For a specific method of generating shape data in an embodiment in which the detection device 320 is equipped with a projector, please refer to the description of the detection device 330 described later.

[0046] Detection device 330 The detection device 330 will be described with reference to Figure 6. In Figure 6, the detection device 130 is configured to include cameras 31 and 32, which may be called imaging devices, and a projector 33, which may be called a projection device. For example, cameras 31 and 32 each include optical elements such as lenses and image sensors such as CMOCs and CCDs. As an example, cameras 31 and 32 may be configured as stereo cameras arranged at a predetermined baseline length apart from each other. Cameras 31 and 32 are configured to detect light incident on themselves. Examples of such incident light include light reflected by an object (for example, at least a part of the circuit board T), light scattered by an object, and light transmitted through an object. In other words, cameras 31 and 32 detect light from objects present within their respective fields of view and capture images of those objects. In other words, cameras 31 and 32 detect light from objects present within their respective fields of view and capture images of those objects. In other words, cameras 31 and 32 are configured to detect light incident on them and generate image data (i.e., data representing a two-dimensional image) as a result of that detection. To put it another way, cameras 31 and 32 are configured to generate image data representing the captured image. The detection device 330 is configured to output image data generated by at least one of cameras 31 and 32, for use in matching processing, tracking processing, etc., as described later.

[0047] The projector 33 is configured to project structured light having a predetermined intensity distribution (in other words, a predetermined pattern) as shown in Figures 7(a) to 7(c) when the detection device 330 is in operation. Various existing configurations, such as DLP (Digital Light Processing) projectors, can be applied to the projector 33.

[0048] Here, the detection device 330 is configured to project structural light onto the object from the projector 33 and generate image data of the object onto which the structural light has been projected using cameras 31 and 32. By projecting a predetermined pattern of structural light using the projector 33, the detection device 330 can generate highly accurate shape data with minimal influence from disturbances, even when the surface of the object is dark or when there are few feature points on the surface of the object.

[0049] The detection device 330 is configured to simultaneously image an object onto which structural light is projected from the projector 33 using cameras 31 and 32, and to generate and output shape data (i.e., shape data showing the three-dimensional shape of the object) based on the two image data outputs from each camera. Here, as an example, the detection device 320 calculates the distance from cameras 31 and 32 to the object using a well-known method based on the difference (i.e., parallax) between the position of the pattern caused by structural light on the image captured by camera 31 and the position of the pattern caused by structural light on the image captured by camera 32, the focal lengths of cameras 31 and 32, and the distance between cameras 31 and 32 (i.e., baseline length), and generates three-dimensional point cloud data (hereinafter also simply referred to as point cloud data). The detection device 330 is configured to output the generated shape data, for example, as shape data used in matching processing and tracking processing described later.

[0050] Furthermore, the detection device 330 does not need to image the object onto which structural light is projected from the projector 33 using cameras 31 and 32 (stereo cameras) and generate shape data based on the two image data outputs from cameras 31 and 32, respectively. In this case, for example, the detection device 330 may be configured to output two image data outputs from cameras 31 and 32, respectively (two image data of the object onto which structural light is projected) as a detection result. The two image data outputs from cameras 31 and 32, respectively may be input to the control device 1000. The control device 1000 may generate shape data based on the two input image data in a well-known manner, as described above. The control device 1000 may perform matching processing and tracking processing, as described later, based on the generated shape data.

[0051] Furthermore, the shape data is not limited to point cloud data; it may also be existing 3D information data such as depth image data, where the distance to the object is associated with the brightness value of each pixel. Various existing methods, such as the phase shift method, random dot method, and TOF (Time-of-Flight) method, can be applied to generate the shape data.

[0052] Furthermore, the detection device 330 is configured to generate image data using at least one of the cameras 31 and 32 when no structural light is projected from the projector 33.

[0053] The detection device 330 is configured to detect at least a portion of the circuit board T with high accuracy when the circuit board T, as the object, and, for example, the light irradiation device 60 are relatively close, by getting closer to a part of the circuit board T (for example, solder placed on the circuit board T, solder pads provided on the circuit board T, elements placed on the circuit board T, etc.). In other words, it is configured to image at least a portion of the circuit board T and its surroundings, and to generate high-precision image data and at least one of shape data. For this reason, cameras 31 and 32 have a higher resolution than cameras 21 and 22 in the detection device 320. In this case, for example, cameras 31 and 32 have a narrower field of view (in other words, a longer focal length) than cameras 21 and 22. For example, the lenses of cameras 31 and 32 each have a narrower field of view (in other words, a longer focal length) than the lenses of cameras 21 and 22, respectively. Also, the lenses of cameras 31 and 32 may have a higher magnification ratio than the lenses of cameras 21 and 22, respectively.

[0054] Because the resolution of the objects captured by cameras 31 and 32 is high, the resolution of the image data and the accuracy of the shape data generated by the detection device 230 equipped with cameras 31 and 32 are higher than those of the image data and shape data generated by the detection device 320 equipped with cameras 21 and 22. For this reason, the detection device 330 can detect at least a portion of the circuit board T with higher accuracy than the detection device 320. Therefore, by using the image data and shape data generated by the detection device 330 in the control device 1000, the accuracy of position and orientation estimation by the matching process in the matching processing unit 200 and the tracking process in the tracking unit 300, which will be described later, is improved.

[0055] Furthermore, the detection device 330 can detect at least a part of the circuit board T with high accuracy not only when the circuit board T as the target object and, for example, the light irradiation device 60 are relatively close together, but also when the circuit board T as the target object and, for example, the light irradiation device 60 are relatively far apart. In other words, even when the circuit board T as the target object and, for example, the light irradiation device 60 are relatively far apart, at least a part of the circuit board T may be detected.

[0056] As a result, the field of view of cameras 31 and 32 of detection device 330 becomes narrower than the field of view of cameras 21 and 22 of detection device 320. Detection device 320 may also be referred to as the first imaging unit, and detection device 330 may be referred to as the second imaging unit.

[0057] The detection device 330 generates, for example, image data and shape data of at least a portion of the circuit board T as the object. Here, "at least a portion of the circuit board T" (i.e., the object) refers to, for example, at least a portion of the circuit board T itself, or markers (e.g., 2D codes such as AR markers or cross marks) or solder pads (lands) formed on the circuit board T. The detection device 330 may also generate, for example, image data and shape data of markers (e.g., 2D codes such as AR markers or cross marks) placed on the circuit board T, or elements or solder provided on the circuit board T. The detection device 330 may also generate, for example, image data and shape data of markers (e.g., 2D codes such as AR markers or cross marks) placed in the vicinity of the circuit board T.

[0058] The detection device 330 may consist of a single camera instead of cameras 31 and 32. In this case, the single camera generates image data of the object. The detection device 330 may also include another camera in addition to cameras 31 and 32. In this case, for example, shape data may be generated by cameras 31 and 32 (stereo cameras) and projector 33, and image data may be generated by the other camera. Alternatively, the detection device 330 may consist of a single camera and projector 33 instead of cameras 31 and 32. In this case, the detection device 330 may be configured to generate and output at least one of the image data and shape data of the object.

[0059] In an embodiment where the detection device 330 includes a single camera and a projector, it is not necessary to image the object onto which structural light is projected from the projector using the single camera and generate shape data based on the image data output from the single camera. In this case, for example, the detection device 320 may be configured to output image data output from the single camera as a detection result. The image data output from the single camera may be input to the control device 1000. The control device 1000 may generate shape data based on the input image data from the single camera using a well-known method. The control device 1000 may perform matching processing and tracking processing described later based on the generated shape data.

[0060] The detection device 330 does not necessarily have to include a projector 33. In this case, the detection device 330 may be configured to generate and output at least one of image data and shape data, similar to the detection device 320.

[0061] The detection device 330 does not need to image the target object with cameras 31 and 32 (stereo cameras) and generate shape data based on the two image data outputs from cameras 31 and 32, respectively. In this case, for example, the detection device 330 may be configured to output the two image data outputs from cameras 31 and 32, respectively, as a detection result. The two image data outputs from cameras 31 and 32, respectively, may be input to the control device 1000. In this case as well, the control device 1000 may generate shape data based on the two input image data using a well-known method, similar to that described above. The control device 1000 may perform matching processing and tracking processing, as described later, based on the generated shape data.

[0062] The fields of view of cameras 21 and 22 of detection device 320 may be the same as those of cameras 31 and 32 of detection device 330, or the fields of view of cameras 31 and 32 of detection device 330 may be larger than those of cameras 21 and 22 of detection device 320. The resolution of cameras 21 and 22 of detection device 320 may be the same as that of cameras 31 and 32 of detection device 330, or the resolution of cameras 31 and 32 of detection device 330 may be lower than that of cameras 21 and 22 of detection device 320.

[0063] Light irradiation device 60 The light irradiation device 60 will be explained with reference to Figure 8. In Figure 8, the light irradiation device 60 is configured to include a galvanometer mirror 61, which can also be called a scanning unit, and an fθ lens 62. Therefore, the light irradiation device 60 can move the irradiation position of the processing light L onto the target object (for example, at least a part of the circuit board T) along a desired direction (in other words, it can scan the irradiation position of the processing light L onto the target object). The galvanometer mirror 61 is configured to change the orientation of its mirror, and by changing the orientation of its mirror, it changes the emission direction of the processing light L incident from a light source (not shown). The processing light L emitted from the galvanometer mirror 61 is incident on the fθ lens 62. The fθ lens 62 focuses the processing light L incident from the galvanometer mirror 61. In other words, the light irradiation device 60 can change the irradiation position of the processing light L irradiated onto the circuit board T, which is the target object, via the fθ lens 62, according to the orientation of its own mirror in the galvanometer mirror 61 (in other words, the change in the direction of emission of processing light L from the galvanometer mirror 61).

[0064] For example, the galvanometer mirror 61 includes a first scanning mirror 61Y and a second scanning mirror 61X, both equipped with mirrors that can swing or rotate around predetermined axes, and is arranged such that the axes of swinging or rotating of the first scanning mirror 61Y and the second scanning mirror 61X intersect (for example, are orthogonal). In this case, the processing light L incident on the first scanning mirror 61Y is reflected by the first scanning mirror 61Y and incident on the second scanning mirror 61X, and is reflected by the second scanning mirror 61X and incident on the fθ lens 62. The fθ lens 62 focuses the processing light L incident on the second scanning mirror 61X. Because the direction of emission of the processing light L from the second scanning mirror 61X differs depending on the orientation of the first scanning mirror 61Y around its axis and the orientation of the second scanning mirror 61X around its axis (in other words, the incident position of the processing light L at the fθ lens 62 differs), the irradiation position of the processing light on the circuit board T changes depending on the orientation of the first scanning mirror 61Y and the second scanning mirror 61X.

[0065] Furthermore, the light irradiation device 60 can melt solder placed on the circuit board T by irradiating it with processing light L. In addition, it is not limited to directly irradiating the solder with processing light L; for example, the solder can be indirectly melted by irradiating solder pads provided on the circuit board T (for example, parts of solder pads where no solder is placed) with processing light, or by irradiating a part of an element (component) placed on the circuit board T (for example, an electrode) with processing light.

[0066] Furthermore, the galvanometer mirror 61 is not limited to two scanning mirrors (first scanning mirror 61Y and second scanning mirror 61X), but may consist of a single scanning mirror or three or more scanning mirrors. Furthermore, the light irradiation device 60 is not limited to the galvanometer mirror 61, but may also use other existing devices that change the direction of light emission, such as a polygon mirror, DMD (Digital Micromirror Device), or spatial light modulator. Furthermore, the light irradiation device 60 is not limited to a configuration that includes an fθ lens 62, but may be without an fθ lens 62, or may be equipped with one or more other lenses instead of the fθ lens 62.

[0067] The light source (not shown) for the processing light L incident on the galvanometer mirror 61 of the light irradiation device 60 may be located outside the soldering system, included in the soldering system, included in the robot 3, or included in the light irradiation device 60. The light source (not shown) can change the intensity of the processing light L irradiated onto the object. The method for changing the intensity of the processing light L irradiated onto the object is not limited to changing the intensity of the light emitted from the light source; methods using existing light intensity changing members such as ND filters can also be applied. The light from the light source (not shown) is incident on the galvanometer mirror 61 of the light irradiation device 60 using existing methods. The light irradiation device 60 may also be equipped with a focus lens. The focus lens consists of one or more lenses, and by changing the position of at least some of the lenses along the optical axis, the focusing position of the processing light L in the optical axis direction of the light irradiation device 60 (i.e., the focal position of the light irradiation device 60) can be changed. In other words, the spot size of the processing light L irradiated onto the object can be changed. For example, the focusing lens may be positioned in the optical path of the processing light L before it enters the galvanometer mirror 61. Furthermore, the configuration for changing the spot size of the processing light L on the object in the light irradiation device 60 is not limited to a focusing lens; existing configurations can also be applied.

[0068] The control device 1000 may control the orientation of the galvanometer mirror 61 for the robot 3 configured as described above, based on at least one of the image data and shape data, which changes with the displacement of at least one of the detection devices 320 and 330, so that the processing light L from the light irradiation device 60, which is displaced along with the displacement of at least one of the detection devices 320 and 330, is irradiated to the same position.

[0069] The control device 1000 may control the drive unit 311 to stop its operation. The control device 1000 may also control the orientation of the galvanometer mirror 61 so that the processing light L from the light irradiation device 60, which is displaced together with at least one of the detection devices 320 and 330, is irradiated to the same position, based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330 after the drive of the drive unit 311 has stopped.

[0070] The control device 1000 may control the drive unit 311 so that the light irradiation device 60 and the detection devices 320 and 330 are moved, and at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 320 and 330 moved by the robot arm 310, so that the processing light L from the light irradiation device 60 moved by the robot arm 310 is irradiated to the same position.

[0071] The control device 1000 may control the orientation of the galvanometer mirror 61 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330, so that the irradiation position of the processing light L from the light irradiation device 60, which is displaced together with the displacement of at least one of the detection devices 320 and 330, is maintained at a first position, and then maintained at a second position different from the first position.

[0072] The control device 1000 may control the drive unit 311 to stop driving the drive unit 311. After the drive unit 311 stops, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L from the light irradiation device 60, which is displaced along with the displacement of at least one of the detection devices 320 and 330, is maintained at a first position, and then at a second position different from the first position, based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330.

[0073] The control device 1000 may control the drive unit 311 to move the light irradiation device 60 and the detection devices 320 and 330, and based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 320 and 330 moved by the robot arm 310, it may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L from the light irradiation device 60 moved by the robot arm 310 is maintained at a first position, and then maintained at a second position different from the first position.

[0074] The control device 1000 may control the drive unit 311 of the robot arm 310 so that the light irradiation device 60 and the detection devices 320 and 330 approach the circuit board T based on at least one of the image data and shape data. When the light irradiation device 60 and the detection devices 320 and 330 approach the circuit board T to a predetermined distance, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the processing light L from the light irradiation device 60, which is displaced together with at least one of the detection devices 320 and 330, is irradiated to the same position, based on the at least one of the data that changes with the displacement of at least one of the detection devices 320 and 330.

[0075] Furthermore, the light irradiation device 60 does not necessarily have to be equipped with a scanning unit such as a galvanometer mirror 61.

[0076] Here, the control device 1000 may be a different device from the robot 3, which can also be called a soldering device, or it may constitute a part of the robot 3 (in other words, the robot 3 may be equipped with the control device 1000). In the latter case, the control device 1000 may be provided solely by the robot 3, or it may be shared by the robot 3 and at least one of the robots 1 and 2 (i.e., the control device 1000, which constitutes a part of the robot 3, may control the robot 3 as well as at least one of the robots 2 and 3). If the robot 3 is equipped solely with the control device 1000, the robots 1 and 2 may each be equipped solely with a control device 1000 that is different from the control device 1000 provided by the robot 3.

[0077] (Robot 1) As described above, robot 1 is a robot that places solder on a predetermined part of a circuit board T (for example, a solder pad or part of a circuit). In Figures 2(a) and 2(b), robot 1 comprises (i) a dispenser 40 that dispenses solder, (ii) detection devices 120 and 220 that detect light from the circuit board T and generate at least one of image data and shape data, and (iii) a robot arm 110 that is equipped with the dispenser 40 and the detection devices 120 and 130 and has a drive unit 111 that moves the dispenser 40 and the detection devices 120 and 130.

[0078] The robot arm 110, like the robot arm 310, has arm sections 110a and 110b and a wrist section 110c. The detection devices 120 and 130 may be configured in the same way as the detection devices 320 and 330, respectively. The dispenser 40 can change the amount of solder paste dispensed, and the control device 1000 can control the amount of solder dispensed from the dispenser 40.

[0079] In Figure 2(a), the detection device 120 is located on the arm portion 110b of the robot arm 110, and the detection device 130 is located on the wrist portion 110c of the robot arm 110. However, the arrangement of the detection devices 120 and 130 is not limited to this. Furthermore, the robot 1 may be equipped with only one of the detection devices 120 and 130, or it may be equipped with other detection devices in addition to the detection devices 120 and 130 (i.e., the robot 1 may be equipped with three or more detection devices). In other words, as long as the dispenser 40 can be brought close to the circuit board T or a predetermined portion of the circuit board T (e.g., a solder pad or a part of the circuit) by the drive unit 111 of the robot arm 110 so that solder can be placed on that predetermined portion of the circuit board T (e.g., a solder pad or a part of the circuit), the configuration (e.g., the number and specifications of cameras in the detection device, the presence or absence of a projector, etc.), the placement position, and the number of detection devices 120 and 130 are all acceptable.

[0080] Furthermore, detection devices 120 and 130 may have the same configuration as detection devices 320 and 330 described above. However, detection devices 120 and 130 do not have to have the same configuration as detection devices 320 and 330 described above. The configuration and specifications of detection device 120 can be appropriately changed as long as it does not contradict the gist or concept that can be gleaned from the description of detection device 320 described above. Similarly, the configuration and specifications of detection device 130 can be appropriately changed as long as it does not contradict the gist or concept that can be gleaned from the description of detection device 330 described above.

[0081] The control device 1000 may control the drive unit 111 for the robot 1 configured as described above, based on at least one of the image data and shape data which changes with the displacement of at least one of the detection devices 120 and 130, so that the solder dispensed from the dispenser 40, which is displaced together with the displacement of at least one of the detection devices 120 and 130, is placed on a predetermined portion of the circuit board T.

[0082] The control device 1000 may control the drive unit 111 to stop the drive of the drive unit 111. The control device 1000 may also control the drive unit 111 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 120 and 130 after the drive of the drive unit 111 has stopped, so that the solder dispensed from the dispenser 40 which displaces together with at least one of the detection devices 120 and 130 is placed on a predetermined portion of the circuit board T.

[0083] The control device 1000 may control the drive unit 111 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 120 and 130, so that the solder dispensed from the dispenser 40, which is displaced along with the displacement of at least one of the detection devices 120 and 130, is initially placed at a first position on the circuit board T, and then at a second position different from the first position.

[0084] The control device 1000 may control the drive unit 111 to stop the drive of the drive unit 111. The control device 1000 may control the drive unit 111 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 120 and 130 after the drive of the drive unit 111 has stopped, so that the solder dispensed from the dispenser 40, which is displaced together with the displacement of at least one of the detection devices 120 and 130, is initially placed at a first position on the circuit board T, and then at a second position different from the first position.

[0085] The control device 1000 may control the drive unit 111 of the robot arm 110 so that the dispenser 40 and the detection devices 120 and 130 approach the circuit board T based on at least one of the image data and shape data, and when the dispenser 40 and the detection devices 120 and 130 approach the circuit board T to a predetermined distance, the control device 1000 may control the drive unit 111 so that the solder dispensed from the dispenser 40, which is displaced together with at least one of the detection devices 120 and 130, is placed on a predetermined portion of the circuit board T, based on the at least one of the data that changes with the displacement of at least one of the detection devices 120 and 130.

[0086] Here, the control device 1000 may be a different device from the robot 1, which can also be called a solder coating device, or it may constitute a part of the robot 1 (in other words, the robot 1 may be equipped with the control device 1000).

[0087] (Robot 2) As described above, robot 2 is a robot that places elements via solder placed on a circuit board T. In Figures 3(a) and 3(b), robot 2 comprises (i) a holding device 50 for holding elements, (ii) detection devices 220 and 230 for detecting light from the circuit board T and generating at least one of image data and shape data, and (iii) a robot arm 210 to which the holding device 50 and the detection devices 220 and 230 are provided, and which has a drive unit 211 for moving the holding device 50 and the detection devices 220 and 230.

[0088] The robot arm 210, like the robot arm 310, has arm sections 210a and 210b and a wrist section 210c. The detection devices 220 and 230 may be configured similarly to the detection devices 220 and 230, respectively. Furthermore, the holding device 50 can be an existing device that can hold the element, such as a tweezers hand or a suction device.

[0089] The element holding force (gripping force) in the holding device 50 is adjustable, and the control device 1000 can control the element holding force in the holding device 50. For example, if the holding device 50 is a tweezers hand, the holding device 50 can control the force with which it grips the element at the tip of the tweezers. The robot 2 may also include a storage unit (not shown) for storing elements and a supply device (not shown) for supplying desired elements from the storage unit to the holding device 50. Examples of storage units include reels, trays, sticks, etc. Various existing configurations can be applied to the storage unit and supply device, so a detailed explanation is omitted. The control device 1000 may control the supply device to supply desired elements to be placed on a part (predetermined portion) of the circuit board T from the storage unit to the holding device 50, and control the holding device 50 to hold the elements. In this case, the robot 2 can move the holding device 50 closer to a separately provided element supply device (so-called parts feeder) (not shown) to hold the desired element in the circuit board T, thus eliminating the need to hold the element in the holding device 50. This makes the element placement work on the circuit board T more efficient.

[0090] In Figure 3(a), the detection device 220 is located on the arm portion 210b of the robot arm 210, and the detection device 230 is located on the wrist portion 210c of the robot arm 210. However, the arrangement of the detection devices 220 and 230 is not limited to this. Furthermore, the robot 2 may be equipped with only one of the detection devices 220 and 230, or it may be equipped with other detection devices in addition to the detection devices 220 and 230 (i.e., the robot 2 may be equipped with three or more detection devices). In other words, as long as the holding device 50 can be brought close to a predetermined portion of the circuit board T (solder pads, solder placed on the circuit, etc.) by the drive unit 211 of the robot arm 210 so that elements can be placed on that portion of the circuit board T, the configuration (for example, the number and specifications of cameras in the detection device, the presence or absence of a projector, etc.), the placement position, and the number of detection devices 120 and 130 are all acceptable.

[0091] Furthermore, detection devices 220 and 230 may have the same configuration as detection devices 320 and 330 described above. However, detection devices 220 and 230 do not have to have the same configuration as detection devices 320 and 330 described above. The configuration and specifications of detection device 220 can be appropriately changed as long as it does not contradict the gist or concept that can be gleaned from the description of detection device 320 described above. Similarly, the configuration and specifications of detection device 230 can be appropriately changed as long as it does not contradict the gist or concept that can be gleaned from the description of detection device 330 described above.

[0092] The control device 1000 may control the drive unit 211 of the robot 2 configured as described above, based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 220 and 230, so that the elements gripped (held) by the holding device 50, which displaces together with the displacement of at least one of the detection devices 220 and 230, are positioned on a predetermined portion of the circuit board T.

[0093] The control device 1000 may control the drive unit 211 to stop its operation. The control device 1000 may also control the drive unit 211 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230 after the drive unit 211 has stopped, so that the element gripped by the holding device 50, which displaces together with at least one of the detection devices 220 and 230, is positioned on a predetermined portion of the circuit board T.

[0094] The control device 1000 may control the drive unit 211 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230, so that after one element gripped by the holding device 50, which displaces together with the displacement of at least one of the detection devices 220 and 230, is positioned at a first position on the circuit board T, the other element gripped by the holding device 50 is positioned at a second position different from the first position.

[0095] The control device 1000 may control the drive unit 211 to stop driving the drive unit 211. After the drive unit 211 has stopped driving, the control device 1000 may control the drive unit 211 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230, so that after one element gripped by the holding device 50, which displaces with the displacement of at least one of the detection devices 220 and 230, is positioned at a first position on the circuit board T, another element gripped by the holding device 50 is positioned at a second position different from the first position.

[0096] The control device 1000 may control the drive unit 211 of the robot arm 210 so that the holding device 50 and the detection devices 220 and 230 approach the circuit board T based on at least one of the image data and shape data, and when the holding device 50 and the detection devices 220 and 230 approach the circuit board T to a predetermined distance, the control device 1000 may control the drive unit 211 so that the element gripped by the holding device 50, which displaces together with at least one of the detection devices 220 and 230, is positioned on a predetermined portion of the circuit board T, based on at least one of the data that changes with the displacement of at least one of the detection devices 220 and 230.

[0097] Here, the control device 1000 may be a different device from the robot 2, which can also be called an element installation device, or it may constitute a part of the robot 2 (in other words, the robot 2 may be equipped with the control device 1000).

[0098] In Figure 1, the control device 1000 controls the drive unit 311 so that the positional relationship between the circuit board T transported by the belt conveyor and, for example, the light irradiation device 60 of the robot 3 is a desired positional relationship, and / or so that the posture of the light irradiation device 60 is a desired posture (for example, by rotating the entire robot arm 310, moving it back and forth, moving the arm portions 310a and 310b up and down, rotating the arm portion 310b and the wrist portion 310c, rotating the wrist portion 310c, or bending it). In other words, by controlling the drive unit 311, the control device 1000 controls at least one of the position and posture of the light irradiation device 60 so that processing light L from the light irradiation device 60 is irradiated onto a predetermined portion of the circuit board T transported by the belt conveyor. At this time, the belt conveyor is temporarily stopped. That is, the belt conveyor is temporarily stopped after transporting the circuit board T to, for example, within the driveable range of the robot arm 310 of the robot 3. Then, after the processing light L from the light irradiation device 60 is irradiated onto a predetermined portion of the stopped circuit board T, the belt conveyor restarts to transport the circuit board T.

[0099] Similarly, the control device 1000 controls the drive unit 111 so that the positional relationship between the circuit board T transported by the belt conveyor and, for example, the dispenser 40 of the robot 1 is a desired positional relationship, and / or so that the posture of the dispenser 40 is a desired posture. In other words, by controlling the drive unit 111, the control device 1000 controls at least one of the position and posture of the dispenser 40 so that the solder dispensed from the dispenser 40 is placed on a predetermined portion of the circuit board T transported by the belt conveyor. At this time, the belt conveyor is temporarily stopped. That is, the belt conveyor is temporarily stopped after transporting the circuit board T to, for example, the drivable range of the robot arm 110 of the robot 1. Then, after the solder dispensed from the dispenser 40 is placed on the predetermined portion of the stopped circuit board T, the belt conveyor resumes driving to transport the circuit board T.

[0100] Similarly, the control device 1000 controls the drive unit 211 so that the positional relationship between the circuit board T transported by the belt conveyor and, for example, the holding device 50 of the robot 2 is a desired positional relationship, and / or so that the posture of the holding device 50 is a desired posture. In other words, by controlling the drive unit 211, the control device 1000 controls at least one of the position and posture of the holding device 50 so that the elements held by the holding device 50 are positioned on a predetermined portion of the circuit board T transported by the belt conveyor. At this time, the belt conveyor is temporarily stopped. That is, the belt conveyor is temporarily stopped after transporting the circuit board T to, for example, the drivable range of the robot arm 210 of the robot 2. Then, after the elements held by the holding device 50 are positioned on a predetermined portion of the stopped circuit board T, the belt conveyor resumes driving to transport the circuit board T.

[0101] The belt conveyor may be driven continuously without pausing in front of each robot arm. In other words, the control device 1000 may control the drive unit 311 so that processing light L from the light irradiation device 60 is irradiated onto a predetermined part of the circuit board T that is being transported (i.e., moving) by the belt conveyor, or control the drive unit 111 so that solder dispensed from the dispenser 40 is positioned, or control the drive unit 211 so that elements held by the holding device 50 are positioned.

[0102] The transport device for the circuit board T is not limited to a belt conveyor; various existing configurations can be applied as long as they can transport the circuit board T. Furthermore, the circuit board T does not have to be transported by a belt conveyor or the like. For example, the robots 1, 2, and 3 described above may be arranged to surround a stage on which the circuit board T is placed. In this case, the control device 1000 may control the drive unit 311 so that processing light L from the light irradiation device 60 is irradiated onto a predetermined part of the circuit board T placed on the stage, control the drive unit 111 so that solder dispensed from the dispenser 40 is placed, or control the drive unit 211 so that elements held by the holding device 50 are placed. At this time, the loading of the circuit board T onto the stage and the unloading of the circuit board T from the stage may be performed by robots other than robots 1, 2, and 3.

[0103] At least one of the robot arms 110, 210, and 310 may be mounted on an AGV (Automatic Guided Vehicle), for example. In this case, the control device 1000 may control at least one of the drive units of at least one of the robot arms 110, 210, and 310, the end effector of at least one of the robot arms 110, 210, and 310, and the drive unit of the AGV, based on the position and orientation information of the object acquired by the matching process and tracking process described later.

[0104] To enable the control of such drive units 311, etc. (in other words, robot arms 310, etc.), the processing performed by the control device 1000 will be explained with reference to Figures 9 to 13. Here, as an example, processing using the outputs of detection devices 320 and 330 provided by robot 3 will be explained. The control device 1000 may perform similar processing using the outputs of detection devices 120 and 130 provided by robot 1, and the outputs of detection devices 220 and 230 provided by robot 2.

[0105] (Calculation of the object's position and orientation through matching) The matching process for the circuit board T, which is the target object, performed in the control device 1000 will be explained with reference to Figures 9 to 11. In order for the position and orientation of the light irradiation device 60 of the robot 3 to be in the desired position and orientation relative to a part of the target object (for example, the part of the circuit board T to be irradiated with processing light L) that has been transported by the belt conveyor, it is necessary to bring the light irradiation device 60 closer to the target object using the robot arm 310. In order to bring the light irradiation device 60 closer to the target object using the robot arm 310, the control device 1000 can perform a matching process to calculate (estimate) the position and orientation of the target object.

[0106] In Figure 9, the control device 1000 includes a robot control unit 100 and a matching processing unit 200, which are either logically implemented processing blocks or physically implemented processing circuits within it. The robot control unit 100 does not necessarily have to be part of the control device 1000 and may be configured separately from it. For example, the robot control unit 100 may be a different processing block or physically implemented processing circuit from the control device 1000.

[0107] Furthermore, the matching processing unit 200 determines whether to use the output of the detection device 320 (for example, at least one of image data and shape data) or the output of the detection device 330 (for example, at least one of image data and shape data) for controlling the drive unit 311 as the light irradiation device 60, which is located relatively far from the circuit board T, approaches the circuit board T.

[0108] The matching processing unit 200 is configured to include a first matching unit 201, a second matching unit 202, and a comparison unit 203.

[0109] The first matching unit 201 performs matching between the output of the detection device 320 (for example, at least one of image data and shape data) and CAD (Computer-Aided Design) data. As a result of the matching, the first matching unit 201 outputs the position and orientation estimation result of the object, which is the result of (calculating) estimating the position and orientation of the object, and the matching rate. The position and orientation estimation result of the object may be expressed in terms of so-called 6DoF (six degrees of freedom). In this case, the position and orientation estimation result is data representing the X coordinate, Y coordinate, Z coordinate, components around the X axis (θX component), components around the Y axis (θY component), and components around the Z axis (θZ component) in the coordinate system of the robot arm 310 (so-called world coordinate system) for the X, Y, and Z axes. These data may be represented as matrices, their values ​​may be represented as tables, or they may be data in other well-known formats. The X, Y, and Z coordinates are estimated results indicating the position of the object, while the components around the X, Y, and Z axes are estimated results indicating the orientation of the object. The components around the X, Y, and Z axes can also be rephrased as yaw, roll, and pitch. In the first matching unit 201, calculating (estimating) the position and orientation of the feature region (described later) of the object is included in the concept of calculating (estimating) the position and orientation of the object.

[0110] The first matching unit 201 calculates (estimates) the position and orientation of an object by, for example, comparing the feature region of the object in the image data output from the detection device 320 (for example, a part of the object's outline in the image data) with the feature region of the object in the object's CAD data (for example, CAD data corresponding to a part of the object's outline in the image data). At this time, the first matching unit 201 first extracts the feature region of the object in the image data output from the detection device 320 and the feature region of the object in the object's CAD data. Next, the first matching unit 201 calculates the position and orientation of the object in the coordinate system (world coordinate system) of the robot arm 310 by, for example, changing the dimensions or rotating the feature region of the object in the CAD data to establish a correspondence between the feature region of the object in the image data and the feature region of the object in the CAD data. More specifically, the first matching unit 201 first determines the correspondence between the CAD coordinate system (so-called local coordinate system) and the coordinate system of the detection device 320 (camera coordinate system) so that the feature region of the object in the CAD data matches the feature region of the object in the image data. Since the position and orientation of the feature region of the object in the CAD coordinate system (i.e., the feature region of the object in the CAD data) are known, the position and orientation of the feature region of the object in the coordinate system of the detection device 320 can be determined by aligning the CAD coordinate system with the coordinate system of the detection device 320. Next, the first matching unit 201 calculates the position and orientation of the object in the coordinate system of the robot arm 310 based on the correspondence between the CAD coordinate system and the coordinate system of the detection device 320, and the correspondence between the coordinate system of the detection device 320 and the coordinate system of the robot arm 310, which was determined in advance by calibration.

[0111] Furthermore, the image data matching process performed by the first matching unit 201 can utilize various existing methods such as SIFT (Scale-Invariant Feature Transform) and SURF (Speed-Upped Robust Feature). The first matching unit 201 may also perform a matching process between multiple feature regions of an object in the image data and multiple feature regions of the same object in the CAD data to calculate the positions and orientations of these multiple feature regions in the coordinate system of the robot arm 310. In this case, the first matching unit 201 may output the calculated positions and orientations of the multiple feature regions of the object, or it may calculate (estimate) and output the position and orientation of the object's center of gravity based on the positions and orientations of the multiple feature regions.

[0112] Furthermore, the feature region of the object used to calculate its position and orientation is not limited to a part of the object's outline, but may be any region in the image that can be distinguished from its surroundings, such as markers on the object or patterns on the object's surface. For example, the feature region of the circuit board T as the object may be, for example, a marker (e.g., a 2D code such as an AR marker or a cross mark) or solder pads or solder formed on at least a part of the circuit board T, or a marker (e.g., a 2D code such as an AR marker or a cross mark) placed on the circuit board T, or an element placed on the circuit board T, or a marker (e.g., a 2D code such as an AR marker or a cross mark) placed in the vicinity of the circuit board T. Furthermore, the data that the first matching unit 201 matches with the feature region in the image data output from the detection device 320 is not limited to CAD data, but other data may be used. For example, the first matching unit 201 may perform a matching process between the feature region of image data generated by imaging a reference object (e.g., a reference circuit board) in advance and the feature region of the image data of the circuit board T output from the detection device 320. In this case, the reference object may be, for example, a good product (e.g., a good circuit board).

[0113] Furthermore, the matching rate is a value that indicates the degree of agreement between the feature region of the object in the image data and the feature region of the same object in the CAD data (here, the higher the degree of agreement between the two, the higher the matching rate).

[0114] Furthermore, the first matching unit 201 can also calculate (estimate) the position and orientation of an object by comparing, for example, the point cloud data of a feature region of the object (for example, a part of the corner of the object in the point cloud data) in the shape data (for example, point cloud data) output from the detection device 320 with the point cloud data of a feature region of the object in the CAD data of the object (for example, the point cloud data in the CAD data corresponding to a part of the corner of the object in the shape data).

[0115] At this time, the first matching unit 201 first extracts the point cloud data of the feature region of the object in the shape data (e.g., point cloud data) output from the detection device 320 and the point cloud data of the feature region of the object in the CAD data of the object. Next, the first matching unit 201 calculates the position and orientation of the object in the coordinate system of the robot arm 310 by, for example, changing the coordinates of each point and the spacing between each point in the point cloud data of the feature region of the object in the CAD data, or by rotating the point cloud, thereby establishing a correspondence between the point cloud data of the feature region of the object in the shape data and the point cloud data of the feature region of the object in the CAD data. More specifically, the first matching unit 201 first determines the correspondence between the CAD coordinate system and the coordinate system of the detection device 320 so that the point cloud data of the feature region of the object in the CAD data matches the point cloud data of the feature region of the object in the shape data. Since the position and orientation of the point cloud of the feature region of the object in the CAD coordinate system are known, the position and orientation of the point cloud of the feature region of the object in the coordinate system of the detection device 320 can be determined by aligning the CAD coordinate system with the coordinate system of the detection device 320. Next, the first matching unit 201 calculates the position and orientation of the object in the coordinate system of the robot arm 310 based on the correspondence between the CAD coordinate system and the coordinate system of the detection device 320, and the correspondence between the coordinate system of the detection device 320 and the coordinate system of the robot arm 310, which was determined in advance by calibration.

[0116] Furthermore, the shape data matching process performed by the first matching unit 201 can utilize various existing methods such as RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), and ICP (Iterative Closest Point). The first matching unit 201 may also output the calculated position and orientation of the object's feature regions as the object's position and orientation. Alternatively, the first matching unit 201 may calculate the position and orientation of multiple feature regions in the coordinate system of the robot arm 310 by matching the point cloud data of multiple feature regions of the object in the shape data with the point cloud data of the same object in the CAD data. In this case, the first matching unit 201 may output the calculated position and orientation of multiple feature regions in the object as the object's position and orientation, or it may calculate (estimate) the position and orientation of the object's center of gravity based on the position and orientation of multiple feature regions and output it as the object's position and orientation.

[0117] Furthermore, the first matching unit 201 may perform matching processing using depth images as shape data output from the detection device 320, rather than being limited to matching processing using point cloud data.The feature region of the object for which the position and orientation are calculated is not limited to a part of the corner of the object, but may be any region in which a difference in shape from the surroundings can be identified, such as the edges or irregularities of the object.For example, the feature region of the circuit board T as the object may be a marker (e.g., a code or cross mark with a three-dimensional shape), solder pads or solder formed on the circuit board T as at least a part of the circuit board T, or a marker (e.g., a code or cross mark with a three-dimensional shape) placed on the circuit board T, or an element placed on the circuit board T.The data that the first matching unit 201 matches with the feature region in the image data output from the detection device 320 is not limited to CAD data, but other data may be used. For example, the first matching unit 201 may perform a matching process between the point cloud data of a feature region in shape data (e.g., point cloud data) generated by imaging a reference object (e.g., a reference circuit board) in advance, and the point cloud data of a feature region in the shape data of the circuit board T output from the detection device 320. In this case, the reference object may be, for example, a good product (e.g., a good circuit board). Note that both CAD data and image data and shape data obtained by imaging a reference object are reference data in the matching process, and can therefore be referred to as reference data.

[0118] The detection device 320 does not necessarily have to be configured to generate shape data. In this case, the detection device 320 outputs image data (for example, two image data generated by imaging the target object with cameras 21 and 22), and the first matching unit 201 may generate shape data based on the image data output from the detection device 320 in a well-known manner, as described above. The first matching unit 201 may perform matching processing based on the generated shape data, as described above, to calculate the position and orientation of the target object.

[0119] Furthermore, the matching rate is a value that indicates the degree of agreement between the point cloud data of the feature region of the object in the shape data and the point cloud data of the feature region of the same object in the CAD data (here, the higher the degree of agreement between the two, the higher the matching rate).

[0120] Similarly, the second matching unit 202 performs matching between the output of the detection device 330 (for example, at least one of image data and shape data) and the CAD data. As a result of the matching, the second matching unit 202 outputs the position and orientation estimation result of the object and the matching rate. The matching process in the second matching unit 202 (i.e., estimation of the position and orientation of the object and calculation of the matching rate) is the same as that of the first matching unit 201 described above, so its explanation is omitted.

[0121] The detection device 330 does not necessarily have to be configured to generate shape data. In this case, the detection device 330 outputs image data (for example, two image data generated by imaging an object onto which structural light is projected from the projector 33 using cameras 31 and 32), and the second matching unit 202 may generate shape data based on the image data output from the detection device 330 using a well-known method as described above. The second matching unit 202 may perform matching processing based on the generated shape data as described above to calculate the position and orientation of the object.

[0122] In other words, the first matching units 201 and 202 each calculate (estimate) the position and orientation of the object and output the calculation result as the position and orientation estimation result. Here, for example, in order to bring the light irradiation device 60 of the robot 3 closer to the area on the circuit board T where the processing light L should be irradiated, the object whose position and orientation are calculated by the first matching unit 201 and the second matching unit 202 each may be the area to be irradiated with the processing light L itself, for example, at least a part of the circuit board T (for example, a solder pad formed on the circuit board T), or an element or solder placed on the circuit board T. Alternatively, the object may be an area whose relative position to the area to be irradiated with the processing light L is known, for example, at least a part of the circuit board T (for example, a two-dimensional code or cross mark such as an AR marker formed on the circuit board T), a marker placed on the circuit board T (for example, a two-dimensional code or cross mark such as an AR marker), or a marker placed near the circuit board T (for example, a two-dimensional code or cross mark such as an AR marker). Furthermore, the object may be located in a location other than the aforementioned area on the circuit board T. In other words, the object whose position and orientation are calculated by the first matching units 201 and 202 may be the characteristic area described above.

[0123] The CAD data and reference object data (image data and shape data) used for matching are assumed to be pre-stored in the control device 1000, for example, in memory (not shown). The first matching unit 201 and the second matching unit 202 may calculate (estimate) both the position and orientation of the object, or they may calculate (estimate) only one of the position and orientation of the object. In other words, the first matching unit 201 and the second matching unit 202 may calculate (estimate) at least one of the position and orientation of the object.

[0124] The comparison unit 203 compares the matching rate output from the first matching unit 201 (hereinafter referred to as the "first matching rate") with the matching rate output from the second matching unit 202 (hereinafter referred to as the "second matching rate").

[0125] The comparison unit 203 outputs the position and orientation estimation result output from the first matching unit 201 if the first matching rate is greater than the second matching rate, in other words, if the second matching rate is less than the first matching rate (first matching rate > second matching rate). On the other hand, the comparison unit 203 outputs the position and orientation estimation result output from the second matching unit 202 if the second matching rate is equal to or greater than the first matching rate, in other words, if the first matching rate is less than or equal to the second matching rate (first matching rate ≤ second matching rate).

[0126] The robot control unit 100, as part of the control device 1000, controls, for example, the drive unit 311 of the robot 3 based on the position and orientation estimation results output from the matching processing unit 200.

[0127] The comparison unit 203 may perform the above-mentioned comparison of matching rates for all matching process results (i.e., calculation results of the position and orientation of the object) output from the first matching unit 201 and the second matching unit 202 at predetermined time intervals. Alternatively, the comparison unit 203 may not perform the above-mentioned comparison of matching rates for all matching process results output from the first matching unit 201 and the second matching unit 202 at predetermined time intervals. In this case, the comparison unit 203 may perform the above-mentioned comparison of matching rates at a predetermined time, and thereafter output the results of the matching process output from the first matching unit 201 or the second matching unit 202 at predetermined time intervals based on the matching rate comparison result performed at the predetermined time.

[0128] Furthermore, the control device 1000 only needs to output at least one of the results of the matching process using image data and the results of the matching process using shape data from at least one of the first matching unit 201 and the second matching unit 202 via the comparison unit 203.

[0129] As mentioned above, each of the detection devices 320 and 330 of the robot 3 is capable of outputting at least one of image data and shape data. Therefore, each of the first matching unit 201 and the second matching unit 202 may perform at least one of two matching processes: matching using image data (i.e., data representing a two-dimensional image) (hereinafter referred to as "2D matching" as appropriate) and matching using shape data (e.g., three-dimensional point cloud data) (hereinafter referred to as "3D matching" as appropriate). 2D matching can calculate the position and orientation of an object faster than 3D matching. On the other hand, 3D matching can calculate (estimate) the position and orientation of an object with higher accuracy than 2D matching. Therefore, depending on the purpose, each of the first matching unit 201 and the second matching unit 202 may perform either 2D matching or 3D matching.

[0130] When performing 3D matching, using all of the shape data (for example, all of the generated point cloud data) can result in a relatively long time required for 3D matching. Therefore, the first matching unit 201 and the second matching unit 202 may perform the following processing to shorten the time required for 3D matching.

[0131] The first matching unit 201 and the second matching unit 202 each first identify the position of an object obj in a two-dimensional image, for example, as shown in Figure 10(a), based on the results of 2D matching. Next, the first matching unit 201 and the second matching unit 202 each determine the range A (see Figure 10(b)) in which 3D matching should be performed based on the identified position of the object obj (in other words, they narrow down the range in which 3D matching should be performed). The object obj whose position is identified by the 2D matching of the first matching unit 201 and the second matching unit 202 may be the feature region of the object as described above. Then, the first matching unit 201 and the second matching unit 202 each perform 3D matching using shape data corresponding to the determined range A (for example, point cloud data included in range A). Therefore, since the first matching unit 201 and the second matching unit 202 each perform 3D matching with the minimum necessary point cloud data, the time required for extracting point cloud data of feature regions can be shortened compared to conventional methods, thus enabling faster 3D matching. Furthermore, 3D matching allows for the highly accurate calculation (estimation) of the position and orientation of the target object (feature region).

[0132] To illustrate this process, let's refer to the timing chart in Figure 11 as an example. In Figure 11, the width of the black band along the time axis represents the length of time required for each process.

[0133] In Figure 11, T 2-1 The results of the first 2D matching are used to determine the area where 3D matching should be performed (see “Area crop” in Figure 11), and then T 3-1 The second 3D matching is performed. 3-1 In parallel with the first 3D matching, T 2-2 The first time and T 2-3 The second 2D matching is performed. 3-1 Once the first 3D matching is complete, T 2-4 The results of the first 2D matching are used to determine the range for which 3D matching should be performed, and then T 3-2The third 3D matching is performed. In this case, the above-described comparison unit 203 (see FIG. 9) sequentially compares the results of 3D matching by each of the first matching unit 201 and the second matching unit 202, and outputs the position and orientation estimation result with a high matching rate to the robot control unit 100 at intervals of a predetermined time. The robot control unit 100 controls, for example, the drive unit 311 of the robot 3 based on the position and orientation estimation result output from the matching processing unit 200 at intervals of a predetermined time.

[0134] Note that each of the first matching unit 201 and the second matching unit 202 does not use T for 3D matching 2-2 times and T 2-3 The second 2D matching may not be performed. Note that the comparison unit 203 may compare the results of 2D matching and output the position and orientation estimation result with a high matching rate to the robot control unit 100. Note that not limited to the timing chart of FIG. 11, each of the first matching unit 201 and the second matching unit 202 performs 2D matching and 3D matching at a predetermined timing (for example, the timing to start the tracking processing described later, etc.), determines the range in which 3D matching should be performed using the result of 2D matching as described above, and may perform 3D matching.

[0135] Note that the above-described matching process is an example and is not limited thereto. The control device 1000 may calculate (estimate) the position and orientation of the object (feature region) only by 3D matching (that is, matching using shape data), or may calculate (estimate) the position and orientation of the object (feature region) only by 2D matching (that is, matching using image data). When calculating (estimating) the position and orientation of the object only by 2D matching, the detection devices 320 and 330 may include only a single camera. Further, when either one of the detection devices 320 and 330 is provided on the robot 3, the matching processing unit 200 may be composed of only one of the first matching unit 201 and the second matching unit 202.

[0136] The robot control unit 100 does not have to be part of the control device 1000, and may be configured separately from the control device 1000. For example, the robot control unit 100 may be a different processing block or a physically implemented processing circuit from the control device 1000. In this case, the control device 1000 may generate a control signal for controlling the robot 3 (the drive unit 311 of the robot 3) based on the position and orientation (position and orientation estimation result) of the object output from the matching processing unit 200 (comparison unit 203). The control device 1000 may output the generated control signal to the robot control unit 100. The robot control unit 100 may generate a drive signal for driving the drive unit 311 of the robot 3 based on the control signal output from the control device 1000. The robot control unit 100 may drive the drive unit 311 of the robot 3 based on the generated drive signal. The control device 1000 may have an output unit (not shown) that outputs the generated control signal to the robot control unit 100. The control signals for controlling the robot 3 (the drive unit 311 of the robot 3) may be generated by the matching processing unit 200 (comparison unit 203) of the control device 1000. The matching processing unit 200 (comparison unit 203) may generate control signals for controlling the robot 3 (the drive unit 311 of the robot 3) based on the calculated (estimated) position and orientation of the object (position and orientation estimation result). The matching processing unit 200 (comparison unit 203) may output the generated control signals to the robot control unit 100.

[0137] (Calculation of the object's position and orientation through tracking) Due to the transport of the circuit board T by the belt conveyor, and / or due to the movement of the light irradiation device 60 by, for example, the robot arm 310 of robot 3, the relative position between the object and the light irradiation device 60 changes over time.

[0138] To ensure that the position and orientation of the light irradiation device 60 of the robot 3 are in the desired position and orientation relative to a part of the circuit board T (for example, the area to be irradiated with processing light L) that is being transported by the conveyor belt, the robot arm 310 of the robot 3 needs to bring the light irradiation device 60 closer to the part of the circuit board T that is being transported by the conveyor belt. In order to bring the light irradiation device 60 closer to the object with the robot arm 310, the control device 1000 can perform tracking processing to calculate (estimate) the changes in the position and orientation of the object. The object for which the changes in position and orientation are calculated (estimated) may be the area to be irradiated with processing light L itself, as described above, or it may be, for example, at least a part of the circuit board T (for example, a solder pad formed on the circuit board T), or an element or solder placed on the circuit board T. Furthermore, the target object may be a location whose relative position to the area to be irradiated with the processing light L described above is known. For example, it may be at least a part of the circuit board T (for example, a two-dimensional code such as an AR marker or a cross mark formed on the circuit board T as a marker), a marker placed on the circuit board T (for example, a two-dimensional code such as an AR marker or a cross mark), or a marker placed near the circuit board T (for example, a two-dimensional code such as an AR marker or a cross mark). In addition, the target object may be a location on the circuit board T other than the aforementioned location.

[0139] The tracking process performed in the control device 1000 will be explained with reference to Figures 12 and 13.

[0140] In Figure 12, the control device 1000 has a tracking unit 300, which is either a logically implemented processing block or a physically implemented processing circuit. The tracking unit 300 is configured to include a matching unit 301, a 2D tracking unit 302, and a 3D tracking unit 303.

[0141] Image data and shape data are input to the tracking unit 300 at predetermined time intervals. Here, if the position and orientation estimation result output from the first matching unit 201 of the matching processing unit 200 described above is output to the robot control unit 100 (i.e., if the first matching rate > the second matching rate), then, for example, image data and shape data output from the detection device 320 of the robot 3 at predetermined time intervals are input to the tracking unit 300. On the other hand, when the position and orientation estimation result output from the second matching unit 202 of the matching processing unit 200 described above is output to the robot control unit 100 (i.e., when the first matching rate ≤ the second matching rate), for example, image data and shape data output from the detection device 330 of the robot 3 at predetermined time intervals are input to the tracking unit 300.

[0142] The matching unit 301 calculates (estimates) the position and orientation of an object by performing a matching process on image data and shape data input at predetermined time intervals. The matching process in the matching unit 301 is the same as that of the first matching unit 201 and the second matching unit 202 described above, so its explanation is omitted. Alternatively, as described above, the matching unit 301 may narrow down the range in which 3D matching should be performed based on the results of 2D matching using the input image data, and then calculate (estimate) the position and orientation of the object by performing 3D matching using the shape data corresponding to that range (see Figure 10).

[0143] Image data is input to the 2D tracking unit 302 at predetermined time intervals. The 2D tracking unit 302 calculates (estimates) the displacement of the object at predetermined time intervals based on two image data sets, for example, a first image data set and a second image data set, which are input at predetermined time intervals. Meanwhile, at least the first image data input to the 2D tracking unit 302 is also input to the matching unit 301. In other words, the image data input to the tracking unit 300 is input to the 2D tracking unit 302 and the matching unit 301 at approximately the same time. The matching unit 301 calculates (estimates) the position and orientation of the object by performing the matching process described above using the input first image data.

[0144] The 2D tracking unit 302 calculates (estimates) the position and orientation of the object at a predetermined time (i.e., the time when the second image data is generated) by adding the calculated (estimated) displacement of the object to the position and orientation of the object calculated (estimated) by the matching unit 301 in the very beginning (hereinafter also referred to as the initial position and orientation). Subsequently, the 2D tracking unit 302 sequentially calculates (estimates) the displacement of the object at predetermined time intervals based on each image data input at predetermined time intervals, and calculates (estimates) the position and orientation of the object at each time by adding the calculated (estimated) displacement of the object to the position and orientation of the object that have been calculated (estimated) each time (i.e., performs tracking processing). The position and orientation of the object calculated (estimated) at each time by the tracking processing will also be expressed as 6DoF values, similar to the position and orientation calculated in the matching processing described above. More specifically, the 2D tracking unit 302 first extracts the feature region of the object in the first image data (for example, a part of the object's outline in the first image data) and the feature region of the object in the second image data (for example, a part of the object's outline in the second image data) at each input timing. Next, the 2D tracking unit 302 associates the feature region of the object in the first image data with the feature region of the object in the second image data and determines the displacement of the feature region of the object in the second image data with respect to the feature region of the object in the first image data in the camera coordinate system. Furthermore, based on the obtained displacement of the feature region of the object in the second image data with respect to the feature region of the object in the first image data in the camera coordinate system and the previously determined correspondence between the camera coordinate system and the coordinate system of the robot arm 310, the 2D tracking unit 302 calculates the displacement of the object in the robot arm 310 with respect to the object at the time the first image data was generated and the object at the time the second image data was generated.

[0145] The 2D tracking unit 302 then calculates the position and orientation of the object at a predetermined time (i.e., the time when the second image data is generated) by adding the aforementioned displacement to the position and orientation (initial position and orientation) of the object's feature region calculated by the matching unit 301. Since the time required for the 2D tracking unit 302 to calculate the position and orientation of the object is shorter than the time required for the 3D tracking unit 303, which will be described later, the position and orientation of the object can be tracked at high speed. Various existing methods can be used for extracting the feature region of the object in the first and second image data, associating the feature region of the object in the first image data with the feature region of the object in the second image data, and calculating the displacement of the feature region of the object in the coordinate system of the robot arm 310.

[0146] Existing methods can be used to extract feature regions of the object in the first and second image data, to associate the feature regions of the object in the first image data with the feature regions of the object in the second image data, and to calculate the displacement of the feature regions of the object in the second image data relative to the feature regions of the object in the first image data in the global coordinate system. The 2D tracking unit 302 may extract multiple feature regions of the object in the first image data and multiple feature regions of the object in the second image data, associate these feature regions with each other, and calculate (estimate) the displacement of each feature region of the object in the second image data relative to each feature region of the object in the first image data. The 2D tracking unit 302 may also use the position and orientation estimation result of the object output from the comparison unit 203 (i.e., the position and orientation estimation result with a high matching rate among the 3D matching results output from the first matching unit 201 and the second matching unit 202, respectively, as referenced in Figures 9 and 10) as the initial position and orientation. The characteristic region of the circuit board T as the object may be, for example, a marker (e.g., a two-dimensional code such as an AR marker or a cross mark) or solder pad or solder formed on the circuit board T as at least a part of the circuit board T, or a marker (e.g., a two-dimensional code such as an AR marker or a cross mark) placed on the circuit board T, or an element placed on the circuit board T, or a marker (e.g., a two-dimensional code such as an AR marker or a cross mark) placed in the vicinity of the circuit board T.

[0147] Shape data is input to the 3D tracking unit 303 at predetermined time intervals. The 3D tracking unit 303 calculates (estimates) the displacement of the object at predetermined time intervals based on two shape data sets, a first shape data set and a second shape data set, which are input at predetermined time intervals. Meanwhile, at least the first shape data input to the 3D tracking unit 303 is also input to the matching unit 301. In other words, the shape data input to the tracking unit 300 is input to the 3D tracking unit 303 and the matching unit 301 at approximately the same time. The matching unit 301 calculates (estimates) the position and orientation of the object by performing the matching process as described above using the input first shape data.

[0148] The 3D tracking unit 303 determines the three-dimensional displacement of the object over a very short time. The 3D tracking unit 303 estimates the current orientation and position of the object by adding the determined displacement to the initial orientation and position detected by the matching unit 301.

[0149] The 3D tracking unit 303 calculates the position and orientation of the object at a predetermined time (i.e., the time when the second shape data is generated) by adding the calculated displacement of the object to the position and orientation of the object (initial position and orientation) calculated by the matching unit 301 in the very beginning. Subsequently, the 3D tracking unit 303 sequentially calculates the displacement of the object at predetermined time intervals based on each shape data input at predetermined time intervals, and calculates the position and orientation of the object at each time by adding the calculated displacement of the object to the calculated position and orientation of the object (i.e., performing tracking processing). More specifically, for example, if the shape data is point cloud data, the 3D tracking unit 303 first extracts the point cloud data of the feature region of the object in the first shape data (e.g., a part of the corner of the object in the first shape data) and the point cloud data of the feature region of the object in the second shape data (e.g., a part of the corner of the object in the second shape data) at each input timing. The 3D tracking unit 303 then associates the point cloud data of the feature region of the object in the first shape data with the point cloud data of the feature region of the object in the second shape data, and determines the displacement in the camera coordinate system of the point cloud data of the feature region of the object in the second shape data with respect to the point cloud data of the feature region of the object in the first shape data. The 3D tracking unit 303 further calculates the displacement in the coordinate system of the robot arm 310 of the object at the time the first shape data was generated and the displacement in the camera coordinate system of the object at the time the second shape data was generated, based on the determined displacement in the camera coordinate system of the point cloud data of the feature region of the object in the second shape data with respect to the point cloud data of the feature region of the object in the first shape data, and the correspondence between the camera coordinate system and the coordinate system of the robot arm 310 that was determined in advance.

[0150] The 3D tracking unit 303 then calculates the position and orientation of the object at a predetermined time (i.e., the time when the second shape data is generated) by adding the aforementioned displacement to the position and orientation (initial position and orientation) of the point cloud data of the feature region of the object calculated by the matching unit 301. When the shape data is point cloud data, the tracking process performed by the 3D tracking unit 303 can utilize various existing methods such as RANSAC (Random Sample Consensus), SIFT (Scale-Invariant Feature Transform), and ICP (Iterative Closest Point).

[0151] Furthermore, for example, if the shape data is depth image data, the 3D tracking unit 303 first extracts the feature region of the object in the first shape data (for example, a part of the corner of the object in the first shape data) and the feature region of the object in the second shape data (for example, a part of the corner of the object in the second shape data) at each input timing. Next, the 3D tracking unit 303 associates the feature region of the object in the first shape data with the feature region of the object in the second shape data and calculates the displacement in the camera coordinate system of the feature region of the object in the second shape data with respect to the feature region of the object in the first shape data. The 3D tracking unit 303 further calculates the displacement in the coordinate system of the robot arm 310 of the object at the time the first shape data was generated and the displacement of the object at the time the second shape data was generated, based on the obtained displacement in the camera coordinate system of the object in the second shape data with respect to the feature region of the object in the first shape data and the correspondence between the camera coordinate system and the coordinate system of the robot arm 310 that was determined in advance. The 3D tracking unit 303 then calculates the position and orientation of the object at a predetermined time (i.e., the time when the second shape data is generated) by adding the aforementioned displacement to the position and orientation of the feature region of the object (initial position and orientation) calculated by the matching unit 301. If the shape data is depth image data, the tracking process by the 3D tracking unit 303 can utilize various existing methods such as DSO (Direct Sparse Odometry). Since the accuracy of the position and orientation of the object calculated by the 3D tracking unit 303 is higher than that of the position and orientation of the object calculated by the 2D tracking unit 302, the position and orientation of the object can be tracked with high accuracy.

[0152] Furthermore, the position and orientation of the object estimated by the 2D tracking unit 302, the position and orientation of the object estimated by the 3D tracking unit, and the position and orientation of the object detected by the matching unit 301 may each be output in a format corresponding to the position and orientation estimation results output from the first matching unit 201 and the second matching unit 202 described above (i.e., in 6DoF format).

[0153] The detection device 320 does not necessarily have to be configured to generate shape data. When the position and orientation estimation result output from the first matching unit 201 of the matching processing unit 200 described above is output to the robot control unit 100 (i.e., when the first matching rate > the second matching rate), the detection device 320 outputs image data (for example, two image data generated by imaging the object with cameras 21 and 22) at predetermined time intervals, and the tracking unit 300 (3D tracking unit 303) may generate shape data at predetermined time intervals based on the image data output from the detection device 320 at predetermined time intervals using a well-known method as described above. The tracking unit 300 (3D tracking unit 303) may perform the above-described tracking process based on the shape data generated at predetermined time intervals (for example, first shape data and second shape data) and calculate the displacement of the object. The tracking unit 300 (3D tracking unit 303) may calculate the position and orientation of the object at a predetermined time (the time when the image data used to generate the second shape data is generated) by adding the calculated displacement of the object to the position and orientation of the object calculated by the matching unit 301. When the position and orientation estimation result output from the second matching unit 202 of the matching processing unit 200 described above is output to the robot control unit 100 (i.e., when the first matching rate ≤ the second matching rate), the detection device 330 may output image data (for example, two image data generated by imaging an object onto which structural light is projected using cameras 31 and 32) at predetermined time intervals, and the tracking unit 300 (3D tracking unit 303 described later) may generate shape data at predetermined time intervals based on the image data output from the detection device 330 at predetermined time intervals using a well-known method as described above. The tracking unit 300 (3D tracking unit 303) may perform the above-described tracking process based on shape data (for example, first shape data and second shape data) generated at predetermined time intervals, and calculate the displacement of the object.The tracking unit 300 (3D tracking unit 303) may calculate the position and orientation of the object at a predetermined time (the time when the image data used to generate the second shape data is generated) by adding the calculated displacement of the object to the position and orientation of the object calculated by the matching unit 301.

[0154] Here, we will explain the tracking process by referring to the timing chart in Figure 13 as an example. In Figure 13, the width of the black band along the time axis represents the length of time required for each process.

[0155] In Figure 13, the initial position and orientation of the object (initial position and orientation) have already been detected by the matching unit 301, and the current position and orientation of the object are estimated by adding the displacement of the object at predetermined time intervals to this initial position and orientation. Furthermore, the position and orientation estimation accuracy of the 3D tracking unit 303 and the matching unit 301 is assumed to be higher than that of the 2D tracking unit 302.

[0156] For example, let x1 be the position and orientation of the object at time t1 (i.e., the position and orientation estimated by the matching unit 301). For example, let Δx be the displacement of the object from time t1 to time t2, as determined by the 2D tracking unit 302. 12 The displacement of the object from time t2 to time t3 is Δx 23 In this case, the position and orientation of the object at time t3, estimated by the 2D tracking unit 302, are "x1 + Δx 12 +Δx 23 It is expressed as ".

[0157] Displacement “Δx 12 " and "Δx 23Since there is an estimation error in each of these, the error in the position and orientation of the object estimated by the 2D tracking unit 302 increases each time the displacement detected by the 2D tracking unit 302 is cumulatively added to the position and orientation "x1". On the other hand, the processing time required in the 2D tracking unit 302 is shorter than the processing time required in the matching unit 301 and the 3D tracking unit 303.

[0158] Therefore, the position and orientation of the object estimated by the 2D tracking unit 302 can be corrected using at least one of the position and orientation of the object estimated by the matching unit 301 and the position and orientation of the object estimated by the 3D tracking unit 303, thereby achieving both faster tracking and improved estimation accuracy.

[0159] Specifically, the T of the 2D tracking unit 302 2-1 The 3D tracking unit 303 uses the image data and shape data from the same period used in the first processing step. 3-1 The position and orientation of the object estimated by the first processing, and the above T 2-1 The difference between the position and orientation of the object estimated by the first processing is considered as the error in the position and orientation of the object estimated by the 2D tracking unit 302. Then, T 3-1 When the first processing is completed, the processing that is in progress in the 2D tracking unit 302 (for example, T 2-2 The error may be corrected in the second processing step or in the processing step following the said processing step.

[0160] Similarly, the matching unit 301 is T i-1 In the first processing step, the T of the 2D tracking unit 302 2-1 When the position and orientation of an object are calculated (estimated) from the image data used in the first processing step, T i-1 The position and orientation calculated (estimated) by the first processing, and the above T 2-1 The difference between the position and orientation of the object estimated by the first processing is considered as the error in the position and orientation of the object estimated by the 2D tracking unit 302. Then, T i-1When the first processing is completed, the processing that is in progress in the 2D tracking unit 302 or the processing that will be performed next (for example, T 2-9 The error may be corrected in the (1st) processing step.

[0161] Furthermore, the matching unit 301 is T i-1 In the first processing step, the T of the 3D tracking unit 303 3-1 When the position and orientation of an object are calculated (estimated) from the shape data used in the first processing step, T i-1 The position and orientation detected by the first processing, and the above T 3-1 The difference between the position and orientation of the object estimated by the first processing is considered as the error in the position and orientation of the object estimated by the 3D tracking unit 303. Then, T i-1 When the first processing is completed, the error may be corrected in the processing currently underway in the 3D tracking unit 303 or in the processing that follows said processing.

[0162] The position and orientation of the object estimated by the 2D tracking unit 302 are output to the robot control unit 100. The robot control unit 100, as part of the control device 1000, controls, for example, the drive unit 311 of the robot 3 based on the estimated position and orientation of the object.

[0163] The robot control unit 100 does not have to be part of the control device 1000, and may be configured separately from the control device 1000. For example, the robot control unit 100 may be a different processing block or a physically implemented processing circuit from the control device 1000. For example, when the 2D tracking unit 302 outputs the position and orientation of an object calculated (estimated) by the 2D tracking unit 302 through the above-described processing, the control device 1000 may generate a control signal for controlling the robot 3 (the drive unit 311 of the robot 3) based on the position and orientation of the object output from the 2D tracking unit 302. The control device 1000 may output the generated control signal to the robot control unit 100. The robot control unit 100 may generate a drive signal for driving the drive unit 311 of the robot 3 based on the control signal output from the control device 1000. The robot control unit 100 may drive the drive unit 311 of the robot 3 based on the generated drive signal. The control device 1000 may have an output unit (not shown) that outputs the generated control signal to the robot control unit 100. The control device 1000 may generate control signals for controlling the robot 3 (the drive unit 311 of the robot 3) at predetermined time intervals based on the position and orientation of the object output from the 2D tracking unit 302 at predetermined time intervals, as described above. The control device 1000 may output the generated control signals to the robot control unit 100 at predetermined time intervals. The robot control unit 100 may generate drive signals for driving the drive unit 311 of the robot 3 at predetermined time intervals based on the control signals output from the control device 1000. The robot control unit 100 may drive the drive unit 311 of the robot 3 based on the generated drive signals. Note that the control signals for controlling the robot 3 described above may be generated by the tracking unit 300 (2D tracking unit 302) of the control device 1000. The tracking unit 300 may generate control signals for controlling the robot 3 (the drive unit 311 of the robot 3) based on the calculated (estimated) position and orientation of the object. The tracking unit 300 may output the generated control signals to the robot control unit 100.

[0164] The tracking unit 300 may receive either image data or shape data as input, but may not receive the other. The tracking unit 300 may include either a 2D tracking unit 302 or a 3D tracking unit 303, but may not include the other. In this case, either the position and orientation of the object estimated by the 2D tracking unit 302 or the position and orientation of the object estimated by the 3D tracking unit 303 may be output to the robot control unit 100. Furthermore, the position and orientation of the object estimated by the 2D tracking unit 302 may not be corrected using the position and orientation of the object estimated by the 3D tracking unit 303. Alternatively, the position and orientation of the object estimated by the 3D tracking unit 303 may be corrected using the position and orientation of the object estimated by the 2D tracking unit 302. In addition, the tracking unit 300 may select the data to be used for tracking processing regardless of the matching rate comparison result by the matching processing unit 200. In this case, the matching unit 301 of the tracking unit 300 may, for example, compare the matching rate when the position and orientation of the object are calculated (estimated) using image data and shape data output from the detection device 320 with the matching rate when the position and orientation of the object are calculated (estimated) using image data and shape data output from the detection device 330, and select (in other words, switch) the data to be used for tracking processing based on the result of the comparison.

[0165] Alternatively, the tracking unit 300 may always perform tracking processing using image data and shape data output from the detection device 330. Alternatively, the tracking unit 300 may always perform tracking processing using image data and shape data output from the detection device 320. In other words, the tracking processing described above can be performed using image data and shape data output from only one of the detection devices 320 and 330. In this case, for example, the robot 3 may have only one of the detection devices 320 and 330. For example, if the robot 3 has only one of the detection devices 320 and 330, the tracking processing described above may be performed using only the image data output from that one detection device, or using only the shape data output from that one detection device.

[0166] The robot control unit 100 does not have to be part of the control device 1000, and may be configured separately from the control device 1000. For example, the robot control unit 100 may be a different processing block or a physically implemented processing circuit from the control device 1000. For example, when the 3D tracking unit 303 outputs the position and orientation of an object calculated (estimated) by the 2D tracking unit 302 through the above-described processing, the control device 1000 may generate a control signal for controlling the robot 3 (the drive unit 311 of the robot 3) based on the position and orientation of the object output from the 3D tracking unit 303. The control device 1000 may output the generated control signal to the robot control unit 100. The robot control unit 100 may generate a drive signal for driving the drive unit 311 of the robot 3 based on the control signal output from the control device 1000. The robot control unit 100 may drive the drive unit 311 of the robot 3 based on the generated drive signal. The control device 1000 may have an output unit (not shown) that outputs the generated control signal to the robot control unit 100. The control device 1000 may generate control signals for controlling the robot 3 (the drive unit 311 of the robot 3) at predetermined time intervals based on the position and orientation of the object output from the 3D tracking unit 303 at predetermined time intervals, as described above. The control device 1000 may output the generated control signals to the robot control unit 100 at predetermined time intervals. The robot control unit 100 may generate drive signals for driving the drive unit 311 of the robot 3 at predetermined time intervals based on the control signals output from the control device 1000. The robot control unit 100 may drive the drive unit 311 of the robot 3 based on the generated drive signals. Note that the control signals for controlling the robot 3 described above may be generated by the tracking unit 300 (3D tracking unit 303) of the control device 1000. The tracking unit 300 may generate control signals for controlling the robot 3 (the drive unit 311 of the robot 3) based on the calculated (estimated) position and orientation of the object. The tracking unit 300 may output the generated control signals to the robot control unit 100.

[0167] The tracking process described above is merely an example and is not limited thereto. In other words, the control device 1000 may perform a well-known tracking process instead of the tracking process described above. Furthermore, the control device 1000 does not have to perform the tracking process described above. In this case, the control device 1000 does not have to include the tracking unit 300 described above, but only needs to include the matching processing unit 200 described above. The control device 1000 may perform a matching process using at least one of the image data and shape data at predetermined time intervals, and control, for example, the drive unit 311 of the robot 3 based on the calculated (estimated) position and orientation of the object.

[0168] (Operation of each robot) The operation of robots 1, 2, and 3 will be explained with reference to the flowchart in Figure 14. Here, we will mainly explain robot 3, and explanations for robots 1 and 2 that are common to robot 3 will be omitted as appropriate.

[0169] Here, the belt conveyor temporarily stops after transporting the circuit board T within the driveable range of, for example, the robot arm 110 of robot 1. Then, after solder dispensed from the dispenser 40 is placed on a predetermined portion of the stopped circuit board T, the belt conveyor resumes driving to transport the circuit board T. The belt conveyor temporarily stops again after transporting the circuit board T within the driveable range of, for example, the robot arm 210 of robot 2. Then, after elements held by the holding device 50 are placed on a predetermined portion of the stopped circuit board T, the belt conveyor resumes driving to transport the circuit board T. The belt conveyor temporarily stops again after transporting the circuit board T within the driveable range of, for example, the robot arm 310 of robot 3. Then, after processing light L from the light irradiation device 60 is irradiated onto a predetermined portion of the stopped circuit board T, the belt conveyor resumes driving to transport the circuit board T.

[0170] Robot 3 First, the control device 1000 may perform calibration of the light irradiation device 60 before performing the following steps S131 to S138.

[0171] As a prerequisite, the detection device 330 and the light illuminator 60 are mounted on the robot arm 310 in such a positional relationship that a part of the light illuminator 60 (for example, its tip) is within the field of view of the cameras 31 and 32 of the detection device 330, respectively.

[0172] As calibration of the light irradiation device 60, the control device 1000 performs the matching process described above using shape data including a part of the light irradiation device 60 output from the detection device 330 and CAD data of the light irradiation device 60, and pre-calculates the position and orientation of the light irradiation device 60 (for example, the position and orientation of the tip of the light irradiation device 60 included in the field of view of the cameras 31 and 32 of the detection device 330). In other words, the control device 1000 pre-calculates the position and orientation of the light irradiation device 60 in the coordinate system of the robot arm 310 based on the shape data of at least a part of the light irradiation device 60.

[0173] The control device 1000 may, as a calibration of the light irradiation device 60, determine the correspondence between the coordinate system of the detection device 330 and the coordinate system of the light irradiation device 60 based on at least some of the shape data of the light irradiation device 60. Then, based on this correspondence between the coordinate system of the detection device 330 and the coordinate system of the light irradiation device 60, and the previously determined correspondence between the coordinate system of the detection device 330 and the coordinate system of the robot arm 310, the control device 1000 may calculate the position and orientation of the light irradiation device 60 in the coordinate system of the robot arm 310. The control device 1000 does not need to calculate the position and orientation of the light irradiation device 60 in the coordinate system of the robot arm 310 as a calibration of the light irradiation device 60, and may simply calculate the correspondence between the coordinate system of the detection device 330 and the coordinate system of the light irradiation device 60. The correspondence between the coordinate system of the detection device 330 and the coordinate system of the light irradiation device 60 may be a transformation matrix between the coordinate system of the detection device 330 and the coordinate system of the light irradiation device 60. Furthermore, if the light irradiation device 60 is calibrated, the control device 1000 may, for example, control the drive unit 311 in step S132 to move the robot arm 310 based on the position and orientation of the object (e.g., circuit board T) calculated in step S131 and the calibration result of the light irradiation device 60. Alternatively, the control device 1000 may, for example, control the drive unit 311 in step S135 to move the robot arm 310 based on the position and orientation of the object (e.g., element) calculated in step S134 and the calibration result of the light irradiation device 60. The calibration result of the light irradiation device 60 may, for example, be the position and orientation of the light irradiation device 60 in the coordinate system of the robot arm 310, or the correspondence between the coordinate system of the detection device 330 and the coordinate system of the light irradiation device 60.

[0174] As described above, the robot control unit 100 does not have to be part of the control device 1000, and may be configured separately from the control device 1000. In this case, the control device 1000 may generate a control signal for controlling the robot arm 310 (drive unit 311) based on the calculated position and orientation of the object and the calibration results of the light irradiation device 60. The control device 1000 may output the generated control signal to the robot control unit 100. The robot control unit 100 may generate a drive signal for driving the drive unit 311 based on the control signal output from the control device 1000. The robot control unit 100 may drive the drive unit 311 based on the generated drive signal.

[0175] Markers may be provided on a portion of the light illuminator 60 that is included within the field of view of each of the cameras 31 and 32 of the detection device 330. In this case, the control device 1000 may perform the above-described calibration based, for example, on shape data including the markers output from the detection device 330.

[0176] Furthermore, the control device 1000 may perform calibration of the light irradiation device 60 by performing a matching process using not only shape data, but also image data output from the detection device 330 and CAD data of the light irradiation device 60. In addition, as described above, the control device 1000 may use not only CAD data, but also shape data or image data of the light irradiation device 60 that have been acquired in advance in the matching process.

[0177] Furthermore, the control device 1000 may use shape data or image data output from the detection device 320, rather than being limited to using the detection device 330 for calibration of the light irradiation device 60. In this case, it is assumed that the detection device 320 and the light irradiation device 60 are mounted on the robot arm 310 in such a positional relationship that a part of the light irradiation device 60 is within the field of view of the cameras 21 and 22 of the detection device 320, respectively.

[0178] In the following steps S131 to S138, the position and orientation of the light irradiator 60 relative to the detection device 330 may change, for example, by the light irradiator 60 coming into contact with a predetermined object. In this case, the control device 1000 can detect the change in the position and orientation of the light irradiator 60 relative to the detection device 330 based on a partial change in the light irradiator 60 in the image data or shape data output from the detection device 330 (for example, a partial change in the light irradiator 60 in the image). If a change in the position and orientation of the light irradiator 60 relative to the detection device 330 is detected, the control device 1000 may perform calibration.

[0179] The control device 1000, which controls the robot 3, calculates (estimates) the position and orientation of the circuit board T as an example of an object (step S131). In the process of step S131, the control device 1000 calculates the initial position and orientation of the circuit board T (i.e., the initial position and orientation) by the matching process of the matching unit 301. Furthermore, the control device 1000 calculates the position of each solder pad formed on the circuit board T. The control device 1000 calculates the position and orientation of each solder pad on the circuit board T based on the Gerber data of the circuit board T (i.e., the design data of the circuit board T). More specifically, the control device 1000 calculates the position and orientation of each solder pad on the circuit board T from the calculated initial position and orientation of the circuit board T and the positional relationship of each solder pad on the circuit board T. In addition, the Gerber data of the circuit board T contains data regarding the order in which elements are mounted on each solder pad, and the control device 1000 identifies the mounting order for each solder pad based on the Gerber data. Furthermore, the control device 1000 does not need to calculate the initial position and orientation of the circuit board T, and may calculate the position and orientation of any object usable for the matching process described above, such as a cross mark formed on the circuit board T, or an AR marker placed near the circuit board T or the circuit layout T. Furthermore, the control device 1000 does not need to identify the position and orientation of each solder pad or the mounting order based on Gerber data, and may identify them using other design data of the circuit board T (e.g., CAD data), or using information input by the user via an interface not shown. Furthermore, the control device 1000 does not need to calculate the position and orientation of each solder pad on the circuit board T, and may calculate, for example, the position and orientation of the area to be irradiated with processing light L, such as the element itself, the area where the element is placed, or the area where solder is placed, or the area near that area.

[0180] Next, the control device 1000 controls the drive unit 311 to move the robot arm 310 so that the detection devices 320 and 330 (and furthermore, the light irradiation device 60) move closer to the circuit board T (step 132). In step S132, the control device 1000 controls the drive unit 311 of the robot arm 310 so that the element placed on the first solder pad to be mounted (for example, a chip LED with two electrodes) comes into the field of view of at least one of the detection devices 320 and 330. More specifically, the control device 1000 uses the initial position and orientation of the circuit board T and the position and orientation information of the first solder pad to be mounted calculated (estimated) in step S131 to control the drive unit 311 to move the robot arm 310 based on the position and orientation of the circuit board T output from the 2D tracking unit 302 of the tracking unit 300 at predetermined time intervals.

[0181] Next, the control device 1000 determines whether the element placed on the first solder pad to be mounted is within the field of view of at least one of the detection devices 320 and 330 (step S133). In the process of S133, the control device 1000 determines whether the detection devices 320 and 330 are in the desired position and orientation relative to the first solder pad to be mounted, based on the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals and the position and orientation information of the first solder pad to be mounted calculated in step S131. At this time, if the detection devices 320 and 330 are in the desired position and orientation relative to the first solder pad to be mounted, the control device 1000 determines that the element placed on the solder pad is within the field of view of at least one of the detection devices 320 and 330. Furthermore, the control device 1000 is not limited to the above determination method. It may also determine whether the position and orientation information output from the 2D tracking unit 302 at predetermined time intervals includes position and orientation information of the element placed on the first solder pad, or it may determine whether the image data and shape data generated by at least one of the detection devices 320 and 330 include information about the element.

[0182] In step S133, if it is determined that the element placed on the first solder pad to be mounted is not within the field of view of at least one of the detection devices 320 and 330 (step S133: No), the control device 1000 controls the drive unit 311 to continue moving the robot arm 310 based on the position and orientation information of the first solder pad to be mounted calculated (estimated) in step S131 and the position and orientation information of the circuit board T output from the 2D tracking unit 302 of the tracking unit 300 at predetermined time intervals. In other words, the process in step S132 is performed until it is determined that the element placed on the first solder pad to be mounted is within the field of view of at least one of the detection devices 320 and 330.

[0183] On the other hand, in the process of step S133, if it is determined that the element placed on the first solder pad to be mounted has entered the field of view of at least one of the detection device 320 and the detection device 330 (step S133: Yes), the control device 1000 calculates (estimates) the position and orientation of the element placed on the first solder pad to be mounted (step S134). In the process of step S134, the control device 1000 calculates (estimates) the initial position and orientation (initial position and orientation) of the element placed on the first solder pad to be mounted by the matching process of the matching unit 301. Note that the control device 1000 does not have to calculate (estimate) the initial position and orientation of the element, and may instead calculate (estimate) the position and orientation of any object that can be used for the matching process described above, such as solder pads and cross marks formed on the circuit board T, or AR markers and solder placed on the circuit board T.

[0184] Next, the control device 1000 moves the robot arm 310 by controlling the drive unit 311 so that the position and orientation of the light irradiation device 60 are in a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad (step S135). In the process of step S135, the control device 1000 uses the initial position and orientation information of the element calculated (estimated) in the process of step S134 to control the drive unit 311 based on the position and orientation of the element output from the 2D tracking unit 302 of the tracking unit 300 at predetermined time intervals, thereby moving the robot arm 310. In other words, the control device 1000 moves the robot arm 310 by controlling the drive unit 311 so that the light irradiation device 60 (detection devices 320 and 330) approaches the element placed on the first solder pad of the circuit board T.

[0185] Next, the control device 1000 determines whether the position and orientation of the light irradiation device 60 are in a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad (step S136). In the process of step S136, the control device 1000 determines whether the position and orientation of the light irradiation device 60 relative to the element is in a desired position and orientation, for example, based on information on the position and orientation of the element output from the 2D tracking unit 302 at predetermined time intervals. At this time, if the position and orientation of the light irradiation device 60 relative to the element is in a desired position and orientation, the control device 1000 determines that the position and orientation of the light irradiation device 60 are in a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad.

[0186] In step S136, if it is determined that the position and orientation of the light irradiation device 60 are not the desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad (step S136: No), the control device 1000 controls the drive unit 311 to move the robot arm 310 by determining that the position and orientation of the element output from the 2D tracking unit 302 at predetermined time intervals, so that the light irradiation device 60 approaches the element placed on the first solder pad. In other words, the process in step S135 is performed until it is determined that the position and orientation are the desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad.

[0187] On the other hand, if the position and orientation of the light irradiation device 60 are determined to be a desired position and orientation that allows the processing light L to melt the solder placed on the first solder pad, the control device 1000 controls the light irradiation device 60 to irradiate the electrodes of the element placed on the solder pad, i.e., the two electrodes of the chip LED, with the processing light L so as to melt the solder placed on the solder pad to be mounted first (step S137). As a result, the solder placed on the solder pad melts and the element is soldered to the circuit board T (the solder pad to be mounted first). Here, two specific embodiments of the process in step S137 can be cited as examples.

[0188] In other words, the first embodiment is such that, as shown in Figures 15(a) and (b), for example, when the robot arm 310 is driven by the drive unit 311, that is, when the light irradiation device 60 and the detection devices 320 and 330 are moved by the robot arm 310, the processing light L is irradiated onto the electrodes of the chip LED. In this embodiment, the control device 1000 controls the drive unit 311 so that the light irradiation device 60 and the detection devices 320 and 330 are moved, and may also control the orientation of the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the light irradiation device 60 moved by the robot arm 310 is maintained on one electrode of the chip LED (for example, so that the processing light L from the light irradiation device 60 is irradiated onto the same position on one electrode of the chip LED). For example, the control device 1000 may control the drive unit 311 so that the light irradiation device 60 and the detection devices 320 and 330 are moved, while also controlling the orientation of the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the light irradiation device 60, which is moved by the robot arm 310, is maintained at the same position on one electrode of the chip LED (in other words, so that the processing light L from the light irradiation device 60 continues to irradiate the same position on one electrode of the chip LED for a predetermined time). Note that "same position" is a concept that also includes the possibility that the irradiation position of the processing light L may fluctuate to an extremely small extent at the location where the processing light L should be irradiated, to the extent that it does not affect the melting of the solder.

[0189] For example, as shown in Figures 15(a) and (b), when the light irradiation device 60 is being moved from right to left on the plane of the paper by the robot arm 310, the control device 1000 may control the orientation of the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the moving light irradiation device 60 is maintained on one electrode of the chip LED.

[0190] In the first embodiment described above, for example, the control device 1000 may control the orientation of the galvanometer mirror 61 so that processing light L from the light irradiation device 60 moved by the robot arm 310 is irradiated onto the work object (for example, so that it is irradiated onto the same position on the work object) based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 320 and 330 moved by the robot arm 310.

[0191] In a second embodiment, the control device 1000 controls the drive unit 311 of the robot arm 310, which was driven in step S135, to stop driving, and after the drive unit 311 stops, processing light L is irradiated onto the electrode of the chip LED. In this embodiment, the control device 1000 controls the drive unit 311 to stop driving. After the drive unit 311 stops driving, the control device 1000 may control the orientation of the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the light irradiation device 60 is maintained on one electrode of the chip LED. For example, the control device 1000 controls the drive unit 311 to stop driving. The control device 1000 may, after the drive unit 311 stops, control the orientation of the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the light irradiation device 60 is maintained at the same position on one electrode of the chip LED (so that the processing light L from the light irradiation device 60 continues to irradiate the same position on one electrode of the chip LED for a predetermined time).

[0192] In the second embodiment described above, for example, the control device 1000 may control the orientation of the galvanometer mirror 61 so that processing light L from the light irradiation device 60, which is displaced together with at least one of the detection devices 320 and 330, is irradiated onto the work object (for example, so that it is irradiated onto the same position on the work object), based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 320 and 330 after the drive unit 311 has stopped moving.

[0193] In the first embodiment, for example, when mounting chip LEDs at high speed to multiple solder pads on a circuit board T, the control device 1000 controls the drive unit 311 of the robot arm 310 and the light irradiation device 60 so as to move the light irradiation device 60 (detection devices 320 and 330) from the first solder pad to be mounted to the second solder pad, and sequentially irradiate the electrodes of the chip LEDs placed on the first solder pad (in other words, the first position) and the electrodes of the chip LEDs placed on the second solder pad (in other words, the second position) with processing light L. In this case, in parallel with step S137, the control device 1000 moves the light irradiation device 60 (detection devices 320 and 330) from the first solder pad to be mounted to the second solder pad based on the position and orientation of the second solder pad to be mounted calculated (estimated) in step S131.

[0194] First, the control device 1000 controls the orientation of the galvano mirror 61 of the light irradiation device 60 (detection devices 320 and 330) so that processing light L is irradiated onto the electrodes of the chip LED placed on the first solder pad to be mounted for a predetermined time from the light irradiation device 60 (detection devices 320 and 330) which moves (displaces) relative to the electrodes of the chip LED. More specifically, as the light irradiation device 60 and detection devices 320 and 330 move by the robot arm 310, the control device 1000 gradually changes the orientation of the galvano mirror 61 so that the irradiation position of the processing light L is maintained on one electrode of the chip LED (for example, the same position on one electrode of the chip LED) based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals. At this time, the control device 1000 can recognize changes in the position and orientation of one electrode of the chip LED relative to the light irradiation device 60 based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, and can control the orientation of the galvano mirror 61 so that the irradiation position of the processing light L is maintained on one electrode of the chip LED (for example, the same position on one electrode of the chip LED). Note that the position and orientation of the chip LED in the image shown by the image data output sequentially from at least one of the detection devices 320 and 330, and the position and orientation of the chip LED in the point cloud shown by the shape data, change over time due to the relative displacement between the chip LED and the light irradiation device 60 (detection devices 320 and 330).

[0195] After the control device 1000 has finished irradiating one electrode of the chip LED with processing light L for a predetermined time, it then gradually changes the orientation of the galvano mirror 61 so that the irradiation position of the processing light L is maintained on the other electrode of the chip LED (for example, the same position on the other electrode of the chip LED) based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, as the light irradiation device 60 and detection devices 320 and 330 continue to move by the robot arm 310. Then, after the control device 1000 has finished irradiating the other electrode of the chip LED with processing light L for a predetermined time, it repeats steps S132 to S137 described above to similarly irradiate the electrodes of the chip LEDs placed on the second solder pad to be mounted with processing light L.

[0196] For example, the control device 1000 may control the orientation of the galvanometer mirror 61 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330, so that the irradiation position of the processing light L from the light irradiation device 60, which is displaced together with the displacement of at least one of the detection devices 320 and 330, is maintained at a first position, and then at a second position different from the first position.

[0197] For example, the control device 1000 may control the drive unit 311 to move the light irradiation device 60 and the detection devices 320 and 330, and based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 320 and 330 moved by the robot arm 310, it may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L from the light irradiation device 60 moved by the robot arm 310 is maintained at a first position, and then maintained at a second position different from the first position.

[0198] In the second embodiment, inertial and elastic forces act on the light irradiation device 60 and detection devices 320 and 330 provided on the robot arm 310. For example, the control device 1000 controls the drive unit 311 of the robot arm 310, which was driven in step S135, to stop driving. After the drive unit 311 is stopped, the relative position between the light irradiation device 60 (detection devices 320 and 330) and the chip LED fluctuates to a greater or lesser extent over time due to displacement of the light irradiation device 60 and detection devices 320 and 330 due to vibration or the like. For this reason, the control device 1000 controls the galvanometer mirror 61 of the light irradiation device 60 so that the irradiation position of the processing light L from the light irradiation device 60 is maintained on one electrode of the chip LED (in other words, the first position) for a predetermined time, even if the light irradiation device 60 (detection devices 320 and 330) is displaced due to vibration or the like. Next, the control device 1000 controls the galvanometer mirror 61 of the light irradiation device 60 so that even if the light irradiation device 60 (detection devices 320 and 330) continues to be displaced due to vibration or the like, the irradiation position of the processing light L from the light irradiation device 60 is maintained on the other electrode (in other words, the second position) of the chip LED for a predetermined time.

[0199] More specifically, the control device 1000 first changes the orientation of the galvano mirror 61 over time based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals (due to displacement such as vibration of the light irradiation device 60), so that the irradiation position of the processing light L is maintained on one electrode of the chip LED (for example, the same position on one electrode of the chip LED). Since the control device 1000 can recognize changes in the position and orientation of one electrode of the chip LED relative to the light irradiation device 60, it can control the orientation of the galvano mirror 61 so that the irradiation position of the processing light L is maintained on one electrode of the chip LED (for example, the same position on one electrode of the chip LED). Note that the position and orientation of the chip LED in the image shown by the image data output sequentially from at least one of the detection devices 320 and 330, and the position and orientation of the chip LED in the point cloud shown by the shape data, change over time due to the relative displacement between the chip LED and the light irradiation device 60 (detection devices 320 and 330).

[0200] After the control device 1000 has finished irradiating one electrode of the chip LED with processing light L for a predetermined time, the control device 1000 then changes the orientation of the galvanometer mirror 61 over time, based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals (due to displacement such as vibration of the light irradiation device 60), so that the irradiation position of the processing light L is maintained on the other electrode of the chip LED (for example, the same position on the other electrode of the chip LED).

[0201] Furthermore, when processing light L is irradiated onto one or the other electrode of the chip LED, the position of the chip LED relative to the solder pad may change over time due to the surface tension of the molten solder, etc. In this case, the control device 1000 gradually changes the orientation of the galvano mirror based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, so that the irradiation position of the processing light L is maintained on one or the other electrode of the chip LED. At this time, the control device 1000 can recognize the change in the position and orientation of one or the other electrode of the chip LED relative to the light irradiation device 60 based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, and can control the orientation of the galvano mirror so that the irradiation position of the processing light L is maintained on one electrode of the chip LED.

[0202] Furthermore, not only when the drive unit 311 of the driven robot arm 310 is stopped, but also when the robot arm 310 accelerates or decelerates, or after acceleration or deceleration, the light irradiation device 60 and detection devices 320 and 330 provided on the robot arm 310 are displaced by vibration, etc., and the relative position between the light irradiation device 60 (detection devices 320 and 330) and the area to be irradiated with processing light L (for example, the electrodes of a chip LED) changes over time. However, even in this case, the control device 1000 can recognize the temporal change in the position and orientation of the area to be irradiated with processing light L relative to the light irradiation device 60 based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, and can control the orientation of the galvanometer mirror so that the irradiation position of the processing light L is maintained at the area to be irradiated with processing light L.

[0203] In step S135, the control device 1000 may control the light irradiation device 60 to change the spot size and intensity of the processing light L when irradiating the area to be irradiated with the processing light L (for example, the electrode of the chip LED). If the intensity of the processing light L is changed by changing the intensity of the light emitted from a light source (not shown), and the light source (not shown) is located outside the light irradiation device 60, then the external light source (not shown) should be controlled.

[0204] For example, the control device 1000 may control the orientation of the galvanometer mirror 61 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330, so that the irradiation position of the processing light L from the light irradiation device 60, which is displaced together with the displacement of at least one of the detection devices 320 and 330, is maintained at a first position, and then at a second position different from the first position.

[0205] For example, the control device 1000 may control the drive unit 311 to stop driving the drive unit 311. The control device 1000 may, based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 320 and 330 after the drive unit 311 has stopped driving, control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L from the light irradiation device 60, which is displaced together with the displacement of at least one of the detection devices 320 and 330, is maintained at a first position, and then maintained at a second position different from the first position.

[0206] Furthermore, when the light irradiation device 60, etc., is being moved by the robot arm 310 and processing light L is being irradiated, the control device 1000 may control at least one of the posture and position of the light irradiation device 60, or the orientation of the galvanometer mirror 61, etc., based on prediction results that predict the movement of the robot arm 310, in addition to the results of the tracking process.

[0207] In the processing of steps S131 to S137 described above, the control device 1000 may, for example, control the drive unit 311 of the robot arm 310 so that the light irradiation device 60 and the detection devices 320 and 330 approach the circuit board T based on at least one of the image data and shape data generated by at least one of the detection devices 320 and 330, and when the light irradiation device 60 and the detection devices 320 and 330 approach the circuit board T to a predetermined distance, control the orientation of the galvanometer mirror 61 so that the processing light L from the light irradiation device 60, which is displaced together with at least one of the detection devices 320 and 330, is irradiated onto the area where the processing light L should be irradiated as part of the object (for example, a solder pad formed on the circuit board T, or elements and solder placed on the circuit board T) based on the at least one of the data that changes with the displacement of the detection devices 320 and 330 (for example, so that it is irradiated onto the same position on the area where the processing light L should be irradiated).

[0208] After the processing in step S137, the control device 1000 performs a quality inspection of the soldered elements and solder based on at least one of the image data and shape data output from at least one of the detection devices 320 and 330 (step S138). Examples of inspection items include misalignment of the elements relative to the solder pad and lifting of the element electrodes relative to the solder pad (the so-called Manhattan phenomenon where the element electrodes separate from the solder pad). For example, when performing a quality inspection of misalignment of elements relative to the solder pad, the control device 1000 recognizes the solder pad and elements in the image shown in the image data based on the image data output from at least one of the detection devices 320 and 330, and detects the misalignment of the elements relative to the solder pad. For example, the control device 1000 may determine that the element is good (good quality) if at least a part of the element electrodes overlaps the solder pad, and may determine that it is of poor quality if at least a part of the element electrodes do not overlap the solder pad. Furthermore, the control device 1000 may detect misalignment of the element relative to the solder pad not only based on image data, but also based on shape data output from at least one of the detection devices 320 and 330, or based on both image data and shape data. The control device 1000 does not necessarily need to determine whether at least a portion of the element's electrodes overlaps the solder pad as a quality defect. For example, the control device 1000 may determine quality based on the area where the element's electrodes overlap the solder pad.

[0209] The control device 1000 may also display at least one piece of information used for the solder quality inspection in step S138 on a display device (not shown), including image data, shape data, and the results of the solder quality inspection.

[0210] The control device 1000 may perform machine learning using an existing method, with training data obtained by associating at least one piece of information from the image data and shape data used in the processes of steps S131 to S137 described above with the soldering quality determined in the process of step S138 described above. In this case, the control device 1000 may use the results of machine learning to control each device of the robot 3 (for example, position and attitude control of the light irradiation device 60 or control of the light irradiation device 60). Here, the control of the light irradiation device 60 includes setting conditions for the processing light L irradiated from the light irradiation device 60 (for example, at least one of the intensity of the processing light L, the spot size of the processing light L, the irradiation time of the processing light L, and the irradiation range of the processing light L). The control device 1000 may also use the results of machine learning to control at least one of the control of each device of the robot 1 and the control of each device of the robot 2.

[0211] After the processing in step S138, the control device 1000 starts moving the light irradiation device 60 (detection devices 320 and 330) to the second solder pad to be mounted, based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals in the processing of step S137 or step S136, and the position and orientation of the second solder pad to be mounted calculated in step S131, and repeats the processing in steps S133 to S138 described above. Note that when the control device 1000 starts moving the light irradiation device 60 (detection devices 320 and 330) to the second solder pad to be mounted, it may perform the processing in step S131 described above before executing the processing in steps S133 to S138 described above. The control device 1000 repeats the above steps until all elements (e.g., chip LEDs, etc.) placed on each solder pad of the circuit board T have been mounted (i.e., soldered).

[0212] If there is only one component placed on the solder pad of the circuit board T, the control device 1000 may, after processing in step S136, control the drive unit 311, etc., so that the robot arm 310, etc., is in a predetermined initial position.

[0213] As described above in each step (particularly the first and second aspects of step S137), the control device 1000 can recognize the position and orientation of the object at predetermined time intervals through tracking processing. As a result, the control device 1000 can irradiate the processing light L to the desired position of the object (in other words, the location where the processing light L should be irradiated) even if the relative position between the object and the light irradiation device 60 (light detection devices 320 and 330) changes (displaces) over time, by controlling at least one of the drive unit 311 of the robot arm 310 and the galvanometer mirror 61.

[0214] In addition, in the first and second embodiments of step S137, the control device 1000 may control at least one of the drive unit 311 of the robot arm 310 and the galvanometer mirror 61 so that the irradiation position of the processing light L changes over time over a wide area of ​​the area to be irradiated with the processing light L (for example, the entire area to be irradiated with the processing light L). For example, when the control device 1000 irradiates the electrode of an element as the area to be irradiated with the processing light L, it may control the orientation of the mirror of the galvanometer mirror 61 so that the irradiation position of the processing light L changes over time over the entire electrode while recognizing the position and orientation of the element (the electrode) at predetermined time intervals by tracking processing. In other words, the orientation of the mirror of the galvanometer mirror 61 may be controlled so that the processing light L is scanned over the electrode. By such control, localized heat input to the electrode (in other words, localized heat input to the solder) can be suppressed, and damage to the element due to heat, solder melting failure due to localized heating, and damage to the circuit board due to localized heating can be prevented. In this case, as in the first embodiment of step S137, the robot arm 310 may move the light irradiation device 60 and the detection devices 320 and 330, and the orientation of the galvano mirror may be changed so that the irradiation position of the processing light L changes over time across the entire electrode of one of the chip LEDs, based on the position and orientation of the chip LED (electrode of the chip LED) output from the 2D tracking unit 302 at predetermined time intervals. At this time, the control device 1000 can recognize the temporal change in the position and orientation of one electrode of the chip LED relative to the light irradiation device 60 based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals, and can control the orientation of the galvano mirror so that the irradiation position of the processing light L changes over time across the entire electrode of one of the chip LEDs.

[0215] Furthermore, the position and orientation of the chip LED in the image shown by the image data sequentially output from at least one of the detection devices 320 and 330, and the position and orientation of the chip LED in the point cloud or depth image shown by the shape data, change over time due to the relative displacement between the chip LED and the light irradiation device 60 (detection devices 320 and 330). After the control device 1000 has finished irradiating one electrode of the chip LED with processing light L for a predetermined time, it then changes the orientation of the galvanometer mirror so that the irradiation position of the processing light L changes over time across the entire other electrode of the chip LED, based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals as the light irradiation device 60 and detection devices 320 and 330 continue to move by the robot arm 310. Then, after the control device 1000 has finished irradiating the other electrode of the chip LED with processing light L for a predetermined time, it repeats steps S132 to S137 described above to similarly irradiate the electrode of the chip LED placed on the second solder pad with processing light L.

[0216] Furthermore, for example, the control device 1000 changes the orientation of the galvano mirror over time based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals (due to displacement such as vibration of the light irradiation device 60), so that the irradiation position of the processing light L changes over time across the entire electrode of one of the chip LEDs. Since the control device 1000 can recognize changes in the position and orientation of one electrode of the chip LED relative to the light irradiation device 60, it can control the orientation of the galvano mirror so that the irradiation position of the processing light L changes over time across the entire electrode of one of the chip LEDs. Note that the position and orientation of the chip LED (electrode of the chip LED) in the image shown by the image data output sequentially from at least one of the detection devices 320 and 330, and the position and orientation of the chip LED in the point cloud shown by the shape data, change over time due to the relative displacement between the chip LED and the light irradiation device 60 (detection devices 320 and 330). After the control device 1000 has finished irradiating one electrode of the chip LED with processing light L for a predetermined time, the control device 1000 then changes the orientation of the galvanometer mirror over time, based on the position and orientation of the chip LED output from the 2D tracking unit 302 at predetermined time intervals (due to displacement caused by vibration of the light irradiation device 60, etc.), so that the irradiation position of the processing light L changes over time across the entire other electrode of the chip LED.

[0217] Therefore, the control device 1000 recognizes the temporal changes in the position and orientation of the area to be irradiated with processing light L on the light irradiation device 60, based on the position and orientation of the area to be irradiated with processing light L (for example, the electrodes of the element) output from the 2D tracking unit 302 at predetermined time intervals, and can irradiate the area to be irradiated with processing light L over a desired range (the same position or the entire area).

[0218] As mentioned above, the area to be irradiated with the processing light L as part of the object is not limited to the electrodes of the element, but may also be a solder pad or solder. Even in this case, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L changes over time across the entire area to be irradiated with the processing light L (solder pad or solder). Furthermore, if the area to be irradiated with the processing light L is large, the control device 1000 may drive the drive unit 311 of the robot arm 310 and control the orientation of the galvanometer mirror 61 to scan the area to be irradiated with the processing light L.

[0219] Furthermore, for example, the control device 1000 may alternately irradiate one electrode and the other electrode of an element having multiple electrodes (for example, a chip LED having two electrodes) as the area to be irradiated with processing light L in a time-dependent manner to melt the solder. In this case, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L changes alternately in time between one electrode and the other electrode, while recognizing the position and orientation of the element (one electrode and the other electrode) at predetermined time intervals by tracking processing. In addition, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L changes over time across the entire electrode during the time period when the processing light L is irradiated onto one electrode. Similarly, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L changes over time across the entire electrode during the time period when the processing light L is irradiated onto the other electrode. This type of control can further suppress localized heat input to the electrode (in other words, localized heat input to the solder).

[0220] As mentioned above, the area to be irradiated with the processing light L is not limited to electrodes; it may also be a solder pad or solder. Even in this case, the control device 1000 may control the orientation of the galvanometer mirror 61 so that the irradiation position of the processing light L alternates over time between the first area to be irradiated with the processing light L (one solder pad or one solder) and the second area (the other solder pad or the other solder). Furthermore, if the first area to be irradiated with the processing light L and the second area are far apart, the control device 1000 may scan the processing light L by controlling the orientation of the galvanometer mirror 61 while driving the drive unit 311 of the robot arm 310.

[0221] In step S134 described above, the control device 1000 may perform CAD matching using the shape data output from the detection device 330 and the CAD data related to the element to measure the position and orientation of the element. At this time, the control device 1000 may perform CAD matching after removing data corresponding to, for example, the substrate surface of the circuit board T from the shape data. By configuring it in this way, the time required for CAD matching (in other words, the time required for the processing in step S134) can be shortened. Note that various existing methods can be applied to remove shape data corresponding to, for example, the substrate surface, so a detailed explanation of these methods will be omitted.

[0222] Furthermore, if the solder placed on the solder pad can be melted and the element mounted, the control device 1000 does not need to perform the processes of steps S131 and S133, or steps S135 and S133, described above.

[0223] The control device 1000 may display at least one of the following information used in the processing of steps S131 to S137 above on a display device (not shown): image data, shape data, information on the position and orientation of the circuit board T, information on the position and orientation of each solder pad, information on the order in which components are mounted on each solder pad, and information on the position and orientation of the elements.

[0224] Furthermore, in the processing of steps S131 to S137 described above, the control device 1000 may detect at least one of the solder state (at least one piece of information regarding the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and orientation of the solder) and the area of ​​the solder pad based on at least one of the image data and shape data output from the detection devices 320 and 330. In other words, the control device 1000 may detect information regarding the state of the area to be irradiated with processing light L. The control device 1000 may then use the information regarding the state of the area to be irradiated with processing light L, as detected as described above, to control the conditions of the processing light L irradiated from the light irradiation device 60 (for example, at least one of the intensity of the processing light L, the spot size of the processing light L, the irradiation time of the processing light L, and the irradiation range of the processing light L). In other words, the control device 1000 may determine the conditions of the processing light L based on the detected information regarding the state of the area to be irradiated with processing light L. In addition to the information described above, the information regarding the state of the area to be irradiated with processing light L may also include information regarding elements, solder, and solder pads that can be detected as areas to be irradiated with processing light L, based on at least one of the image data and shape data output from at least one of the detection devices 320 and 330.

[0225] As shown in Figure 16, an air blower and a fume extractor 70 may be placed near the light irradiation device 60, which serves as the end effector of the robot arm 310 (not shown). Alternatively, either the air blower or the fume extractor may be placed, while the other is omitted.

[0226] With this configuration, it is possible to effectively prevent smoke generated when the solder is heated by the processing light L emitted from the light irradiation device 60 from adhering to the optical system of the light irradiation device 60.

[0227] Robot 1 The control device 1000 that controls the robot 1 performs the processes of steps S111 and S112, which correspond to the processes of steps S131 and S132 described above. However, the control device 1000 performs the processes of steps S111 and S112 using the output of at least one of the detection devices 120 and 130 provided by the robot 1.

[0228] The control device 1000 may perform calibration of the dispenser 40 before performing the following steps S111 to S117.

[0229] As a prerequisite, it is assumed that the detection device 130 and the dispenser 40 are mounted on the robot arm 110 in such a positional relationship that a part of the dispenser 40 (for example, its tip) is within the field of view of the cameras 31 and 32 of the detection device 130, which have a configuration similar to that of the detection device 330. For the sake of explanation, the cameras 31 and 32 of the detection device 130 will be referred to as the cameras 31 and 32 of the detection device 130, as an example of a detection device 130 having cameras 31 and 32 similar to those of the detection device 330.

[0230] As a calibration of the dispenser 40, the control device 1000 performs the matching process described above using the shape data including the dispenser 40 output from the detection device 230 and the CAD data of the dispenser 40, and pre-calculates the position and orientation of the dispenser 40 (for example, the position and orientation of the tip of the dispenser 40 included in the field of view of the cameras 31 and 32 of the detection device 130). In other words, the control device 1000 pre-calculates the position and orientation of the dispenser 40 in the coordinate system of the robot arm 110 based on the shape data of at least a part of the dispenser 40.

[0231] The control device 1000 may, as a calibration of the dispenser 40, determine the correspondence between the coordinate system of the detection device 130 and the coordinate system of the dispenser 40 based on the shape data of at least a portion of the dispenser 40. Then, based on this correspondence between the coordinate system of the detection device 130 and the coordinate system of the dispenser 40, and the previously determined correspondence between the coordinate system of the detection device 130 and the coordinate system of the robot arm 110, the control device 1000 may calculate the position and orientation of the dispenser 40 in the coordinate system of the robot arm 110. The control device 1000 does not need to calculate the position and orientation of the dispenser 40 in the coordinate system of the robot arm 110 as a calibration of the dispenser 40, and may simply calculate the correspondence between the coordinate system of the detection device 130 and the coordinate system of the dispenser 40. The correspondence between the coordinate system of the detection device 330 and the coordinate system of the dispenser 40 may be a transformation matrix between the coordinate system of the detection device 330 and the coordinate system of the dispenser 40. Furthermore, if the dispenser 40 is calibrated, the control device 1000 may, for example, control the drive unit 111 in step S112 to move the robot arm 110 based on the position and orientation of the object (e.g., circuit board T) calculated in step S111 and the calibration result of the dispenser 40. Alternatively, the control device 1000 may, for example, control the drive unit 111 in step S115 to move the robot arm 110 based on the position and orientation of the object (e.g., solder pad) calculated in step S114 and the calibration result of the dispenser 40. The calibration result of the dispenser 40 may, for example, be the position and orientation of the dispenser 40 in the coordinate system of the robot arm 110, or the correspondence between the coordinate system of the detection device 130 and the coordinate system of the dispenser 40.

[0232] As described above, the robot control unit 100 does not have to be part of the control device 1000, and may be configured separately from the control device 1000. In this case, the control device 1000 may generate a control signal for controlling the robot arm 110 (drive unit 111) based on the calculated position and orientation of the object and the calibration results of the dispenser 40. The control device 1000 may output the generated control signal to the robot control unit 100. The robot control unit 100 may generate a drive signal for driving the drive unit 111 based on the control signal output from the control device 1000. The robot control unit 100 may drive the drive unit 111 based on the generated drive signal.

[0233] Furthermore, markers may be provided on a portion of the dispenser 40 that is included within the field of view of each of the cameras 31 and 32 of the detection device 130. In this case, the control device 1000 may perform the above-described calibration based, for example, on shape data including the markers output from the detection device 130.

[0234] Furthermore, the control device 1000 may perform calibration of the dispenser 40 by executing a matching process using not only shape data, but also image data output from the detection device 130 and CAD data of the dispenser 40. In addition, as described above, the control device 1000 may use shape data or image data of the dispenser 40 that have been acquired in advance, not limited to CAD data, in the matching process.

[0235] Furthermore, the control device 1000 may use shape data or image data output from the detection device 120, rather than being limited to using the detection device 130 for the calibration of the dispenser 40. In this case, it is assumed that the detection device 120 and the dispenser 40 are mounted on the robot arm 110 in such a positional relationship that a part of the dispenser 40 is within the field of view of the cameras 21 and 22 of the detection device 120, which has the same configuration as the detection device 320.

[0236] In the following steps S111 to S117, the dispenser 40 may come into contact with a predetermined object, causing a change in the position and orientation of the dispenser 40 relative to the detection device 130. In this case, the control device 1000 can detect a change in the position and orientation of the dispenser 40 relative to the detection device 130 based on a partial change in the dispenser 40 in the image data or shape data output from the detection device 130 (for example, a partial change in the dispenser 40 in the image). If a change in the position and orientation of the dispenser 40 relative to the detection device 130 is detected, the control device 1000 may perform calibration.

[0237] The control device 1000 that controls the robot 1 calculates the position and orientation of the circuit board T as an example of an object (step S111). Similar to step S131 described above, in the process of step S111, the control device 1000 calculates the initial position and orientation of the circuit board T (i.e., the initial position and orientation) by the matching process of the matching unit 301. Furthermore, the control device 1000 calculates the position of each solder pad formed on the circuit board T. For example, the control device 1000 calculates the position and orientation of each solder pad on the circuit board T based on the Gerber data of the circuit board T (i.e., the design data of the circuit board T). Based on the Gerber data, the control device 1000 identifies the order in which to mount (in this case, place solder) each solder pad.

[0238] Next, the control device 1000 controls the drive unit 111 to move the robot arm 110 so that the dispenser 40 (detection devices 120 and 130) approaches the circuit board T (step 112). In the process of step S112, similar to step S132 described above, the control device 1000 controls the drive unit 111 of the robot arm 110 so that the first solder pad to be mounted comes into the field of view of at least one of the detection devices 120 and 130.

[0239] Next, the control device 1000 determines whether the solder pad to be first mounted is within the field of view of at least one of the detection devices 120 and 130 (step S113). Similar to step S133 described above, in the process of S113, the control device 1000 determines whether the detection devices 120 and 130 are in the desired position and orientation relative to the solder pad to be first mounted, based on the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals and the position and orientation information of the solder pad to be first mounted calculated in step S111. At this time, if the detection devices 120 and 130 are in the desired position and orientation relative to the solder pad, the control device 1000 determines that the solder pad is within the field of view of at least one of the detection devices 120 and 130.

[0240] In step S113, if it is determined that the solder pad is not within the field of view of at least one of the detection devices 120 and 130 (S113: No), the control device 1000 controls the drive unit 111 to continue moving the robot arm 110 based on the position and orientation information of the solder pad calculated in step S111 and the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals.

[0241] On the other hand, in the process of step S113, if it is determined that the solder pad has entered the field of view of at least one of the detection device 120 and the detection device 130 (step S113: Yes), the control device 1000 calculates the position and orientation of the first solder pad to be mounted (step S114). Similar to step S134 described above, in the process of step S114, the control device 1000 calculates the initial position and orientation (initial position and orientation) of the solder pad by the matching process of the matching unit 301. Note that in the process of step S111 described above, the position and orientation of the solder pad are calculated based on Gerber data, but since Gerber data is design data, there is an error between it and the actual position and orientation of the solder pad on the circuit board T. Therefore, the control device 1000 executes the process of step S114.

[0242] Next, the control device 1000 controls the drive unit 111 so that the position and orientation of the dispenser 40 are in a desired position and orientation that allows solder to be dispensed onto the first solder pad to be mounted, and moves the robot arm 110 (step S115). Similar to step S135 described above, in the process of step S115, the control device 1000 uses the initial position and orientation information of the solder pad calculated in the process of step S114 to control the drive unit 111 based on the position and orientation of the solder pad output from the 2D tracking unit 302 at predetermined time intervals, and moves the robot arm 110. In other words, the control device 1000 controls the drive unit 111 so that the dispenser 40 (detection devices 120 and 130) approaches the first solder pad to be mounted on the circuit board T, and moves the robot arm 110.

[0243] Next, the control device 1000 determines whether the position and orientation of the dispenser 40 are in a desired position and orientation that allows solder to be dispensed onto the first solder pad to be mounted (step S116). In the process of step S116, similar to step S136 described above, the control device 1000 determines whether the position and orientation of the dispenser 40 relative to the solder pad is in a desired position and orientation, for example, based on the position and orientation information of the first solder pad to be mounted output from the 2D tracking unit 302 at predetermined time intervals.

[0244] In step S116, if it is determined that the position and orientation of the dispenser 40 are not the desired position and orientation for dispensing solder to the solder pad (step S116: No), the control device 1000 controls the drive unit 111 to continue moving the robot arm 110 so that the dispenser 40 approaches the solder pad, based on the position and orientation of the solder pad output from the 2D tracking unit 302 at predetermined time intervals. On the other hand, if it is determined that the position and orientation of the dispenser 40 are the desired position and orientation for dispensing solder to the solder pad (step S116: Yes), the control device 1000 controls the dispenser 40 so that solder is placed on at least a portion of the solder pad (step S117). Specifically, for example, the control device 1000 controls the dispenser 40 to dispense solder. Here, the control device 1000 may estimate the area of ​​the solder pad based on the position and orientation of the solder pad output from the 2D tracking unit 302 at predetermined time intervals, and control the amount of solder dispensed from the dispenser 40 according to the area of ​​the solder pad. In this case, the control device 1000 may control the dispenser 40 so that the larger the estimated area of ​​the solder pad, the greater the amount of solder dispensed. This makes it possible to place an appropriate amount of solder on the solder pad.

[0245] Furthermore, at this time, the relative position between the dispenser 40 and the workpiece may change due to, for example, vibration of the robot arm 110. Therefore, the position of the solder pad as the workpiece in the image shown by, for example, image data output sequentially from the detection device 130 may also change over time due to the aforementioned change in relative position.

[0246] Therefore, in order to suppress or eliminate the influence of the relative position fluctuations on the processing of step S117, the control device 1000 controls at least one of the attitude and position of the dispenser 40 based on the results of the tracking process described above.

[0247] For example, the control device 1000 may control the drive unit 111 so that the solder dispensed from the dispenser 40, which is displaced along with the displacement of at least one of the detection devices 120 and 130, is placed on the solder pad, which is the target object, based on at least one of the image data and shape data, which changes with the displacement of at least one of the detection devices 120 and 130.

[0248] For example, the control device 1000 may control the drive unit 111 to stop the drive of the drive unit 111. The control device 1000 may also control the drive unit 111 based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 120 and 130 after the drive of the drive unit 111 has stopped, so that the solder dispensed from the dispenser 40 which displaces together with at least one of the detection devices 120 and 130 is placed on the solder pad which is the target object.

[0249] In the process of steps S111 to S117 described above, the control device 1000 may control the drive unit 111 of the robot arm 110 so that the dispenser 40 and the detection devices 120 and 130 approach the circuit board T based on at least one of the image data and shape data generated by at least one of the detection devices 120 and 130. When the dispenser 40 and the detection devices 120 and 130 approach the circuit board T to a predetermined distance, the control device 1000 may control the drive unit 111 so that the solder dispensed from the dispenser 40, which is displaced together with at least one of the detection devices 120 and 130, is placed on the solder pad, which is the target object, based on the at least one of the data that changes with the displacement of at least one of the detection devices 120 and 130.

[0250] After the processing in step S117, the control device 1000 starts moving the dispenser 40 to the second solder pad to be mounted, based on the position and orientation of the first solder pad to be mounted output from the 2D tracking unit 302 at predetermined time intervals during the processing of step S117 or step S116, and the position and orientation of the second solder pad to be mounted calculated in step S111, and repeats the processing in steps S113 to S117 described above. Note that when the control device 1000 starts moving the dispenser 40 to the second solder pad to be mounted, it may perform the processing in step S111 described above before executing the processing in steps S113 to S117 described above. The control device 1000 repeats the above steps until the placement of solder on each solder pad of the circuit board T is completed.

[0251] If there is only one solder pad on the circuit board T, the control device 1000 may, after processing in step S117, control the drive unit 111, etc., so that the robot arm 110, etc., is in a predetermined initial position.

[0252] The control device 1000 may also display at least one of the following information on a display device (not shown): image data, shape data, position and orientation information of the circuit board T, position and orientation information of each solder pad, and information on the order in which components are mounted on each solder pad.

[0253] Furthermore, if solder can be placed on the solder pad, the control device 1000 does not need to perform the processes described in steps S111 and S113 or steps S115 and S116 above.

[0254] Furthermore, in the processing of steps S115 to S117 described above, the control device 1000 may detect (calculate) the area of ​​the solder pad based on at least one of the image data or shape data output from the detection devices 120 and 130.

[0255] Furthermore, after the processing in step S117, the control device 1000 may detect the state of the solder placed on the solder pad based on at least one of the image data and shape data output from at least one of the detection devices 120 and 130, or based on the matching process. Examples of the solder state to be detected include the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, the position and orientation of the solder, etc. The control device 1000 may then determine the quality of the solder based on the detected solder state.

[0256] For example, when detecting the distance between a solder pad and solder, the control device 1000 recognizes the solder pad and solder in the image shown by the image data based on the image data output from at least one of the detection devices 120 and 130, and detects the distance between the solder pad and solder. For example, the control device 1000 may determine whether the solder placement position is good or bad based on the detected distance between the solder pad and solder. For example, the control device 1000 may determine that the solder placement position is bad if the detected distance between the solder pad and solder is greater than or equal to a predetermined threshold (for example, a state where no solder is placed on the solder pad). Also, for example, when detecting the shape of solder, the control device 1000 recognizes the solder in the point cloud shown by the shape data based on the shape data output from at least one of the detection devices 120 and 130, and detects the shape of the solder. For example, the control device 1000 may determine whether the shape of the solder is good or bad based on the detected shape of the solder. For example, the control device 1000 may determine that the shape of the solder is defective if the difference between the detected solder shape and the desired shape is greater than or equal to a threshold. Also, for example, when detecting the volume of solder, the control device 1000 may estimate the volume of solder using an existing method based on the solder shape detected by the method described above. For example, the control device 1000 may determine whether the solder volume is good or bad based on the estimated volume of solder. For example, the control device 1000 may determine that the solder volume is defective if the detected solder volume is outside the threshold (for example, if the solder volume is too large or too small). Also, for example, when detecting the position and orientation of solder, the control device 1000 recognizes the solder in the point cloud indicated by the shape data output from at least one of the detection devices 120 and 130, and detects the position and orientation of the solder. For example, the control device 1000 may determine whether the solder shape is good or bad based on the detected position and orientation of the solder. For example, the control device 1000 may determine that the solder position and orientation are defective if at least one of the detected solder position and orientation is outside a threshold.

[0257] The control device 1000 may perform machine learning using an existing method, with training data obtained by associating at least one piece of information from the image data and shape data used in the above-described steps S111 to S117 with the solder quality detected as described above. In this case, the control device 1000 may use the results of machine learning to control each device of robot 1 (for example, position and attitude control of dispenser 40 or dispensing control of dispenser 40). The control device 1000 may also use the results of machine learning to control at least one of the devices of robot 2 and each device of robot 3.

[0258] Furthermore, the control device 1000 may use the information detected as described above regarding the area of ​​the solder pad and the state of the solder (at least one piece of information regarding the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and orientation of the solder) to control the position and orientation of the holding device 50 on the robot arm 210 of the robot 2. In this case, the elements held by the holding device 50 can be efficiently placed on the solder.

[0259] Furthermore, the control device 1000 may use the information detected as described above regarding the area of ​​the solder pad and the state of the solder (at least one piece of information regarding the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and orientation of the solder) to control at least one of the position and orientation control of the light irradiation device 60 by the robot arm 310 of the robot 3, and the control of the galvanometer mirror 61. In this case, the light irradiation device 60 can efficiently irradiate the processing light L to the areas on the circuit board T where the processing light L should be irradiated (for example, placed elements, placed solder, solder pads, etc.).

[0260] Further, the control device 1000 may use at least one of the information detected as described above, i.e., the information regarding the area of the solder pad and the state of the solder (at least one of the information on the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and posture of the solder), to control the conditions of the processing light L irradiated from the light irradiation device 60 by the robot arm 310 of the robot 3 (for example, at least one of the intensity of the processing light L, the spot size of the processing light L, the irradiation time of the processing light L, and the irradiation range of the processing light L). The irradiation range of the processing light L includes, for example, at least a part of elements, solder, solder pads, etc. as the location where the processing light L should be irradiated. Here, at least one of the information on the area of the solder pad and the information regarding the state of the solder (at least one of the information on the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and posture of the solder) can be rephrased as the information regarding the state of the location where the processing light L should be irradiated. For example, the control device 1000 may determine the conditions of the processing light L based on the detected information regarding the state of the location where the processing light L should be irradiated. As an example, the control device 1000 may determine the spot size of the processing light L based on the area of the solder pad. The information regarding the state of the location where the processing light L should be irradiated may include, in addition to the above-mentioned information, at least one of the image data and the shape data output from at least one of the detection devices 120 and 130, and the information regarding elements, solder, solder pads as the location where the processing light L should be irradiated that can be detected based on the matching process.

[0261] Further, the control device 1000 may use at least one of the information on the position and orientation of the circuit board T calculated in the above-described processes of steps S111 to S117, the information on the position and orientation of each solder pad, and the information on the mounting order for each solder pad for at least one of the control of the position and orientation of the holding device 50 by the robot arm 210 of the robot 2 and the control of the holding force of the holding device 50. In this case, the element held by the holding device 50 can be efficiently placed on the solder. For example, when the above information is used for the control of the position and orientation of the holding device 50 by the robot arm 210 of the robot 2, the process of step S121 described later by the control device 1000 can be omitted, so that the element held by the holding device 50 can be efficiently placed on the solder pad (solder).

[0262] Further, the control device 1000 may use at least one of the information on the position and orientation of the circuit board T calculated in the above-described processes of steps S111 to S117, the information on the position and orientation of each solder pad, and the information on the mounting order for each solder pad for at least one of the control of the position and orientation of the light irradiation device 60 by the robot arm 310 of the robot 3 and the control of the galvanometer mirror 61. In this case, the processing light L can be efficiently irradiated to the location on the circuit board T where the processing light L should be irradiated by the light irradiation device 60. For example, when the above information is used for the control of the position and orientation of the light irradiation device 60 by the robot arm 310 of the robot 3, the process of step S131 described later by the control device 1000 can be omitted, so that the melting of the solder by the processing light L can be efficiently performed by the light irradiation device 60.

[0263] Note that the control device 1000 may cause a display device (not shown) to display at least one of the image data, shape data, and the detection result of the solder state used in the above-described solder state detection process.

[0264] For example, the control device 1000 may control the drive unit 111 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 120 and 130, so that the solder dispensed from the dispenser 40, which is displaced together with the displacement of at least one of the detection devices 120 and 130, is initially placed at a first position on the circuit board T, and then at a second position different from the first position.

[0265] For example, the control device 1000 may control the drive unit 111 to stop driving the drive unit 111. The control device 1000 may control the drive unit 111 based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 120 and 130 after the drive unit 111 has stopped driving, so that the solder dispensed from the dispenser 40, which is displaced together with the displacement of at least one of the detection devices 120 and 130, is initially placed at a first position on the circuit board T, and then at a second position different from the first position.

[0266] Robot 2 The control device 1000 that controls the robot 2 performs the processes of steps S122 and S123, which correspond to the processes of steps S131 and S132 described above. However, the control device 1000 performs the processes of steps S121 and S123 using the output of at least one of the detection devices 220 and 230 provided by the robot 2.

[0267] For example, the holding device 50 is equipped with a tweezers hand that can open and close the tips of the tweezers. The holding device 50 may also be equipped with a suction device that can attract and hold the element.

[0268] The control device 1000 may perform calibration of the holding device 50 before performing the following steps S121 to S129.

[0269] As a prerequisite, it is assumed that the detection device 230 and the holding device 50 are mounted on the robot arm 210 in such a positional relationship that the tip of the holding device 50 (i.e., in the case of a tweezers hand, the tip of the tweezers that contacts the element when holding the element) is within the field of view of the cameras 31 and 32 of the detection device 230, which has a configuration similar to that of the detection device 330. For the sake of explanation, the cameras 31 and 32 of the detection device 230 will be referred to as the cameras 31 and 32 of the detection device 230, as an example in which the detection device 230 has cameras 31 and 32 similar to those of the detection device 330.

[0270] As a calibration of the holding device 50, the control device 1000 performs the matching process described above using the shape data output from the detection device 230 when the holding device 50 is not gripping an element and the CAD data of the holding device 50, and pre-calculates the position and orientation of the holding device 50 (for example, the position and orientation of the tip of the tweezers hand included in the field of view of the cameras 31 and 32 of the detection device 230). In other words, the control device 1000 pre-calculates the position and orientation of the holding device 50 in the coordinate system of the robot arm 210 based on the shape data of at least a part of the holding device 50 (for example, the tip of the tweezers hand).

[0271] The control device 1000 may, as a calibration of the holding device 50, determine the correspondence between the coordinate system of the detection device 230 and the coordinate system of the holding device 50 based on the shape data of at least a portion of the holding device 50. Then, based on this correspondence between the coordinate system of the detection device 230 and the coordinate system of the holding device 50, and the previously determined correspondence between the coordinate system of the detection device 230 and the coordinate system of the robot arm 210, the control device 1000 may calculate the position and orientation of the holding device 50 in the coordinate system of the robot arm 210. The control device 1000 does not need to calculate the position and orientation of the holding device 50 in the coordinate system of the robot arm 210 as a calibration of the holding device 50, and may simply calculate the correspondence between the coordinate system of the detection device 230 and the coordinate system of the holding device 50. The correspondence between the coordinate system of the detection device 330 and the coordinate system of the holding device 50 may be a transformation matrix between the coordinate system of the detection device 330 and the coordinate system of the holding device 50. Furthermore, if the holding device 50 is calibrated, the control device 1000 may control the drive unit 211 to move the robot arm 210 based on the position and orientation of the object (e.g., an element) calculated in step S121 described later and the calibration results of the holding device 50. Alternatively, the control device 1000 may control the drive unit 211 to move the robot arm 210 in step S123 described later, based on the position and orientation of the object (e.g., a circuit board T) calculated in step S122 described later and the calibration results of the holding device 50. Alternatively, the control device 1000 may control the drive unit 211 to move the robot arm 210 in step S126 described later, based on the position and orientation of the object (e.g., a solder pad or solder) calculated in step S125 described later and the calibration results of the holding device 50. The calibration result of the holding device 50 may be, for example, the position and orientation of the holding device 50 in the coordinate system of the robot arm 210, or the correspondence between the coordinate system of the detection device 230 and the coordinate system of the holding device 50.

[0272] As described above, the robot control unit 100 does not have to be part of the control device 1000, and may be configured separately from the control device 1000. In this case, the control device 1000 may generate a control signal for controlling the robot arm 210 (drive unit 211) based on the calculated position and orientation of the object and the calibration results of the holding device 50. The control device 1000 may output the generated control signal to the robot control unit 100. The robot control unit 100 may generate a drive signal for driving the drive unit 211 based on the control signal output from the control device 1000. The robot control unit 100 may drive the drive unit 211 based on the generated drive signal.

[0273] Furthermore, markers may be provided on a portion of the holding device 50 that is included within the field of view of each of the cameras 31 and 32 of the detection device 230. In this case, the control device 1000 may perform the above-described calibration based, for example, on shape data including the markers output from the detection device 230.

[0274] Furthermore, the control device 1000 may perform calibration of the holding device 50 by executing a matching process using not only shape data, but also image data output from the detection device 230 and CAD data of the holding device 50. In addition, as described above, the control device 1000 may use not only CAD data, but also shape data or image data of the holding device 50 that have been acquired in advance, in the matching process.

[0275] Furthermore, the control device 1000 may use shape data or image data output from the detection device 220, rather than being limited to using the detection device 230 for calibration of the holding device 50. In this case, it is assumed that the detection device 220 and the holding device 50 are mounted on the robot arm 210 in such a positional relationship that the tip of the holding device 50 is within the field of view of the cameras 21 and 22 of the detection device 220, which has a configuration similar to that of the detection device 320.

[0276] In the following steps S121 to S129, the holding device 50 may come into contact with a predetermined object, causing a change in the position and orientation of the holding device 50 relative to the detection device 230. In this case, the control device 1000 can detect a change in the position and orientation of the holding device 50 relative to the detection device 230 based on a partial change in the holding device 50 in the image data or shape data output from the detection device 230 (for example, a partial change in the holding device 50 in the image). If a change in the position and orientation of the holding device 50 relative to the detection device 230 is detected, the control device 1000 may perform calibration.

[0277] The control device 1000 that controls the robot 2 holds the element (step S121). The control device 1000 controls the drive unit 211 of the robot arm 210 and the holding device 50 so that the robot 2's holding device 50 is brought closer to an element supply device (so-called parts feeder) not shown, and the robot arm 210 holds the desired element with the holding device 50. For example, the control device 1000 performs at least one of the matching process and tracking process described above, calculates the position and orientation of the desired element placed on the element supply device not shown, and then brings the holding device 50 closer to the desired element placed on the element supply device not shown, so that the holding device 50 can hold the element. Here, the control device 1000 may determine the holding force (gripping force) of the element in the holding device 50 according to the size of the element calculated in at least one of the matching process and tracking process. This prevents damage to the element caused by the holding of the element by the holding device 50 and prevents the element from falling out of the holding device 50.

[0278] After the processing in step S121, the control device 1000 calculates the position and orientation of the circuit board T as an example of an object (step S122). Similar to steps S111 and S131 described above, in the processing of step S122, the control device 1000 calculates the initial position and orientation of the circuit board T (i.e., the initial position and orientation) by the matching process of the matching unit 301. Furthermore, the control device 1000 calculates the position of each solder pad formed on the circuit board T. The control device 1000 calculates the position of each solder pad on the circuit board T based on the Gerber data of the circuit board T (i.e., the design data of the circuit board T). Based on the Gerber data, the control device 1000 identifies the order in which components are mounted (in this case, placed) on each solder pad.

[0279] Next, the control device 1000 controls the drive unit 211 to move the robot arm 210 so that the holding device 50 (detection devices 220 and 230) approaches the circuit board T (step 123). In the process of step S123, similar to steps S112 and S132 described above, the control device 1000 controls the drive unit 211 of the robot arm 210 so that the first solder pad to be mounted comes into the field of view of at least one of the detection devices 220 and 230.

[0280] Next, the control device 1000 determines whether the marker provided near the first solder pad to be mounted is within the field of view of at least one of the detection devices 220 and 230 (step S124). Similar to steps S113 and S133 described above, in step S124, the control device 1000 determines whether the detection devices 220 and 230 are in the desired position and orientation relative to the first solder pad to be mounted, based on the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals and the position and orientation information of the first solder pad to be mounted calculated in step S121. At this time, if the detection devices 220 and 230 are in the desired position and orientation relative to the first solder pad to be mounted, the control device 1000 determines that the marker provided near the solder pad is within the field of view of at least one of the detection devices 220 and 230.

[0281] In step S124, if it is determined that the marker placed near the first solder pad to be implemented is not within the field of view of at least one of the detection devices 220 and 230 (S124: No), the control device 1000 controls the drive unit 211 to continue moving the robot arm 210 based on the position and orientation information of the first solder pad to be implemented calculated in step S122 and the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals.

[0282] On the other hand, in the process of step S124, when it is determined that a marker provided near the first soldering pad implemented has entered the field of view of at least one of the detection devices 220 and 230 (step S124: Yes), the control device 1000 calculates the position and orientation of the marker provided near the first soldering pad implemented (step S125). In the process of step S125, similar to the above steps S114 and S134, the control device 1000 calculates the initial position and orientation (initial position and orientation) of the marker provided near the first soldering pad implemented by the matching process of the matching unit 301.

[0283] Next, the control device 1000 controls the drive unit 211 so that the position and orientation of the holding device 50 become the desired position and orientation where the element can be placed on the solder on the first soldering pad implemented, and moves the robot arm 210 (step S126). In the process of step S126, similar to the above steps S115 and S135, the control device 1000 uses the information on the initial position and orientation of the marker provided near the soldering pad calculated by the process of step S125, and based on the information on the position and orientation of the marker arranged near the soldering pad output from the 2D tracking unit 302 at intervals of a predetermined time and the positional relationship between the soldering pad and the marker, controls the drive unit 211 to move the robot arm 210. Note that the positional relationship between the soldering pad and the marker is known.

[0284] In other words, the control device 1000 controls the drive unit 211 to move the robot arm 210 so that the holding device 50 (detection devices 220 and 230) approaches the first soldering pad (solder arranged on the soldering pad) of the circuit board T. Note that the control device 1000 may also control the drive unit 211 to move the robot arm 210 based on the information on the distance between the soldering pad and the solder detected after the solder is placed on the soldering pad by the dispenser 40 of the robot 1. In this case, the holding device 50 can be brought closer to the soldering pad (solder arranged on the soldering pad) more accurately (more efficiently).

[0285] Next, the control device 1000 determines whether the position and orientation of the holding device 50 are the desired position and orientation that allows the element to be placed on the solder placed on the first solder pad to be mounted (step S127). In the process of step S127, similar to steps S116 and S136 described above, the control device 1000 determines whether the position and orientation of the holding device 50 relative to the solder pad (the solder placed on the solder pad) is the desired position and orientation, based, for example, on the position and orientation information of a marker placed near the first solder pad to be mounted, which is output from the 2D tracking unit 302 at predetermined time intervals, and the positional relationship between the solder pad and the marker.

[0286] In step S127, if it is determined that the position and orientation of the holding device 50 are not the desired position and orientation that would allow the element to be placed on the solder placed on the first solder pad to be mounted (step S127: No), the control device 1000 controls the drive unit 211 to continue moving the robot arm 210 based on the position and orientation of the solder pad (solder) output from the 2D tracking unit 302 at predetermined time intervals, so that the holding device 50 approaches the solder pad (solder).

[0287] On the other hand, if it is determined that the position and orientation of the holding device 50 are a desired position and orientation that allows the element to be placed on the solder pad (solder) (step S127: Yes), the control device 1000 calculates the position and orientation of the element held by the holding device 50 (step S128).

[0288] First, as a prerequisite, as described above, the tip of the holding device 50 (i.e., in the case of a tweezers hand, the tip of the tweezers that contacts the element when holding it) is included within the field of view of each of the cameras 31 and 32 of the detection device 230. Furthermore, the detection device 230 is positioned at a desired location on the robot arm 210 such that at least a portion of the element held by the holding device 50 is also included within the field of view of each of the cameras 31 and 32.

[0289] The control device 1000 performs the above-described CAD matching process using, for example, the shape data output from the detection device 230 (i.e., data including the shape data of the tip of the holding device 50 and at least a part of the element) and the CAD data of the element while the holding device 50 is holding the element, and calculates the position and orientation of the element. The position and orientation of an element held by the holding device 50 changes each time it is held by the holding device 50, even if the elements are of the same type (i.e., have the same shape). Therefore, by executing the process in step S125, the control device 1000 can recognize the position and orientation of the element, and in step S129 described later, the element can be placed with high precision on the first solder pad (solder) to be mounted.

[0290] Furthermore, the control device 1000 may, for example, recognize the shape data of the tip of the holding device 50 from the shape data output from the detection device 230 while the holding device 50 is holding the element, perform a process to remove the shape data of the tip of the holding device 50 from the shape data output from the detection device 230, and then perform a CAD matching process using the CAD data of the element to calculate the position and orientation of the element. In this case, the shape data of the tip of the holding device 50 can act as noise in the process of calculating the position and orientation of the element, which can suppress a decrease in the accuracy of calculating the position and orientation of the element.

[0291] Furthermore, the control device 1000 may, for example, remove the shape data of the tip of the holding device 50, which was calculated in the matching process during the calibration of the holding device 50 described above, from the shape data output from the detection device 230 while the holding device 50 is holding the element, and then perform a CAD matching process using the CAD data of the element to calculate the position and orientation of the element. In this case as well, a decrease in the accuracy of calculating the position and orientation of the element can be suppressed.

[0292] In addition, for the purpose of calibration of the holding device 50 described above, the detection device 230 is mounted on the robot arm 210 so that the tip of the holding device 50 is included in the field of view of each of the cameras 31 and 32 of the detection device 230. However, if calibration of the holding device 50 is not performed, the detection device 230 may be mounted on the robot arm 210 so that the holding device 50 is not included in the field of view of each of the cameras 31 and 32 of the detection device 230, but at least a portion of the elements held by the holding device 50 is included.

[0293] After the processing in step S128, the control device 1000 controls the holding device 50 so that the element is placed on the first solder pad (solder) to be mounted (step S129). Based on the position and orientation of the element held by the holding device 50 calculated in the processing in step S128 above, and the position and orientation of the solder pad (solder) output from the 2D tracking unit 302 at predetermined time intervals, the control device 1000 controls the drive unit 211 of the robot arm 210 so that the position and orientation of the element held by the holding device 50 becomes a desired position and orientation that allows the element to be placed on the solder pad (solder). Then, the control device 1000 controls the holding device 50 so that the element is released from being held, and the element is placed on the solder pad (solder).

[0294] At this time, similar to the process in step S117 described above, the relative position between the holding device 50 and the solder pad (solder) as the object may fluctuate due to, for example, vibration of the robot arm 210. Therefore, in order to suppress or eliminate the influence of the fluctuation in the relative position on the process in step S129, the control device 1000 controls at least one of the attitude and position of the holding device 50 based on the results of the tracking process described above.

[0295] For example, the control device 1000 may control the drive unit 211 so that the element gripped by the holding device 50, which displaces along with the displacement of at least one of the detection devices 220 and 230, is positioned on the solder pad (solder) based on at least one of the image data and shape data, which changes with the displacement of at least one of the detection devices 220 and 230.

[0296] For example, the control device 1000 may control the drive unit 211 to stop its operation. The control device 1000 may also control the drive unit 211 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230 after the drive unit 211 has stopped, so that the element gripped by the holding device 50, which displaces together with at least one of the detection devices 220 and 230, is positioned on the solder pad (solder).

[0297] In the processing of steps S121 to S129 described above, the control device 1000 may control the drive unit 211 of the robot arm 210 so that the holding device 50 and the detection devices 220 and 230 approach the circuit board T based on at least one of the image data and shape data generated by at least one of the detection devices 220 and 230. When the holding device 50 and the detection devices 220 and 230 approach the circuit board T to a predetermined distance, the control device 1000 may control the drive unit 211 so that the element gripped by the holding device 50, which displaces together with at least one of the detection devices 220 and 230, is positioned on the solder pad (solder) as the target object, based on the at least one of the data that changes with the displacement of at least one of the detection devices 220 and 230.

[0298] After the processing in step S129, the control device 1000 drives the drive unit 211 of the robot arm 210 so that the holding device 50 approaches an element supply device (not shown), and picks up the element to be placed on the second solder pad (solder) to be mounted. The control device 1000 then repeats the processing in steps S122 to S129 described above. The control device 1000 repeats the element picking and the steps described above until the placement of elements on the solder of each solder pad on the circuit board T is complete.

[0299] If there is only one solder pad on the circuit board T (i.e., only one element to be placed on the solder), the control device 1000 may, after processing in step S129, control the drive unit 211, etc., so that the robot arm 210, etc., is in a predetermined initial position.

[0300] The control device 1000 may perform the process in step S128 described above before the process in step S127. Alternatively, the control device 1000 may not perform the process in step S128 described above. In this case, the control device 1000 only needs to control the holding device 50 so that the element is placed on the first solder pad (solder) to be mounted after the process in step S127 (more precisely, step S127: Yes) described above.

[0301] The control device 1000 may also display at least one of the following information on a display device (not shown): image data, shape data, position and orientation information of the circuit board T, position and orientation information of the markers, position and orientation information of each solder pad, and information on the order in which components are mounted on each solder pad.

[0302] Furthermore, if the element can be placed on the solder on the solder pad, the control device 1000 does not need to perform the processes of steps S122 and S124 or steps S126 and S127 described above.

[0303] Furthermore, after the processing in step S129, the control device 1000 may detect the state of the elements placed on the solder pad based on at least one of the image data and shape data output from at least one of the detection devices 220 and 230, or based on the matching process. The elements to be detected include the position and orientation of the elements, the distance between the elements and the solder, and the distance between the elements and the solder pad. The control device 1000 may then determine the quality of the placed elements based on the detected state of the elements.

[0304] For example, when detecting the position and orientation of an element, the control device 1000 calculates the position and orientation of the element based on the matching process described above. For example, the control device 1000 may determine whether the placement of the element is good or bad based on the detected position and orientation of the element. For example, if at least one of the detected position and orientation of the element falls outside a predetermined threshold, the control device 1000 may determine that the solder placement is poor.

[0305] For example, when detecting the distance between an element and solder, the control device 1000 recognizes the element and solder in the point cloud indicated by the shape data output from at least one of the detection devices 220 and 230, and calculates the distance between the element and the solder. For example, the control device 1000 may also determine the quality of the element's placement based on the detected distance between the element and the solder. For example, if the detected distance between the element and the solder is greater than or equal to a predetermined threshold (for example, the element is not placed on the solder), the control device 1000 may determine that the solder placement is poor.

[0306] For example, when detecting the distance between an element and a solder pad, the control device 1000 recognizes the element and the solder pad in the image data based on the image data output from at least one of the detection devices 220 and 230, and calculates the distance between the element and the solder pad. For example, the control device 1000 may also determine whether the placement of the element is good or bad based on the detected distance between the element and the solder pad. For example, if the detected distance between the element and the solder pad is greater than or equal to a predetermined threshold (for example, the element is not placed on the solder pad), the control device 1000 may determine that the placement of the element is bad.

[0307] The control device 1000 may also perform machine learning using an existing method, with training data obtained by associating at least one piece of information from the image data and shape data used in the processing of steps S122 to S128 described above with the quality of the element determined as described above. In this case, the control device 1000 may use the results of machine learning to control each device of the robot 2 (for example, position and attitude control of the holding device 50 or holding control of the holding device 50). The control device 1000 may also use the results of machine learning to control at least one of the devices of the robot 1 and the devices of the robot 3.

[0308] Furthermore, the control device 1000 may use at least one piece of information detected as described above—the position and orientation of the element, the distance between the element and the solder, and the distance between the element and the solder pad—to control the position and orientation of the light irradiation device 60 by the robot arm 310 of the robot 3, to control the galvanometer mirror 61 of the light irradiation device 60, and to control at least one of the changes in the intensity and spot size of the processing light L. In this case, the processing light L can be efficiently irradiated by the light irradiation device 60 to the areas on the circuit board T where the processing light L should be irradiated (for example, placed elements, placed solder, solder pads, etc.). The control device 1000 may also use at least one piece of information detected as described above—the position and orientation of the element, the distance between the element and the solder, and the distance between the element and the solder pad—to control the position and orientation of the dispenser 40 by the robot arm 110 of the robot 1.

[0309] Furthermore, in the processing of steps S122 to S129 described above, the control device 1000 may detect at least one of the solder state (at least one piece of information regarding the distance between the solder pad and the solder, the shape of the solder, the volume of the solder, and the position and orientation of the solder) and the area of ​​the solder pad based on at least one of the image data and shape data output from the detection devices 120 and 130. In other words, the control device 1000 may detect information regarding the state of the area to be irradiated with processing light L. The control device 1000 may then use the information regarding the state of the area to be irradiated with processing light L, as detected as described above, to control the conditions of the processing light L irradiated from the light irradiation device 60 by the robot arm 310 of the robot 3 (for example, at least one of the intensity of the processing light L, the spot size of the processing light L, the irradiation time of the processing light L, and the irradiation range of the processing light L). In other words, the control device 1000 may determine the conditions of the processing light L based on the detected information regarding the state of the area to be irradiated with processing light L. In addition to the information described above, the information regarding the state of the area to be irradiated with processing light L may also include information regarding elements, solder, and solder pads that can be detected as areas to be irradiated with processing light L, based on at least one of the image data and shape data output from at least one of the detection devices 220 and 230, and a matching process.

[0310] Furthermore, the control device 1000 may use at least one of the information obtained in the processing steps S122 to S129 above—the position and orientation information of the circuit board T, the information on the mounting order for each solder pad, the position and orientation information of the marker, and the position and orientation information of the solder pads (solder)—to control at least one of the following: the position and orientation control of the light irradiation device 60 by the robot arm 310 of the robot 3, the control of the galvanometer mirror 61, and the change of the intensity and spot size of the processing light L. In this case, the processing light L can be efficiently irradiated to the areas on the circuit board T where the processing light L should be irradiated by the light irradiation device 60. The control device 1000 may also use at least one of the information obtained as described above—the position and orientation information of the circuit board T and the position and orientation information of the solder pads (solder)—to control the position and orientation of the dispenser 40 by the robot arm 110 of the robot 1.

[0311] The control device 1000 may also display at least one piece of information—image data, shape data, and the detection result of the element's state—used in the above-described element state detection process, on a display device (not shown).

[0312] For example, the control device 1000 may control the drive unit 211 based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230, such that after one element gripped by the holding device 50, which displaces together with the displacement of at least one of the detection devices 220 and 230, is positioned at a first position on the circuit board T, the other element gripped by the holding device 50 is positioned at a second position different from the first position.

[0313] For example, the control device 1000 may control the drive unit 211 to stop driving the drive unit 211. Based on at least one of the image data and shape data that change with the displacement of at least one of the detection devices 220 and 230 after the drive unit 211 has stopped driving, the control device 1000 may control the drive unit 211 so that after one element gripped by the holding device 50, which displaces together with the displacement of at least one of the detection devices 220 and 230, is positioned at a first position on the circuit board T, another element gripped by the holding device 50 is positioned at a second position different from the first position.

[0314] Furthermore, the processes in steps S111 to S117, steps S121 to S129, and steps S131 to S138 described above are performed repeatedly and concurrently.

[0315] The control device 1000 may perform machine learning using an existing method, using as training data data that associates at least one piece of information from the image data and shape data used in at least one of the processes described in steps S131 to S138, steps S111 to S117, and steps S121 to S129 with at least one piece of information from the soldering quality determined in step S138, the solder quality determined as described above, and the element quality determined as described above. In this case, the control device 1000 may use the results of machine learning to control at least one of the devices of robot 1, robot 2, and robot 3.

[0316] (Technical effects) LDS (Lase Direct Structuring) and SMT (Surface Mount Technology) are known technologies for mounting components onto the surface of circuit boards with a three-dimensional shape (i.e., 3D circuit boards). However, LDS requires the use of special heat-resistant molding materials (resins). Furthermore, the 3D circuit board must be placed inside a mounting machine (chip mounter) or reflow oven. For these reasons, LDS and similar technologies can only be applied to relatively high-cost and relatively small products.

[0317] For this reason, for relatively large products, a method is often adopted in which, for example, sub-boards are connected to a main board with cables. However, with this method, the weight of the cables can make the product relatively heavy, and manual labor is required for assembling the wiring components. On the other hand, in fields such as automobiles, there is a demand for technology to mount elements on relatively large 3D circuit boards due to space constraints and weight constraints.

[0318] For relatively large 3D circuit boards, it is desirable to use relatively inexpensive substrate materials from the perspective of cost reduction. Since relatively inexpensive substrate materials have relatively low heat resistance, laser soldering, which allows for pinpoint heat input, is a method for mounting components. However, there is a technical problem in that if the irradiation position of the processing light L, such as laser light, cannot be precisely controlled, the 3D circuit board will suffer thermal damage.

[0319] In contrast, the robot 3 described above controls the orientation of the galvanometer mirror 61, etc., based on at least one of the image data and shape data output from the detection device 330, so that even if the positional relationship between the light irradiation device 60, etc., and the object (e.g., solder pad, element, solder) is displaced, the irradiation position of the processing light L remains the same. In other words, the robot 3 allows for precise control of the irradiation position of the processing light L.

[0320] Soldering with Robot 3 can suppress thermal damage to the circuit board, thus increasing the range of circuit board materials that can be used. In other words, soldering with Robot 3 allows the use of relatively inexpensive circuit board materials.

[0321] According to robot 3, the robot arm 310 moves the light irradiation device 60, etc. (i.e., while the light irradiation device 60, etc. is moving), and the galvanometer mirror 61 changes the irradiation position of the processing light L so that the processing light L can be irradiated to the desired position. Therefore, one or more elements can be efficiently mounted on relatively large substrates.

[0322] According to robot 3, after the robot arm 310 moves the light irradiation device 60, etc. (in other words, when the robot arm 310 is not being driven by the drive unit 311), the galvanometer mirror 61 can be used to adjust the irradiation position of the processing light L, thereby irradiating the processing light L to the desired position. Therefore, the processing light L can be irradiated without waiting for the vibrations of the light irradiation device 60, etc., moved by the robot arm 310 to subside. Furthermore, even if there is an error in the movement of the light irradiation device 60, etc., by the robot arm 310, this error can be corrected by controlling the orientation of the galvanometer mirror 61, and the processing light L can be irradiated to the desired position.

[0323] Because robot 3 has detection devices 320 and 330, it is possible to perform a quality inspection of the soldering after soldering (after the process in step S137 described above) (see the process in step S138 described above). In other words, because robot 3 has detection devices 320 and 330, it is possible to perform a quality inspection of the soldering on the spot after soldering. To put it another way, robot 3 can perform quality inspections of soldering efficiently.

[0324] Robot 1 can recognize the position and orientation of an object (e.g., a solder pad) at predetermined time intervals through tracking processing by the control device 1000. As a result, the drive unit 111 of the robot arm 110 is controlled so that even if the relative position between the object and the dispenser 40 (detection devices 120 and 130) changes (displaces) over time, solder can be placed at the desired position on the object (in other words, where the solder should be placed).

[0325] Robot 2 can recognize the position and orientation of an object (for example, solder placed on a solder pad) at predetermined time intervals through tracking processing by the control device 1000. As a result, the drive unit 211 of the robot arm 210 is controlled so that even if the relative position between the object and the holding device 50 (detection devices 220 and 230) changes (displaces) over time, the element can be placed at the desired position on the object (in other words, where the element should be placed).

[0326] Furthermore, robot 2 may be used for purposes other than the soldering described above. In the soldering described above, robot 2 holds the element to be soldered with the holding device 50 and places the held element on the object (for example, the circuit board T) in order to place the element to be soldered on the object. However, robot 2 may also hold an object other than the element to be soldered with the holding device 50 and place the held object on another object. For example, robot 2 may be used to assemble multiple objects. Robot 2 may assemble the first and second objects by holding the first object with the holding device 50 and placing the held first object on the object (second object). Even in this case, similar to the soldering described above, the control device 1000 may control robot 2 (the drive unit 211 of the robot arm 210) to hold the first object and place the held first object on the object (second object) based on image data and shape data from at least one of the detection devices 220 and 230 of robot 2. Furthermore, the first object can also be called an object, since it is the object that is held by the holding device 50.

[0327] Furthermore, the first object and the second object may be objects that fit together. For example, one of the first and second objects may have a protrusion, and the other of the first and second objects may have a recess that fits into the protrusion. One of the first and second objects may have a first recess and a first protrusion, and the other of the first and second objects may have a second protrusion and a second recess that fit into the first recess and the first protrusion, respectively. For example, the first object may be a rod-shaped object, and the second object may have a hole that fits into the rod-shaped object. For example, the first object may be a plate-shaped object, and the second object may have a slit that fits into at least a part of the plate-shaped object. For example, the first and second objects may each be connectors that fit together. The robot 2 may hold the first object with the holding device 50 and fit the held first object with the second object. Furthermore, fitting the first object with the second object can also be described as installing the first object onto the second object.

[0328] Note that the first object and the second object do not necessarily have to be objects that fit together. The first object and the second object may be objects that are joined together. Adhesive may be applied to at least one of the first object and the second object, and the first object and the second object may be objects that are joined together via the adhesive. Robot 2 may hold the first object with the holding device 50 and join the held first object to the second object. Note that joining the first object to the second object can also be said to be installing the first object to the second object.

[0329] Furthermore, fitting or joining the first object to the second object can also be described as assembling the first object to the second object. Note that the first and second objects do not necessarily have to be objects whose relative positions are fixed by the placement of the first object on the second object. The second object may be a tray or box for placing the first object. Robot 2 may hold the first object with the holding device 50 and place the held first object on the second object. Placing the first object on the second object can also be described as installing the first object on the second object. Robot 2 may pick up any one first object from a tray or box containing multiple first objects in a random stack and place the held first object on the second object.

[0330] As an example of installing an object different from the elements to be soldered as described above, the operation by robot 2 to install the first object onto the second object when fitting a first object having a protrusion and a second object having a recess that can be fitted into the protrusion will be explained with reference to the flowchart in Figure 25.

[0331] First, the control device 1000 that controls the robot 2 performs calibration of the holding device 50 (step S171). As a prerequisite, it is assumed that the detection device 230 and the holding device 50 are mounted on the robot arm 210 in such a positional relationship that the tip of the holding device 50 (i.e., in the case of a tweezers hand, the tip of the tweezers that contacts the first object when holding the first object) is within the field of view of the cameras 31 and 32 of the detection device 230, which has a configuration similar to that of the detection device 330. For the sake of explanation, the cameras 31 and 32 of the detection device 230 will be referred to as the cameras 31 and 32 of the detection device 230, as an example in which the detection device 230 has cameras 31 and 32 similar to those of the detection device 330.

[0332] As a calibration of the holding device 50, the control device 1000 performs the matching process described above using the shape data output from the detection device 230 when the holding device 50 is not holding the first object, and the CAD data of the holding device 50, and pre-calculates the position and orientation of the holding device 50 (for example, the position and orientation of the tip of the tweezers hand included in the field of view of the cameras 31 and 32 of the detection device 230). In other words, the control device 1000 pre-calculates the position and orientation of the holding device 50 in the coordinate system of the robot arm 210 based on the shape data of at least a part of the holding device 50 (for example, the tip of the tweezers hand).

[0333] Furthermore, as calibration of the holding device 50, the control device 1000 may determine the correspondence between the coordinate system of the detection device 230 and the coordinate system of the holding device 50 based on the shape data of at least a portion of the holding device 50. Then, based on this correspondence between the coordinate system of the detection device 230 and the coordinate system of the holding device 50, and the previously determined correspondence between the coordinate system of the detection device 230 and the coordinate system of the robot arm 210, the control device 1000 may calculate the position and orientation of the holding device 50 in the coordinate system of the robot arm 210. Furthermore, as calibration of the holding device 50, the control device 1000 does not need to calculate the position and orientation of the holding device 50 in the coordinate system of the robot arm 210, and may only calculate the correspondence between the coordinate system of the detection device 230 and the coordinate system of the holding device 50.

[0334] Furthermore, markers may be provided on a portion of the holding device 50 that is included within the field of view of each of the cameras 31 and 32 of the detection device 230. In this case, the control device 1000 may perform the above-described calibration based, for example, on shape data including the markers output from the detection device 230.

[0335] Furthermore, the control device 1000 may perform calibration of the holding device 50 by executing a matching process using not only shape data, but also image data output from the detection device 230 and CAD data of the holding device 50. In addition, as described above, the control device 1000 may use not only CAD data, but also shape data or image data of the holding device 50 that have been acquired in advance, in the matching process.

[0336] Furthermore, the control device 1000 may use shape data or image data output from the detection device 220, rather than being limited to using the detection device 230 for calibration of the holding device 50. In this case, it is assumed that the detection device 220 and the holding device 50 are mounted on the robot arm 210 in such a positional relationship that the tip of the holding device 50 is within the field of view of the cameras 21 and 22 of the detection device 220, which has a configuration similar to that of the detection device 320.

[0337] Next, the control device 1000 controls the drive unit 211 of the robot arm 210 and the holding device 50 so that the holding device 50 holds the first object (step S172). The control device 1000 controls the robot arm 210's drive unit 211 and the holding device 50 so that the robot 2's holding device 50 can be brought closer to the tray (not shown) on which at least one first object is placed, and the robot arm 210 can hold the desired first object with the holding device 50. For example, the control device 1000 calculates the position and orientation of the desired first object placed on the tray (not shown) by performing at least one of the matching process and tracking process described above. Based on the calculated position and orientation of the desired first object and the result of the calibration of the holding device 50 performed in step S171, the control device 1000 brings the holding device 50 closer to the desired first object placed on the tray (not shown) so that the robot arm 2 can hold the desired first object with the holding device 50. In step S172, the first object is the object to be held by the holding device 50, so the first object can also be called the object being held. The calibration result of the holding device 50 may be the position and orientation of the holding device 50 in the coordinate system of the robot arm 210, as described above, or it may be the correspondence between the coordinate system of the detection device 230 and the coordinate system of the holding device 50. The control device 1000 may also determine the holding force (gripping force) of the first object in the holding device 50 according to the desired size of the first object calculated in at least one of the matching process and the tracking process. This prevents damage to the first object caused by the holding device 50 or the first object falling out of the holding device 50.

[0338] After the processing in step S172, the control device 1000 calculates the position and orientation of the second object as the target object (step S173). The control device 1000 calculates the initial position and orientation of the second object (i.e., the initial position and orientation) by the matching process of the matching unit 301.

[0339] Next, the control device 1000 controls the drive unit 211 so that the holding device 50 (detection devices 220 and 230) approaches the second object, thereby moving the robot arm 210 (step S174). Based on the position and orientation of the second object calculated in step S173 and information regarding the position of the recess in the second object (i.e., design data of the second object), the control device 1000 controls the drive unit 211 of the robot arm 210 so that the holding device 50 (detection devices 220 and 230) approaches the second object and the recess of the second object comes into the field of view of at least one of the detection devices 220 and 230. The control device 1000 may also use the results of the calibration of the holding device 50 performed in step S171 to control the drive unit 211 of the robot arm 210 in step S174.

[0340] Next, the control device 1000 determines whether the recess of the second object is within the field of view of at least one of the detection devices 220 and 230 (step S175). Based on the position and orientation information of the second object output from the 2D tracking unit 302 at predetermined time intervals and the information regarding the position of the recess in the second object, the control device 1000 determines whether the detection devices 220 and 230 are in the desired position and orientation relative to the recess of the second object. At this time, if the detection devices 220 and 230 are in the desired position and orientation relative to the recess of the second object, the control device 1000 determines that the recess of the second object is within the field of view of at least one of the detection devices 220 and 230.

[0341] In step S175, if it is determined that the recess of the second object is not within the field of view of at least one of the detection devices 220 and 230 (S175: No), the control device 1000 controls the drive unit 211 to continue moving the robot arm 210 so that the recess of the second object comes into the field of view of at least one of the detection devices 220 and 230, based on the position and orientation information of the second object output from the 2D tracking unit 302 at predetermined time intervals and the information regarding the position of the recess in the second object.

[0342] On the other hand, in the process of step S175, if it is determined that the recess of the second object has entered the field of view of at least one of the detection devices 220 and 230 (step S175: Yes), the control device 1000 calculates the position and orientation of the recess of the second object (step S176). The control device 1000 calculates the initial position and orientation (initial position and orientation) of the recess provided in the second object by the matching process of the matching unit 301.

[0343] Next, the control device 1000 controls the drive unit 211 so that the position and orientation of the holding device 50 are in a desired position and orientation such that the convex portion of the first object (held by the holding device 50) can be fitted into the recess of the second object, and moves the robot arm 210 (step S177). The control device 1000 controls the drive unit 211 to move the robot arm 210 based on the position and orientation information of the recess of the second object, which is output from the 2D tracking unit 302 at predetermined time intervals using the initial position and orientation information of the recess of the second object calculated by the processing in step S176, and the result of the calibration of the holding device 50 performed in step S171. In other words, the control device 1000 controls the drive unit 211 so that the holding device 50 (first object) approaches the recess of the second object, and moves the robot arm 210.

[0344] Next, the control device 1000 determines whether the position and orientation of the holding device 50 are in a desired position and orientation such that the convex portion of the first object (held by the holding device 50) can be fitted into the recess of the second object (step S178). For example, the control device 1000 determines whether the position and orientation of the holding device 50 relative to the recess is in a desired position and orientation based on the position and orientation information of the recess of the second object output from the 2D tracking unit 302 at predetermined time intervals and the results of the calibration of the holding device 50 performed in step S171.

[0345] In step S178, if it is determined that the position and orientation of the holding device 50 are not in the desired position and orientation that allows the convex portion of the first object (held by the holding device 50) to fit into the recess of the second object (step S178: No), the control device 1000 controls the drive unit 211 to move the robot arm 210 so that the position and orientation of the holding device 50 become the desired position and orientation that allows the convex portion of the first object (held by the holding device 50) to fit into the recess of the second object, based on the position and orientation information of the recess output from the 2D tracking unit 302 at predetermined time intervals and the results of the calibration of the holding device 50 performed in step S171, so that the holding device 50 approaches the recess.

[0346] On the other hand, if it is determined that the position and orientation of the holding device 50 is a desired position and orientation that allows the protrusion of the first object to be fitted into the recess of the second object (step S178: Yes), the control device 1000 calculates the position and orientation of the first object held by the holding device 50 (step S179).

[0347] As a prerequisite, as described above, the tip of the holding device 50 (i.e., in the case of a tweezers hand, the tip of the tweezers that contacts the first object when holding the first object) is included within the field of view of each of the cameras 31 and 32 of the detection device 230. Furthermore, the detection device 230 is positioned at a desired location on the robot arm 210 such that at least a portion of the first object held by the holding device 50 is also included within the field of view of each of the cameras 31 and 32. The control device 1000 performs the above-described CAD matching process using, for example, the shape data output from the detection device 230 while the holding device 50 is holding the first object (i.e., data including the shape data of the tip of the holding device 50 and at least a portion of the first object) and the CAD data of the first object, to calculate the position and orientation of the first object. The position and orientation of the first object held by the holding device 50 change each time it is held by the holding device 50, even if the objects are of the same type (i.e., the same shape). Therefore, by executing the process in step S179, the control device 1000 can recognize the position and orientation of the first object, and in step S180 described later, the protrusion of the first object can be fitted into the recess of the second object with high precision. The control device 1000 may also calculate the position and orientation of the protrusion of the first object held by the holding device 50 based on the position information of the protrusion on the first object (i.e., the design data of the first object) and the calculated position and orientation information of the first object.

[0348] Furthermore, the control device 1000 may, for example, recognize the shape data of the tip of the holding device 50 from the shape data output from the detection device 230 while the holding device 50 is holding the first object, perform a process to remove the shape data of the tip of the holding device 50 from the shape data output from the detection device 230, and then perform a CAD matching process using the CAD data of the first object to calculate the position and orientation of the first object. In this case, the shape data of the tip of the holding device 50 can act as noise in the process of calculating the position and orientation of the first object, which can suppress a decrease in the accuracy of calculating the position and orientation of the first object.

[0349] Furthermore, the control device 1000 may, for example, remove the shape data of the tip of the holding device 50, which was calculated in the matching process during the calibration of the holding device 50 in step S171 described above, from the shape data output from the detection device 230 while the holding device 50 is holding the first object, and then perform a CAD matching process using the CAD data of the first object to calculate the position and orientation of the first object. In this case as well, a decrease in the accuracy of calculating the position and orientation of the first object can be suppressed.

[0350] After the processing in step S179, the control device 1000 controls the holding device 50 so that the protrusion of the first object fits into the recess of the second object (step S180). Based on the position and orientation of the first object held by the holding device 50 calculated in the processing in step S179, the position and orientation of the recess of the second object output from the 2D tracking unit 302 at predetermined time intervals, and the results of the calibration of the holding device 50 performed in step S171, the control device 1000 controls the drive unit 211 of the robot arm 210 so that the position and orientation of the first object (protrusion of the first object) held by the holding device 50 becomes a desired position and orientation that allows it to fit into the recess of the second object. Then, the control device 1000 controls the drive unit 211 of the robot arm 210 to fit the protrusion of the first object into the recess of the second object, and controls the holding device 50 to release the holding of the first object, thereby installing the protrusion of the first object in the recess of the second object.

[0351] After the processing in step S180, the control device 1000 performs an inspection of the state of installation of the first object on the second object based on at least one of the image data and shape data output from at least one of the detection devices 220 and 230 (step S181).

[0352] As an example, the control device 1000 determines whether the installation posture of the first object on the second object is good or bad as an inspection of the state of installation of the first object on the second object. For example, the control device 1000 calculates the posture of the first object relative to the second object based on the shape data of the first object and the second object output from at least one of the detection devices 220 and 230. Based on the calculated posture of the first object relative to the second object, the control device 1000 determines whether the installation posture of the first object on the second object is good or bad. For example, if the posture of the first object relative to the second object deviates from a predetermined posture, the control device 1000 determines that the installation posture of the first object on the second object is bad. The control device 1000 may also calculate the posture of the first object relative to the second object based on image data of the first object and the second object output from at least one of the detection devices 220 and 230. The control device 1000 may also calculate the posture of the first object relative to the second object by calculating the postures of the first object and the second object respectively through the matching process described above.

[0353] Furthermore, the control device 1000 is not limited to the case where the protrusion of the first object is fitted into the recess of the second object, but may also determine whether the installation posture of the first object on the second object is good or bad. For example, in the case where the first object and the second object are objects that are joined together, the control device 1000 may determine whether the installation posture of the first object on the second object is good or bad after the first object has been joined (i.e., installed) to the second object by the process of steps S171 to S180 described above. For example, in the case where the first object is placed on the second object, the control device 1000 may determine whether the installation posture of the first object on the second object is good or bad after the first object has been placed (i.e., installed) to the second object by the process of steps S171 to S180 described above.

[0354] Furthermore, the control device 1000 may determine whether the installation position of the first object on the second object is good or bad as an inspection of the state of installation of the first object on the second object. For example, if the second object has a plurality of recesses into which the protrusions of the first object can be fitted, the control device 1000 may inspect whether the protrusions of the first object are fitted into a desired recess among the recesses formed in the second object. In this case, for example, the control device 1000 may calculate the position of the first object on the second object based on the shape data of the first object and the second object output from at least one of the detection devices 220 and 230. The control device 1000 may determine whether the installation position of the first object on the second object is good or bad based on the calculated position of the first object on the second object. For example, if the position of the first object on the second object is outside a predetermined position, the control device 1000 may determine that the installation position of the first object on the second object is bad.

[0355] The control device 1000 may calculate the position of the first object in the second object based on image data of the first object and the second object output from at least one of the detection devices 220 and 230. Alternatively, the control device 1000 may calculate the position of the first object in the second object by calculating the respective positions of the first object and the second object through the matching process described above. Furthermore, the control device 1000 is not limited to cases where the second object has multiple recesses into which the protrusions of the first object can be fitted, and may also determine the quality of the installation position of the first object on the second object. For example, in the case where the first object and the second object are objects that are joined together, the control device 1000 may determine the quality of the installation position of the first object on the second object after the first object has been joined (i.e., installed) on the second object through the processes described in steps S171 to S180 above. For example, when the control device 1000 places the first object on the second object, it may determine whether the orientation of the first object on the second object is good or bad after the first object has been placed (i.e., installed) on the second object by the processing of steps S171 to S180 described above.

[0356] The control device 1000 may also determine whether the first object is installed on the second object as an inspection of the installation status of the first object on the second object. For example, the control device 1000 may calculate the distance between the first object and the second object based on shape data output from at least one of the detection devices 220 and 230. For example, the control device 1000 may determine that the installation of the first object on the second object is faulty if the calculated distance between the first object and the second object is greater than or equal to a predetermined distance, or if the first object does not exist. The control device 1000 may also calculate the distance between the first object and the second object based on image data output from at least one of the detection devices 220 and 230. The control device 1000 may also calculate the distance between the first object and the second object by calculating the respective positions and orientations of the first object and the second object through the matching process described above.

[0357] Furthermore, the control device 1000 is not limited to the case where the protrusion of the first object is fitted into the recess of the second object, but may also determine whether or not the first object has been installed on the second object. For example, in the case where the first object and the second object are objects that are joined together, the control device 1000 may determine whether or not the first object has been installed on the second object after the first object has been joined (i.e., installed) on the second object by the process of steps S171 to S180 described above. For example, when the first object is placed on the second object, the control device 1000 may determine whether or not the first object has been installed on the second object after the first object has been placed (i.e., installed) on the second object by the process of steps S171 to S180 described above.

[0358] Furthermore, the control device 1000 may display at least one piece of information, such as the shape data and image data used in the inspection performed in step S181 above, on a display device (not shown), based on the results of the inspection regarding the state of installation of the first object on the second object.

[0359] The control device 1000 may perform machine learning using an existing method, with training data obtained by associating at least one piece of image data and shape data used in the processing of steps S171 to S180 described above with the results of the inspection regarding the state of installation of the first object on the second object performed in step S181 described above. In this case, the control device 1000 may use the results of machine learning to control each device of the robot 2 (for example, position and attitude control of the holding device 50 and holding control of the holding device 50).

[0360] The control device 1000 may perform the process in step S179 described above before the process in step S178. The control device 1000 does not have to perform the process in step S179 described above. If the process in step S179 described above is not performed, the detection device 230 may be provided on the robot arm 210 such that the first object held by the holding device 50 is not included in the field of view of each of the cameras 31 and 32 of the detection device 230.

[0361] The control device 1000 may also display at least one of the following information on a display device (not shown): image data, shape data, position and orientation information of the second object, calibration results of the holding device 50, and position and orientation information of the first object held by the holding device 50.

[0362] Furthermore, if the control device 1000 can place the first object on the second object, it does not need to perform the processes of steps S173 and S175 or steps S177 and S178 described above.

[0363] The control device 1000 may perform the process of step S171 described above in addition to step S171, between steps S172 to S181. The control device 1000 may not perform the process of step S171 described above before step S172, but rather between steps S172 to S181.

[0364] The control device 1000 does not have to perform the process in step S171 described above. If the process in step S171 is not performed, the detection device 230 may be mounted on the robot arm 210 such that the holding device 50 is not included in the field of view of the cameras 31 and 32 of the detection device 230.

[0365] In addition, during the processing of steps S172 to S181 described above, the position and orientation of the holding device 50 relative to the detection device 230 may change, for example, when the holding device 50 comes into contact with a predetermined object. In this case, the control device 1000 can detect a change in the position and orientation of the holding device 50 relative to the detection device 230 based on a partial change in the holding device 50 in the image data or shape data output from the detection device 230 (for example, a partial change in the holding device 50 in the image). If a change in the position and orientation of the holding device 50 relative to the detection device 230 is detected, the control device 1000 may perform the calibration described above.

[0366] The control device 1000 does not have to perform the process in step S181 described above. Although the processes in steps S171 to S181 described above were explained as an example of the operation of holding the first object with the holding device 50 and placing the first object on the second object which is the target object, the second object may be held with the holding device 50 and the second object may be placed on the first object which is the target object. In this case, the second object can also be called the target object, since it is the object that is held by the holding device 50.

[0367] In the process described in steps S171 to S181 above, the detection device that outputs at least one of the image data and shape data used to control the robot arm 210 (drive unit 211) (at least one of the detection devices 220 and 230) and the detection device that outputs at least one of the image data and shape data for inspecting the state of the installation of the first object on the second object (at least one of the detection devices 220 and 230) are the same. However, the detection device that outputs at least one of the image data and shape data used to control the robot arm 210 (drive unit 211) and the detection device that outputs at least one of the image data and shape data for inspecting the state of the installation of the first object on the second object may be different detection devices.

[0368] As described above, the robot control unit 100 does not have to be part of the control device 1000, and may be configured separately from the control device 1000. In this case, the control device 1000 may generate a control signal for controlling the robot arm 210 (drive unit 211) based on the calculated position and orientation of the object (e.g., the second object) and the calibration results of the holding device 50. The control device 1000 may output the generated control signal to the robot control unit 100. The robot control unit 100 may generate a drive signal for driving the drive unit 211 based on the control signal output from the control device 1000. The robot control unit 100 may drive the drive unit 211 based on the generated drive signal.

[0369] Furthermore, the robot 5 described later may be used for the processes from steps S171 to S181 described above.

[0370] <Second Embodiment> The second embodiment will be described with reference to Figures 17 and 18. Similar to the first embodiment described above, this embodiment also includes a soldering system that includes a soldering robot. Regarding the second embodiment, explanations that overlap with the first embodiment will be omitted, common parts in the drawings will be denoted by the same reference numerals, and only the fundamentally different points will be explained with reference to Figures 17 and 18.

[0371] (overview) An overview of the soldering system according to the second embodiment will be described with reference to Figures 17(a) and 17(b). In Figure 17, the soldering system is a system for soldering elements to a circuit board T. The soldering system includes a robot 4. The robot 4, which may also be called a processing device, solder coating device, element placement device, or soldering device, is provided with a dispenser 40 for dispensing solder, a holding device 50 capable of holding elements, a light irradiation device 60 for irradiating processing light L to melt the solder, a storage section (not shown) for storing elements, a supply device (not shown) for supplying desired elements from the storage section to the holding device 50, and detection devices 420 and 430 for detecting light from the circuit board T. The robot arm 410 has a drive unit 411 for moving the dispenser 40, the holding device 50, the light irradiation device 60, and the detection devices 420 and 430.

[0372] Here, detection devices 420 and 430 correspond to detection devices 320 and 330 described above, respectively. Detection device 420 may have the same configuration as detection device 320 described above. Detection device 430 may have the same configuration as detection device 330 described above.

[0373] The soldering system includes a control device 1000 that (i) controls a drive unit 411 so that a dispenser 40, a holding device 50, a light irradiation device 60, and detection devices 420 and 430 move closer to the circuit board T; (ii) controls the dispenser 40 so that solder is placed on a predetermined portion of the circuit board T; (iii) controls the holding device 50 so that elements are placed on the circuit board T via the placed solder; and (iv) controls the light irradiation device 60 so that the placed solder melts.

[0374] In Figure 17(a), the control device 1000 controls the dispenser 40, etc., so that solder is placed on a predetermined portion of the circuit board T. The control device 1000 then controls the holding device 50, etc., so that elements are placed via the placed solder. The control device 1000 then controls the light irradiation device 60 to melt the solder. After that, the control device 1000 may perform a quality inspection related to soldering, for example, based on the detection results of the detection device 430. In other words, robot 4 performs the work that was divided among robots 1, 2, and 3 according to the first embodiment described above, all by itself. With this configuration, it is possible to improve productivity, etc., while suppressing the initial investment in introducing robots.

[0375] (Robot 4) In Figures 17(a) and 17(b), the robot 4 comprises a dispenser 40, a holding device 50, a light irradiation device 60, a storage unit (not shown) for storing elements, and a supply device (not shown) for supplying desired elements from the storage unit to the holding device 50. The robot 4 further comprises (i) detection devices 220 and 230 that detect light from the circuit board T and generate data of at least one of image data and shape data, and (ii) a robot arm 410 that is equipped with the dispenser 40, the holding device 50, the light irradiation device 60, and the detection devices 420 and 430, and has a drive unit 411 that moves the holding device 50 and the detection devices 420 and 430. Examples of the storage unit include reels, trays, sticks, etc. Various existing configurations can be applied to the storage unit and the supply device, so a detailed explanation thereof is omitted.

[0376] The robot arm 410, like the robot arm 310 described above, has arm sections 410a and 410b and a wrist section 410c.

[0377] In Figure 17(a), the detection device 420 is located on the arm portion 410b of the robot arm 410, and the detection device 430 is located on the wrist portion 410c of the robot arm 410. However, the arrangement of the detection devices 420 and 430 is not limited to this. Furthermore, the robot 4 may be equipped with only one of the detection devices 420 and 430, or it may be equipped with other detection devices in addition to the detection devices 420 and 430 (i.e., the robot 4 may be equipped with three or more detection devices). Also, the robot 4 may be equipped with at least one detection device other than the detection devices 320 and 330. In other words, as long as the dispenser 40, holding device 50, and light irradiation device 60 can be brought close to the circuit board T or a predetermined part of the circuit board T (for example, solder pads provided on the circuit board T, or elements and solder placed on the circuit board T) by the drive unit 411 of the robot arm 410, the configuration (for example, the number and specifications of cameras in the detection device, the presence or absence of a projector, etc.), placement position, and number of detection devices 420 and 430 do not matter.

[0378] The control device 1000 may, with respect to the robot 4 configured as described above, control the drive unit 411 based on at least one of the image data and shape data that changes with the displacement of at least one of the detection devices 420 and 430, so that (i) the solder dispensed from the dispenser 40 which displaces with the displacement of at least one of the detection devices 420 and 430 is placed on a predetermined part of the circuit board T; (ii) the drive unit 411 is controlled so that the element gripped (held) by the holding device 50 which displaces with the displacement of at least one of the detection devices 420 and 430 is placed on a predetermined part of the circuit board T; or (iii) the orientation of the galvanometer mirror 61 is controlled so that the processing light L from the light irradiation device 60 which displaces with the displacement of at least one of the detection devices 420 and 430 is irradiated to the same position.

[0379] Here, the control device 1000 may be a separate device from the robot 4, or it may constitute a part of the robot 4 (in other words, the robot 4 may be equipped with the control device 1000).

[0380] The robot arm 410 may be mounted on an AGV (Automatic Guided Vehicle), for example. In this case, the control device 1000 may control at least one of the drive unit of the robot arm 410, the end effector of the robot arm 410, and the drive unit of the AGV based on the position and orientation information of the object acquired by the matching process and tracking process described above and below.

[0381] (Robot movements) First, the control device 1000 may perform calibration of the dispenser 40, the holding device 50, and the light irradiation device 60 before performing the following steps.

[0382] As a prerequisite, the detection device 430, dispenser 40, holding device 50, and light illumination device 60 are mounted on the robot arm 410 in such a positional relationship that a portion of the dispenser 40, a portion of the holding device 50, and a portion of the light illumination device 60 are within the field of view of the cameras 31 and 32 of the detection device 430, respectively.

[0383] The positions and orientations of the dispenser 40, the holding device 50, and the light irradiation device 60 are calculated in advance using a process similar to the calibration process described above.

[0384] Next, the operation of robot 4 will be explained with reference to the flowchart in Figure 18. However, the control device 1000 performs each of the processes described in the flowchart in Figure 18 using the output of at least one of the detection devices 420 and 430 provided by robot 4.

[0385] The control device 1000 that controls the robot 1 calculates the position and orientation of the circuit board T as an example of an object (step S111). In the process of step S111, the control device 1000 calculates the initial position and orientation of the circuit board T (i.e., the initial position and orientation) by the matching process of the matching unit 301. Furthermore, the control device 1000 calculates the position of each solder pad formed on the circuit board T. For example, the control device 1000 calculates the position and orientation of each solder pad on the circuit board T based on the Gerber data of the circuit board T (i.e., the design data of the circuit board T). Based on the Gerber data, the control device 1000 identifies the order in which to mount (in this case, place solder) each solder pad.

[0386] Next, the control device 1000 controls the drive unit 411 to move the robot arm 410 so that the dispenser 40 (detection devices 420 and 430) approaches the circuit board T (step 112). In step S112, the control device 1000 controls the drive unit 411 of the robot arm 410 so that the first solder pad to be mounted is within the field of view of at least one of the detection devices 420 and 430.

[0387] Next, the control device 1000 determines whether the first solder pad to be mounted is within the field of view of at least one of the detection devices 420 and 430 (step S113). In step S113, the control device 1000 determines whether the detection devices 420 and 430 are in the desired position and orientation relative to the first solder pad to be mounted, based on the position and orientation information of the circuit board T output from the 2D tracking unit 302 at predetermined time intervals and the position and orientation information of the first so...

Claims

[Claim 1] A control device for performing data processing for controlling the drive unit of a robot arm that moves an end effector, an imaging device, and a shape measuring device that perform processing on an object, the end effector, the imaging device, and the shape measuring device, The position and orientation of the object are calculated based on the image data acquired by the imaging device during the relative movement between the robot arm and the object as the imaging device and the shape measuring device approach the object, and the shape data representing the three-dimensional shape of the object generated by measuring the object with the shape measuring device. Based on two image data obtained by the imaging device imaging the object at different times during the relative movement, the displacement of the object between those different times is calculated. The calculated position and orientation of the object are added to the calculated displacement of the object to output information on the position and orientation of the object. A control device characterized by comprising a processing unit.

Images