Method and apparatus for detecting objects

Abstract

To provide a method and a device for detecting a target which can detect a target with a simple method and a simple structure.SOLUTION: The method for detecting a target includes the steps of: measuring the position and the attitude of a first target by a first positional information generation device; measuring the position and the attitude of the first target by a second positional information generation device; calculating the position and the attitude of one of the first positional information generation device and the second positional information generation device to the other on the basis of the difference between the position and the attitude of the first target measured by the first positional information generation device and the position and the attitude of the first target measured by the second positional information generation device; and measuring the position and the attitude of a second target by the second positional information generation device with using the position and the attitude of one of the first positional information generation device and the second positional information generation device to the other.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a method and an apparatus for detecting an object. [Background Art]

[0002] Conventionally, there has been known a technique of welding a work held by an arm robot to another work. In this case, a dedicated positioning mechanism (including positioning molds and jigs) is used to position the arm robot and thus the work held by the arm robot (detect the position and posture). However, since the positioning mechanism must be provided separately, the equipment cost is high and a large installation space for the equipment is required. Moreover, since one positioning mechanism is required for each vehicle model, when there are many vehicle models, the number of changeover operations increases, leading to a long total changeover time. In addition, the same problem occurs every time a new vehicle model is launched.

[0003] Patent Document 1 describes a method for positioning at least one object in a space to a final posture by an industrial robot. In this method, a first industrial robot, a first optical imaging device, and at least one second optical imaging device are used to position the object. The first industrial robot can be adjusted to a predetermined positioning position. The first optical imaging device is calibrated in a three-dimensional spatial coordinate system and positioned at a known first position in a known orientation. The second optical imaging device is calibrated in the spatial coordinate system and positioned at a known second position in a known orientation.

[0004] The first optical imaging device comprises a first camera, a first drive unit, and a first angle measurement unit. The first camera is optically calibrated for capturing images within a predetermined first field of view. The first drive unit is a component for orienting the first camera to adjust the position of the first field of view. The first angle measurement unit is calibrated in the spatial coordinate system for detecting the angular orientation of the first camera and determining the first field of view in the spatial coordinate system.

[0005] The second optical imaging device comprises a second camera, a second drive unit, and a second angle measuring unit. The second camera is optically calibrated to capture an image within a predetermined second field of view. The second drive unit is a component for orienting the second camera to adjust the position of the second field of view. The second angle measuring unit is calibrated in a spatial coordinate system to detect the angular direction of the second camera and determine the second field of view in the spatial coordinate system. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Patent No. 5290324 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, Patent Document 1 had room for improvement because the equipment required for object positioning (detection of position and orientation) was large-scale (first optical imaging device (first camera, first drive unit, and first angle measuring unit), second optical imaging device (second camera, second drive unit, and second angle measuring unit)), and the positioning method (method for detecting position and orientation) was not simple.

[0008] This invention was made based on the above-mentioned concerns, and aims to provide an object detection method and detection device that can detect objects in a simple manner and configuration. [Means for solving the problem]

[0009] The object detection method of this embodiment is Steps include: using a first position information generation device to irradiate a shape marker as a first object with light and measuring the position coordinates and angle of the shape marker as a first object, which are three-dimensional position information covering a relatively wide measurement range, based on the reflected light from the shape marker as a first object; using a second position information generation device separate from the first position information generation device to image the shape marker as a first object and measuring the position coordinates and angle of the shape marker as a first object, which are point cloud information covering a relatively narrow measurement range; and based on the position coordinates and angle of the origin of the shape marker as a first object from the origin of the first position information generation device and the position coordinates and angle of the origin of the second position information generation device from the origin of the shape marker as a first object, the second position information generation device performs the relative The method is characterized by comprising the steps of: calculating the position coordinates and angle of the shape marker as the first object, which is point cloud information covering a relatively narrow measurement range; using the position coordinates and angle of the shape marker as the first object, which is point cloud information covering a relatively narrow measurement range, the second position information generation device uses the position coordinates and angle of the shape marker as the first object, which is point cloud information covering a relatively narrow measurement range, with the origin of the first position information generation device as the reference point, to image a second object different from the first object, and to measure the position coordinates and angle of the second object, which is point cloud information covering a relatively narrow measurement range; and removing the first position information generation device, and using the origin of the first position information generation device as a fictitious origin, the second position information generation device uses the second position information generation device to image a second object, and to measure the position coordinates and angle of the second object, which is point cloud information covering a relatively narrow measurement range. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an object detection method and detection device that can detect an object in a simple manner and configuration. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of how to represent the information of each point in a point cloud. [Figure 2] This figure shows an example of how the origin can be represented in a 4x4 matrix. [Figure 3] This figure shows an example of origin calculation using a 4x4 matrix. [Figure 4] This figure shows a first example of the configuration of a welding apparatus for vehicle parts to which the object detection method and detection device of this embodiment are applied. [Figure 5] This figure shows a second example of the configuration of a welding apparatus for vehicle parts to which the object detection method and detection device of this embodiment are applied. [Figure 6] This figure shows a third example of the configuration of a welding apparatus for vehicle parts to which the object detection method and detection device of this embodiment are applied. [Figure 7] This figure shows the first example of a shape marker. [Figure 8] This figure shows a second example of a shape marker. [Figure 9] This figure shows an example of the preparatory process for performing final position correction using a 3D scanner and shape markers. [Modes for carrying out the invention]

[0012] First, we will define the terms used in this specification.

[0013] In this specification, "position information generation device (3D scanner)" refers to a device for acquiring shape information of a workpiece (e.g., upper sash, column sash, door frame, bracket, etc.), its gripping device (e.g., robot arm, etc.), and other targets (e.g., a robot arm or a marker or fixed object placed near it) as point cloud data. Each point in the point cloud data can be interpreted as "the position and orientation of the object." Each point in the point cloud data (the position and orientation of the object) can be represented by numerical values ​​of coordinates X, Y, Z and their normal directions I, J, K, with the 3D scanner origin as the origin. In this specification, "3D scanner" may be referred to as "second position information generation device."

[0014] In this specification, a "position information generating device (laser tracker)" refers to a device capable of calculating, for example, the three-dimensional information (position and angle) of a target based on its own coordinate system. The "position information generating device (laser tracker)" acquires the three-dimensional position information of a target by, for example, irradiating it with laser light, and observing the laser light reflected from the target returning to the light source. The three-dimensional information (position and angle) of the target can be interpreted as "the position and orientation of the object." Because the "position information generating device (laser tracker)" has a wide measurement range, if the "position information generating device (laser tracker)" is used with a target, or something like a marker-equipped touch probe described later, it can calculate the dimensions, precision, and origin position between each piece of equipment with high accuracy. In this specification, the "position information generating device (laser tracker)" may be referred to as the "first position information generating device."

[0015] As used in the present specification, the "marker" is a type of target for acquiring position coordinates and angles by, for example, a first position information generation device (laser tracker) or a second position information generation device (3D scanner). The position coordinates and angles of the "marker" are calculated with the first position information generation device (laser tracker) or the second position information generation device (3D scanner) as the origin. This can be represented by coordinates X, Y, Z and angles Rx, Ry, Rz. As will be described later, the angles Rx, Ry, Rz are respectively unit vectors (I, J, K) corresponding to posture changes from the reference X axis, reference Y axis and reference Z axis, and can be represented by 4×4 matrix data using the movement amounts (X, Y, Z) from the reference point. The "marker" is, for example, attached to a gripping portion or the like at the tip of a robot arm, and is used to acquire, as a numerical value, the operation accuracy (the absolute position of the gripping portion in three-dimensional space) of the robot arm with the first position information generation device (laser tracker) or the second position information generation device (3D scanner) as the origin. In addition, the "marker" is also widely used in other forms, such as a marked touch probe that measures a target workpiece by contact as described later, and a type attached to a 3D scanner that measures a target workpiece in a non-contact manner. In any measurement information, the marker is used for positioning with the origin of the first position information generation device (laser tracker) or the second position information generation device (3D scanner).

[0016] As used in the present specification, the "marked touch probe" performs measurement by touching the tip of the probe against a target workpiece in a state where the target portion of the "marked touch probe" is locked on by the first position information generation device (laser tracker) or the second position information generation device (3D scanner), whereby the position of the target workpiece in three-dimensional space with the first position information generation device (laser tracker) or the second position information generation device (3D scanner) as the origin can be calculated.

[0017] In this specification, a "3D scanner with a marker" refers to, for example, a 3D scanner with a marker that has been locked onto the target area of ​​the "3D scanner with a marker" using a first position information generation device (laser tracker). By 3D scanning the target workpiece, it is possible to calculate the origin of the first position information generation device (laser tracker) from the marker origin coordinates / angle from the origin of the first position information generation device (laser tracker) and the origin coordinates / angle of the second position information generation device (3D scanner) from the marker origin (described later). The relationship between the marker origin position and the origin position of the second position information generation device (3D scanner) is determined at the time of installation by calculating the position information of a reference block using the first position information generation device (laser tracker) and a marker-equipped touch probe, and then matching (calibrating) the position information of the reference block imaged by the second position information generation device (3D scanner) to calculate the origin / angle information of the second position information generation device (3D scanner). Typically, the second position information generator (3D scanner) has the characteristic of decreasing accuracy when the imaging range is widened, making it unsuitable for imaging wide areas, and there are limitations to using the second position information generator (3D scanner) alone for the operation of the entire facility as described later. For this reason, it is conceivable to use the first position information generator (laser tracker) to widen the imaging range and measurement range. However, as described later, in the object detection method and detection device of this embodiment, by appropriately combining the first position information generator (laser tracker) and the second position information generator (3D scanner), the first position information generator (laser tracker) can be used only for initial setup (calibration), and then the final position correction can be performed using the second position information generator (3D scanner) and shape marker. For this reason, for example, it becomes unnecessary to install a dedicated first position information generator (laser tracker) in each of multiple manufacturing lines, and the first position information generator (laser tracker) can be reused in multiple manufacturing lines.

[0018] As used herein, the term "point cloud data" refers to three-dimensional information of a workpiece obtained by three-dimensionally scanning (imaging) a workpiece (e.g., an upper sash, a vertical post sash, a door frame, a bracket, etc.) using a device such as a 3D scanner (camera), for example. Each point information of point cloud data can be represented by numerical values of coordinates X, Y, Z and their normal directions I, J, K (Figure 1), as shown in Figure 1, for example. In the example of Figure 1, point cloud 1, point cloud 2, point cloud 3, point cloud 4, point cloud 5, and point cloud** are illustrated.

[0019] As used herein, the term "reference data" refers to data that serves as a design reference for a workpiece (e.g., an upper sash, a vertical post sash, a door frame, a bracket, etc.) (e.g., master workpiece data, design value data, raw data, CAD data).

[0020] As used herein, the term "shape-aligned point cloud data" refers to point cloud data obtained by aligning the above-mentioned "point cloud data" with "reference data".

[0021] As used herein, "movement matrix" and "inverse matrix" are represented by 4×4 matrix data, for example. As shown in Figure 2, for example, "movement matrix" and "inverse matrix" are represented as information on the origin and the position from a reference origin to a target origin. The first column from the left represents unit vectors (I, J, K) for the posture change from the reference X axis; the second column from the left represents unit vectors (I, J, K) for the posture change from the reference Y axis; the third column from the left represents unit vectors (I, J, K) for the posture change from the reference Z axis; and the fourth column from the left represents the movement amounts (X, Y, Z) from the reference point.

[0022] Using a matrix, it is possible to calculate the amount of movement for each coordinate using a matrix, as shown in Figure 3. By performing this matrix calculation, the position and angle quantities for each coordinate system can be calculated. The movement when aligning the point cloud data with the reference data, as described above, can also be represented by 4x4 matrix data. Furthermore, it is possible to output 4x4 matrix data representing the movement when aligning the reference data with the point cloud data by performing an inverse matrix transformation on this 4x4 matrix data. In Figure 3, the origin reference, origin A as seen from the origin reference, origin A as seen from origin B, origin B as seen from the origin reference, and the inverse matrix of origin B are each shown as 4x4 matrix data.

[0023] Figures 4 and 5 show first and second examples of the configuration of a welding apparatus 1 for vehicle parts to which the object detection method and detection device of this embodiment are applied.

[0024] The welding apparatus 1 has a fixing jig (for example, a jig for the tact system) 10 located in the center of the apparatus. The fixing jig 10 supports a door frame 11 in a fixed state, for example, by welding the end of the upper sash (front main frame) and the end of the vertical sash (front vertical frame) in an aligned state. The relative positional relationship between the fixing jig 10 and the door frame 11 may change each time the workpiece is attached or detached, so the repeatable positioning function required of a normal jig is unnecessary (because the workpiece position of the door frame 11 is determined by the inverse matrix described later). For this reason, the fixing jig 10 may fix any position on the door frame 11. For the sake of drawing convenience, Figure 4 shows the state in which the fixing jig 10 supports the door frame 11, and Figure 5 shows the state in which the fixing jig 10 does not support the door frame 11 (the door frame 11 is omitted in the drawing).

[0025] The welding apparatus 1 has a robot arm 20 and a robot arm 30 located on the left and right sides of the fixed jig 10 in the figure, respectively. The robot arm 20 is composed of, for example, an arm robot having multiple axes (e.g., 6 axes), and moves (moves) to align a lock bracket (shown as a callout in Figure 4) gripped by a gripping part 21 at the tip of the robot arm 20 to a predetermined part of the door frame 11. A position correction marker 22 is also provided near the gripping part 21 at the tip of the robot arm 20. The position correction marker 22 has, for example, a reflector that can be tracked by a first position information generation device (laser tracker) 70, which will be described later. The robot arm 30 is composed of, for example, an arm robot having multiple axes (e.g., 6 axes), and moves (moves) to align a hinge bracket (shown as a callout in Figure 4) gripped by a gripping part 31 at the tip of the robot arm 30 to a predetermined part of the door frame 11. Furthermore, a position correction marker 32 is provided near the gripping portion 31 at the tip of the robot arm 30. The position correction marker 32 has a reflector that can be tracked by, for example, the first position information generation device (laser tracker) 70, which will be described later.

[0026] The relative positional relationship between the robot arm 20 and the lock bracket (shown by a callout in Figure 4), and the relative positional relationship between the robot arm 30 and the hinge bracket (shown by a callout in Figure 4) may change each time a workpiece is attached or detached. Therefore, the repeatable positioning function required for a typical jig is unnecessary (because the workpiece gripping positions of the lock bracket and hinge bracket are determined by the inverse matrix described later). Thus, the robot arm 20 may grip any position on the lock bracket, and the robot arm 30 may grip any position on the hinge bracket. If the relative positional accuracy of the fixing jig 10 and the door frame 11, the relative positional accuracy of the robot arm 20 and the lock bracket, and the relative positional accuracy of the robot arm 30 and the hinge bracket are guaranteed, and the absolute positional accuracy of the robot spatial coordinates is guaranteed, then high-precision positioning is possible with a device (for example, a machine tool such as a machining center).

[0027] The welding apparatus 1 has a welding robot 40 positioned offset in the depth direction between the fixing jig 10 and the robot arm 20, and a welding robot 50 positioned offset in the depth direction between the fixing jig 10 and the robot arm 30. The welding robot 40 is composed of, for example, an arm robot having multiple axes (e.g., 6 axes), and uses a welding nozzle 41 provided at its tip to weld the door frame 11 supported by the fixing jig 10 and the lock bracket gripped by the gripping part 21 of the robot arm 20 in a predetermined alignment state. The welding robot 50 is composed of, for example, an arm robot having multiple axes (e.g., 6 axes), and uses a welding nozzle 51 provided at its tip to weld the door frame 11 supported by the fixing jig 10 and the hinge bracket gripped by the gripping part 31 of the robot arm 30 in a predetermined alignment state.

[0028] The welding apparatus 1 has a 3D scanner support arm 60 positioned behind the fixing jig 10 and sandwiched between the welding robot 40 and the welding robot 50. The 3D scanner support arm 60 is composed of, for example, an arm robot having multiple axes (e.g., 6 axes), and a second position information generation device (3D scanner) 61 and a position correction marker 62 are provided at the tip of the arm. The position correction marker 62 has, for example, a reflector that can be tracked by a first position information generation device (laser tracker) 70, which will be described later.

[0029] The welding apparatus 1 may be used in combination with a first position information generating device (laser tracker) 70. The first position information generating device (laser tracker) 70 targets each component of the welding apparatus 1 (for example, the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, the hinge bracket gripped by the robot arm 30, the position correction marker 22 for the robot arm 20, the position correction marker 32 for the robot arm 30, the 3D scanner 61, and the position correction marker 62 for the 3D scanner 61) and calculates its three-dimensional information (position information and angle) based on its own coordinate system.

[0030] As shown in Figure 5, the welding apparatus 1 may be used in combination with a shape marker (fixed object, metal block) 80 supported on a support base 81. In Figure 5, for ease of drawing, the shape marker 80 is depicted as a simplified cube, but the specific configuration and function of the shape marker 80 will be described in detail later. The shape marker 80 works in cooperation with a first position information generation device (laser tracker) 70 and a second position information generation device (3D scanner) 61. The first position information generation device (laser tracker) 70 is used only for initial setup (calibration), and thereafter the second position information generation device (3D scanner) 61 scans (images) the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30.

[0031] In this embodiment, a marker-equipped 3D scanner 61 is used to scan (image) the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30, thereby acquiring point cloud data (three-dimensional information) of these objects. In other words, the marker-equipped 3D scanner 61 functions as a "point cloud data acquisition unit". By acquiring the point cloud with the marker-equipped 3D scanner 61 virtually locked on by the first position information generation device (laser tracker) 70, the point cloud data acquired by the marker-equipped 3D scanner 61 is calculated with the first position information generation device (laser tracker) 70 as the origin (calculated as a fictitious origin associated with the virtual lock-on of the first position information generation device (laser tracker) 70). This point cloud data is represented, for example, by the coordinates X, Y, Z of each point and their normal directions I, J, K, and the origin of the point cloud data is represented by 4x4 matrix data (details will be described later).

[0032] Furthermore, when acquiring point cloud data, instead of 3D scanning the entire door frame 11 supported by the fixed jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30, it is also possible to 3D scan only a part of the door frame 11 supported by the fixed jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30, for example, only the part that serves as a reference for product evaluation. This 3D scanning can be done by acquiring point cloud data while moving a 2D scanner, by positioning a 3D scanner and scanning, or by installing fixed 3D scanners in multiple locations and scanning. As will be described later, when aligning point cloud data with reference data, it is possible to set any number of arbitrary positions on the reference data as references and perform alignment. In addition, a priority can be set for each of these references, so that alignment of the more important parts can be prioritized. This makes it possible to control the alignment method between the reference data and the point cloud data. In this embodiment, if the error amount of each reference part is large, there is a possibility that the welded parts of the lock bracket gripped by the robot arm 20 and the hinge bracket gripped by the robot arm 30 will interfere with each other with the door frame 11. Therefore, by setting the priority of the welded parts higher than that of the other reference parts, it is possible to prioritize positioning by matching the shape of the welded parts.

[0033] Furthermore, in this embodiment, reference data (e.g., CAD data) that serves as the design basis for the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30 is prepared in advance. This reference data includes the origin (e.g., the vehicle origin, in this case the vehicle origin = the position information generation device origin) and can also be data representing the alignment state of the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30. Then, in this embodiment, shape-aligned point cloud data is calculated by aligning the point cloud data with the reference data including the origin (vehicle origin) using a control unit which will be described later. The amount of movement when the point cloud data is moved to the reference data, and the origins of the reference data and shape-aligned point cloud data are represented by 4x4 matrix data (details will be described later).

[0034] The origin coordinates of the shape-aligned point cloud data are equivalent to moving the origin of the point cloud data acquired at the origin of the position information generation device. By performing an inverse matrix transformation on the 4x4 matrix data (matrix) used when aligning the point cloud data to this reference data, it is equivalent to aligning the reference data to the point cloud data, thus assigning an origin to the point cloud data (the state of the shape-aligned point cloud data, which was represented by the origin of the reference data, is now represented based on the point cloud data). This method makes it possible to assign CAD origin information to point cloud data that does not have an origin.

[0035] The result of moving the point cloud data of the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30 to match the CAD data (reference data) can be represented (i.e., equivalent) by moving the origin. By converting the matrix to an inverse matrix, the amount of movement required to match the CAD data (reference data) to the point cloud data can be determined. Matching the point cloud data to the CAD data (reference data) corresponds to the positive matrix, and matching the CAD data (reference data) to the point cloud data corresponds to the inverse matrix (which corresponds to returning to the origin from the perspective of the positive matrix). In other words, from the perspective of the origin, the state in which the CAD data (reference data) and the point cloud data were aligned has been moved so that the CAD coordinates match the point cloud data. For example, if the actual workpiece placement of a portion of the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30 can be reproduced in the CAD coordinate system, then the positions of that portion become known, and calculations between the origins can be performed by calculating the matrix of the other parts of the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30 (in this case, inverse matrix transformation is unnecessary).

[0036] The first position information generator (laser tracker) 70 assists in the alignment (micro-movement) of the lock bracket gripped by the robot arm 20 and the hinge bracket gripped by the robot arm 30 by constantly monitoring (tracking) the position of the position correction markers 22 and 32. The robot arms 20 and 30 are pre-calibrated by the first position information generator (laser tracker) 70 and the position correction markers 22 and 32, improving their absolute accuracy in spatial coordinates.

[0037] When measuring the position correction markers 22 and 32 attached to the robot arms 20 and 30, the vehicle origin of the lock bracket gripped by robot arm 20 and the hinge bracket gripped by robot arm 30 is replaced with the marker, allowing the first position information generation device (laser tracker) 70 to constantly monitor the three-dimensional coordinates. Furthermore, the difference between the marker target position and the current position is used to correct the final position, thereby forcibly improving the spatial coordinate accuracy of the robot.

[0038] Figure 6 shows a third example of the configuration of a welding apparatus 1 for vehicle parts to which the object detection method and detection device of this embodiment are applied. Figure 6 illustrates functional components for performing motion (movement) control of robot arms 20 and 30 (gripping parts 21 and 31) based on the point cloud data, reference data, and shape matching point cloud data described above. Each functional component constitutes a part of the control unit 90, which is composed of a CPU (Central Processing Unit).

[0039] The control unit 90 includes a point cloud data acquisition unit 91, a reference data acquisition unit 92, a movement matrix calculation unit 93, an inverse matrix calculation unit 94, a target movement matrix calculation unit 95, and a robot arm control unit 96.

[0040] The point cloud data acquisition unit 91 acquires (receives input) point cloud data (three-dimensional information) of the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30, using the first position information generation device (laser tracker) 70 and the second position information generation device (3D scanner) 61. This point cloud data may be point cloud data with the first position information generation device (laser tracker) 70 as the origin (fictitious origin).

[0041] The reference data acquisition unit 92 acquires (stores) reference data (e.g., CAD data) that serves as the design standard for the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30. This reference data includes the origin (e.g., the vehicle origin, in this case the vehicle origin = the position information generation device origin) and can also be data on the alignment state of the door frame 11 supported by the fixing jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30.

[0042] The movement matrix calculation unit 93 calculates a movement matrix for aligning point cloud data to reference data representing the alignment state of the door frame 11 supported by the fixed jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30. This movement matrix calculation yields shape-aligned point cloud data obtained by aligning the point cloud data to the reference data based on the movement matrix.

[0043] The inverse matrix calculation unit 94 performs an inverse matrix transformation on the 4x4 matrix data (matrix) used to align the point cloud data with the reference data obtained by the movement matrix calculation unit 93. This transformation adds an origin to the aforementioned point cloud data (the state of the shape-aligned point cloud data, which was represented by the origin of the reference data, is now represented by the point cloud data as the reference). This method enables the addition of CAD origin information to point cloud data that does not have an origin.

[0044] The target movement matrix calculation unit 95 can use an inverse matrix to calculate the amount of movement from the origin of the lock bracket and the hinge bracket to the origin of the other, starting from a state in which an origin (vehicle origin) has been assigned to each of the door frame 11 supported by the fixed jig 10, the lock bracket gripped by the robot arm 20, and the hinge bracket gripped by the robot arm 30.

[0045] The robot arm control unit 96 positions the lock bracket and hinge bracket on the door frame 11 by moving the lock bracket with the robot arm 20 and the hinge bracket with the robot arm 30, based on at least one of the movement matrix and the inverse matrix. More specifically, the robot arm control unit 96 makes small movements of each axis (6 axes) of the robot arms 20 and 30, for example, until the lock bracket and hinge bracket, which are to be positioned, reach the target position from their initial or current position. The command value for the amount of this small movement is the difference between the current position and the target position of the position correction markers 22 and 32 as seen from the first position information generation device (laser tracker) 70. By setting the numerical value of this small movement amount by coefficient adjustment and the number of movements, positioning that does not depend on the accuracy of the robot is made possible. In other words, in this embodiment, the positioning of the workpieces (lock bracket and hinge bracket) is performed in two stages: (1) movement of the workpiece (gripping part) based on at least one of the movement matrix and the inverse matrix (for example, movement from one origin to the other origin based on the origins added to the lock bracket and hinge bracket by the inverse matrix), and (2) movement of the lock bracket and hinge bracket based on the position correction markers 22 and 32 (corrected positioning by the position correction markers 22 and 32, which corrects the operational error in (1)). With the lock bracket and hinge bracket positioned by the robot arm control unit 96, the welding parts of both are welded by the welding robots 40 and 50.

[0046] The functions of the control unit 90 can also be described as follows: The control unit 90 acquires three-dimensional data (e.g., point cloud data) and relative position data (e.g., reference data) of the hand and the workpiece. The control unit 90 performs spatial positioning processing to best fit the acquired three-dimensional data (e.g., point cloud data) and relative position data (e.g., reference data). The control unit 90 outputs coordinate values ​​for the movement of the workpiece by the hand. The control unit 90 performs motion (movement) control of the hand and the workpiece by 6-axis calculation based on robot inverse kinematics.

[0047] The positioning process in this embodiment is performed, for example, by the following processing steps. First, the point cloud data of the lock bracket and hinge bracket are aligned with CAD data (reference data), and the amount of movement is output as a 4x4 matrix data (matrix data), which is a movement matrix. This provides shape-matched point cloud data for the lock bracket and hinge bracket, with the point cloud data aligned with the reference data based on the movement matrix. Next, the point cloud data of the door frame 11 is aligned with CAD data (reference data), and the amount of movement is output as a 4x4 matrix data (matrix data), which is a movement matrix. This provides shape-matched point cloud data for the door frame 11, with the point cloud data aligned with the reference data based on the movement matrix. Finally, an origin is assigned to the point cloud data by performing an inverse matrix operation (inverse matrix operation) from the calculated movement matrix. In other words, the origin of one is aligned with the origin of the other. Alternatively, the origin of the reference data (vehicle origin) may be moved along with the shape-matching point cloud data, and the moved origins of the lock bracket, hinge bracket, and door frame 11 may be aligned. Furthermore, coordinate values ​​(e.g., CAD coordinates) are assigned to the point cloud data of the lock bracket, hinge bracket, and door frame 11 using the calculated inverse matrix. Then, the amount of movement of the point cloud data is calculated in a coordinate system based on the origin (vehicle origin), and the lock bracket and hinge bracket are positioned on the door frame 11 by moving the robot arms 20 and 30 that grip the lock bracket and hinge bracket based on the calculated amount of movement. In this embodiment, since the door frame 11 is fixed and the lock bracket and hinge bracket are movable, the amount of movement corresponds to the amount of movement of the lock bracket and hinge bracket relative to the door frame 11 (the amount of movement from the CAD coordinates of the lock bracket and hinge bracket to the CAD coordinates of the door frame 11). Finally, based on the calculated amount of movement of the lock bracket and hinge bracket relative to the door frame 11, the lock bracket and hinge bracket are moved relative to the door frame 11, thereby positioning them relative to the door frame 11.

[0048] The process for aligning point cloud data to reference data in this embodiment includes, for example, a forced numerical movement step that moves the point cloud information by a forced numerical value (quantitative value), a best-fit step for overall shape alignment that aligns the entire point cloud with the overall CAD shape, and a reference alignment step that performs detailed alignment according to a reference part, for example, a part that serves as the basis for product evaluation.

[0049] Incidentally, for example, it is conceivable to correct the final position of the lock bracket and hinge bracket relative to the door frame 11. Conventionally, this final position correction has been performed by calculating the position and orientation using a combination of a first position information generation device (laser tracker) and a position correction marker for the robot arm. That is, the distance r from the first position information generation device (laser tracker) to the reflector part (point) of the position correction marker is determined by capturing the reflector part (point) of the position correction marker with the first position information generation device (laser tracker), and the polar coordinates to the reflector part relative to the first position information generation device (laser tracker) are determined by obtaining the left-right rotation angle θ and up-down rotation angle φ of the head of the first position information generation device (laser tracker) from an encoder, and these polar coordinates are converted to Cartesian coordinates (x, y, z). For example, distance can be measured using a laser displacement meter, the laser displacement meter can be installed on a two-axis rotary table with orthogonal axes, the rotation angle encoder value of the two-axis rotary table that the laser displacement meter is facing can be output, and the obtained polar coordinates can be converted to Cartesian coordinates. Furthermore, attitude information (Rx, Ry, Rz) is obtained by photographing multiple (e.g., 11) LEDs provided on the position correction marker using a first position information generation device (laser tracker). The camera function of the first position information generation device (laser tracker) is used to confirm the arrangement of the multiple LEDs, and the attitude is calculated from the arrangement of the multiple LEDs.

[0050] However, while the first position information generation device (laser tracker) is highly accurate, it is also very expensive, and the dedicated markers, being optional accessories for the first position information generation device (laser tracker), tend to be expensive as well. Therefore, if, for example, a dedicated first position information generation device (laser tracker) and position correction markers are installed for each of multiple manufacturing lines or pieces of equipment, it becomes unavoidable that the overall cost of the manufacturing line will increase.

[0051] Therefore, in this embodiment, for parts where there is no significant change in the final position, such as the final position correction of the lock bracket and hinge bracket on the door frame 11, the final position correction is performed using a combination of a second position information generation device (3D scanner) 61 and a shape marker 80, without using a first position information generation device (laser tracker) and a dedicated marker. This eliminates the need to provide a dedicated first position information generation device (laser tracker) and position correction marker for each of the multiple manufacturing lines, and the 3D scanner can be used in common across multiple manufacturing lines. Furthermore, for equipment with multiple positioning processes that were waiting for measurement by the first position information generation device (laser tracker), by arranging multiple second position information generation devices (3D scanners), it becomes possible to generate position information and perform positioning calculation processing simultaneously for multiple processes. As a result, it is possible to reduce costs and increase speed for each manufacturing line, and ultimately for the entire manufacturing line.

[0052] Figures 7A to 7D show a first example of a shape marker (fixed object, metal block) 80. Figures 7A to 7D show that the shape marker 80 includes the shape of a triangular pyramid. The shape marker 80 has a base portion 82 that forms a right triangle shape when viewed from the plane of reference numeral 83a (for example, when viewed from a direction perpendicular to the plane of reference numeral 83a), three side portions 83 extending from the center of the right triangle shape in plan view, and a top portion 84 that is a right triangle shape when viewed from the plane of reference numeral 83a (for example, when viewed from a direction perpendicular to the plane of reference numeral 83a), obtained by cutting out the aggregate portion of the three side portions 83. The base portion 82 and the top portion 84 have similar right triangle shapes when viewed from a plan view. A part of the plane 83a may be provided to identify which coordinate axis is pointing where during coordinate extraction, which will be described later. Alternatively, any means of identifying the orientation may be used, such as color-coding some of the faces to identify the coordinate axes. In other words, including a triangular pyramidal shape also includes shapes having a vertex portion 84, as shown in Figures 7A to 7D.

[0053] Figure 8 shows a second example of a shape marker (fixed object, metal block) 80. In Figure 8, the shape marker 80 includes the shape of three spheres 85. The position of an object can be determined by one of the three spheres 85, and the orientation of the object can be determined by all three. By connecting the points defined by the three spheres 85, the position and orientation of the object in spatial coordinates (3D coordinates) based on mutually orthogonal X, Y, and Z axes can be determined. The arrangement of the three spheres may be such that each has different dimensions in order to identify which coordinate axis is pointing where during coordinate extraction, which will be described later. Alternatively, any means of identifying the orientation may be used, such as color-coding some of the faces to identify the coordinate axes.

[0054] The shapes of the shape markers 80 shown in Figures 7A to 7D and Figure 8 are merely examples, and various design modifications are possible. The shape marker 80 only needs to be able to determine the position and orientation of an object in spatial coordinates (3D coordinates) based on mutually orthogonal X, Y, and Z axes. For example, for the markers including the triangular pyramid shape shown in Figures 7A to 7D, the orientation of the shape marker 80 can be determined by calculating the intersection line of the mean planes of the point clouds 83a and 83b as the X-axis, calculating the intersection line of the mean planes of the point clouds 83b and 83c as the Y-axis, and calculating the remaining Z-axis from the relationship between the X and Y axes. If the shape marker 80 rotates during this process, it may become impossible to determine the orientation of the X, Y, and Z axes from a calculation perspective. Therefore, by defining a shape difference that identifies the orientation of the marker, such as making that part the Z-axis direction, the orientation of the shape marker 80 can be accurately determined even if it rotates. The position can be calculated from the intersection of the mean planes of the point clouds 83a, 83b, and 83c. In this case, the position information can also be used by projecting this intersection onto the top surface 84. A characteristic of 3D scanning is that point cloud information obtained perpendicular to the surface is more accurate than that obtained when scanning an inclined surface. Therefore, by including this process, it is possible to reduce positional errors. In the case of the spheres in Figure 8, instead of evenly arranging the spheres, by creating differences in the dimensions between the three spheres 85a, 85b, and 85c, it is possible to determine the direction of the X, Y, or Z axis and thus determine the orientation of the shape marker 80. Furthermore, the three sphere spacing dimensions of 85a, 85b, and 85c offer flexibility, allowing for various design modifications.

[0055] The shape marker 80 may also be provided on the tip side of the robot arms 20 and 30 that grip the workpiece (e.g., lock bracket, hinge bracket). In this case, the shape marker 80 may be provided in place of / in addition to the position correction markers 22 and 32 of the robot arms 20 and 30. Furthermore, the shape marker 80 may also be provided at other predetermined locations.

[0056] Figure 9 shows an example of the preliminary preparations for performing the final position correction process using a 3D scanner 61 and shape markers 80. First, the robot position is measured and calibrated using a first position information generation device (laser tracker) 70. Position correction markers 22 and 32 are used in this system processing stage.

[0057] As shown in Figure 9, the position and orientation of the shape marker 80 as the first object are measured using the first position information generation device (laser tracker) 70 (A). Any means is acceptable for this measurement, such as measuring with a reflector, measuring with a marker-equipped touch probe, or measuring with a marker-equipped 3D scanner, as long as the position and orientation of the shape marker 80 can be determined in the coordinate system of the first position information generation device (laser tracker). The position and orientation of the shape marker 80 as the first object are measured using the second position information generation device (3D scanner) 61 (B). Based on the positional relationship between the position and orientation of the shape marker 80 as the first object measured by the first position information generation device (laser tracker) 70 (A) and the position and orientation of the shape marker 80 as the first object measured by the second position information generation device (3D scanner) 61 (B), the position and orientation of the first position information generation device (laser tracker) 70 and the second position information generation device (3D scanner) 61 relative to each other are calculated (C). In other words, data captured by the second position information generation device (3D scanner) 61 appears as a point cloud at the origin of the second position information generation device (3D scanner) 61, and the position and orientation of the second position information generation device (3D scanner) 61 from the origin of the first position information generation device (laser tracker) 70 are determined by aligning the coordinates extracted from the point cloud. Then, by calculating the position and orientation of the first position information generation device (laser tracker) 70 and the second position information generation device (3D scanner) 61 relative to the other, the position and orientation of the second object from the origin of the first position information generation device (laser tracker) 70 can be detected using the second position information generation device (3D scanner) 61. Here, the second object can be, for example, a marker provided on the tip side of a robot arm 20, 30 that grips a workpiece (for example, a marker 80' of a different shape than the one shown in Figure 4, not shown), or any other predetermined part of any object to be detected. In other words, at least one of the first object and the second object may include the shape of a triangular pyramid (see Figure 7) or the shape of three spheres (see Figure 8) of the shape markers 80, 80'.

[0058] By calculating the position and orientation of the first position information generator (laser tracker) 70 relative to the other, the second position information generator (3D scanner) 61 is used to detect the position and orientation of the second object, which corresponds to generating the position and orientation moved during the final position correction of the lock bracket and hinge bracket relative to the door frame 11. In this final position correction process, the first position information generator (laser tracker) 70 is removed, and the position and orientation of the second object are measured by the second position information generator (3D scanner) 61 while using the origin of the first position information generator (laser tracker) 70 as a hypothetical origin. At that time, the second position information generator (3D scanner) 61 used for final position correction may be moved as needed to widen the imaging range. In this case, when using a device with a large error in absolute spatial coordinates, such as a 6-axis robot, the point cloud accuracy will be significantly reduced. Therefore, it is desirable to operate the second position information generating device (3D scanner) 61 with the fewest possible axes of high-precision linear motion or rotation. Of course, the second position information generating device (3D scanner) 61 may also be used in a fixed state.

[0059] The above series of processes are performed during the final position correction calibration. In other words, the first position information generator (laser tracker) 70 is only required during periodic calibration, and can be removed in other situations. Therefore, it is not necessary to install a dedicated first position information generator (laser tracker) 70 in each of the multiple manufacturing lines or equipment, and the first position information generator (laser tracker) 70 can be reused in multiple manufacturing lines or equipment.

[0060] As described above, the object detection method of this embodiment includes the steps of: measuring the position and orientation of a first object with a first position information generation device; measuring the position and orientation of the first object with a second position information generation device; calculating the position and orientation of one of the first and second position information generation devices relative to the other based on the positional relationship between the position and orientation of the first object measured by the first position information generation device and the position and orientation of the first object measured by the second position information generation device; and measuring the position and orientation of a second object with the second position information generation device using the position and orientation of one of the first and second position information generation devices relative to the other. This enables object detection with a simple method and configuration. By performing final position correction using a 3D scanner and shape markers, expensive laser trackers can be common across multiple manufacturing lines and equipment, eliminating the need to install individual laser trackers on each manufacturing line (laser trackerless), thereby reducing costs. [Explanation of Symbols]

[0061] 1. Welding equipment 10 Fixing fixtures 11. Door frame (second object) 20. Robot arm (second object) 21 Gripping part 22 Position correction marker 30. Robot arm (second object) 31 Gripping part 32 Position correction marker 40 Welding robots 41 Welding Nozzle 50 Welding Robots 51 Welding Nozzle 60 3D scanner support arm 61. Second location information generation device (3D scanner) 62 Position correction marker 70. First position information generating device (laser tracker) 80 Shape Markers (First Object, Fixed Object, Metal Block) 80' Shape marker (second object) 81 Support base 82 Base 83 Side part 84 Top part 85 Three balls 90 Control Unit 91 Point cloud data acquisition unit 92 Reference Data Acquisition Unit 93 Movement Matrix Calculation Unit 94 Inverse Matrix Calculation Unit 95 Target movement matrix calculation unit 96 Robot Arm Control Unit

Claims

1. A first position information generating device irradiates light onto a shape marker as a first object, and measures the position coordinates and angle of the shape marker as a first object, which are three-dimensional position information covering a relatively wide measurement range, based on the reflected light from the shape marker as a first object. The first position information generating device and a second position information generating device capture an image of the shape marker as the first object, and measure the position coordinates and angle of the shape marker as the first object, which are point cloud information covering a relatively narrow measurement range. A step of calculating the position coordinates and angle of the shape marker as the first object, which is point cloud information covering the relatively narrow measurement range, by the second position information generation device with respect to the origin of the first position information generation device, based on the position coordinates and angle of the origin of the shape marker as the first object from the origin of the first position information generation device, and the position coordinates and angle of the origin of the second position information generation device from the origin of the shape marker as the first object. Using the position coordinates and angle of the shape marker as the first object, which is point cloud information covering a relatively narrow measurement range, generated by the second position information generation device with respect to the origin of the first position information generation device, the second position information generation device images a second object, which is different from the first object, and measures the position coordinates and angle of the second object, which is point cloud information covering a relatively narrow measurement range. The first position information generating device is removed, and the second position information generating device uses the origin of the first position information generating device as a fictitious origin, and the second position information generating device images the second object, and measures the position coordinates and angle of the second object, which are point cloud information covering a relatively narrow measurement range. A method for detecting an object, characterized by having the following features.

2. The second position information generating device captures an image of the second object with the second position information generating device while the first position information generating device has locked onto the shape marker as the first object, thereby calculating the position coordinates and angles of the second object, which are point cloud information relating to the relatively narrow measurement range, with respect to the origin of the first position information generating device. The method for detecting an object according to feature 1.

3. At least one of the first object and the second object includes the shape of a triangular pyramid. A method for detecting an object according to claim 1 or 2, characterized by the features described above.

4. At least one of the first object and the second object includes the shape of three spheres, A method for detecting an object according to claim 1 or 2, characterized by the features described above.

5. The second object is provided on the tip side of the robot arm that grips the workpiece, A method for detecting an object according to any one of claims 1 to 4.

6. The first position information generating device is a laser tracker, The second location information generating device is a 3D scanner. A method for detecting an object according to any one of claims 1 to 5.

7. A first position information generating device that irradiates light onto a shape marker as a first object and measures the position coordinates and angle of the shape marker as a first object, which are three-dimensional position information covering a relatively wide measurement range, based on the reflected light from the shape marker as a first object, A first position information generating device and a second position information generating device, which are separate devices, image the shape marker as the first object and measure the position coordinates and angle of the shape marker as the first object, which are point cloud information covering a relatively narrow measurement range. It has, Based on the position coordinates and angle of the origin of the shape marker as the first object relative to the origin of the first position information generation device, and the position coordinates and angle of the origin of the second position information generation device relative to the origin of the shape marker as the first object, the position coordinates and angle of the shape marker as the first object, which are point cloud information covering the relatively narrow measurement range, are calculated by the second position information generation device with respect to the origin of the first position information generation device. Using the position coordinates and angle of the shape marker as the first object, which is point cloud information covering the relatively narrow measurement range, generated by the second position information generation device with respect to the origin of the first position information generation device, the second position information generation device images a second object separate from the first object, and measures the position coordinates and angle of the second object, which is point cloud information covering the relatively narrow measurement range. The first position information generating device is removed, and the second position information generating device uses the origin of the first position information generating device as a fictitious origin. The second position information generating device then images the second object and measures the position coordinates and angles of the second object, which are point cloud information covering a relatively narrow measurement range. A device for detecting objects characterized by the following features.

Images