Method and system for inspecting a repair or assembly operation

Abstract

A method for identifying an object on a workpiece surface is disclosed. An optical device identifies an object on the workpiece surface. An object identification system includes a light projector and a photogrammetry system. One of the light projector and the photogrammetry system generates a three-dimensional coordinate system in a workroom. One of the light projector and the photogrammetry system identifies a position of the workpiece surface in the three-dimensional coordinate system. A controller calculates a geometric position of an object on the workpiece surface in the three-dimensional coordinate system identified by the optical device. The controller sends a signal to the light projector to project a light beam onto the surface of the workpiece, and identifies the arrangement of the object located on the workpiece surface.

Description

[0001] Previous patent applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 089,866, filed on October 9, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention generally relates to a method for identifying objects on a surface of a workpiece placed in a work cell. More specifically, the present invention relates to using a laser projector to project a laser image onto the surface of the workpiece to identify objects, such as defects and the status of work performed by an operator on the object. Background Art

[0004] Significant progress has been made in the automation of automobile assembly and other mass production, etc., with various technologies implemented to assist operators by providing automated instructions for performing work functions. However, it has proven difficult to achieve automatic identification of defects that do not meet specification requirements while guiding the operator where to perform repairs or other work functions. One problem related to the automatic identification of objects such as defects is the high speed of mass production using, for example, moving assembly lines. A non-limiting example is the identification of paint defects at the end of assembly or just after the paint process. For example, once the body of a car is painted, it is transferred to a brightly lit inspection room via a moving assembly line. There, an inspector visually inspects the paint or body defects and repairs the defects or marks them for repair after the vehicle is fully assembled.

[0005] This process is difficult and prone to human error. Often, paint defects are very small, just particles in the clear coat, small undercuts of the paint, or pits. These defects are difficult to locate even for the most skilled operators, and they are never identified. This is further problematic because these defects are randomly distributed across the painted body. Furthermore, these defects are not tracked, which would provide the ability to identify trends. Therefore, there is a need to provide an automated inspection method for identifying paint defects on a body or other painted object. While vision technology that is capable of identifying paint defects exists, translating this information to an operator to make the necessary repairs has hampered the implementation of the technology. Known vision technology that is capable of identifying paint defects is located at a distance from the workshop where the defects are repaired. During the transfer of the body to the workshop, the location of the defect must be identified in a new coordinate system that is specific to the workshop, while addressing the challenges of high-volume production. Therefore, not only does the defect need to be identified, but the location needs to be signaled, and further, operator handling of the defect must be developed. Summary of the invention

[0006] The present invention discloses a method and system for identifying an object on the surface of a workpiece passing through a workshop. An optical device identifies an object on the surface of a workpiece. The optical device sends a signal to a controller indicating that the object has been identified. The controller sends a signal to an object recognition system, which includes a light projector and a photogrammetry system. The light projector or the photogrammetry system generates a three-dimensional coordinate system in the workshop. Alternatively, the light projector and the photogrammetry system collaborate to generate a three-dimensional coordinate system in the workshop. Once the three-dimensional coordinate system is established, the light projector identifies the position of the workpiece surface in the three-dimensional coordinate system. The photogrammetry system can also be used to locate the surface of the workpiece in the three-dimensional coordinate system.

[0007] The controller calibrates the position of the workpiece surface with the identified objects in the three-dimensional coordinate system. Once calibrated, the controller calculates the geometric position of the objects on the workpiece surface in the three-dimensional coordinate system determined by the optical device. The controller then sends a signal to the light projector to scan a light beam onto the workpiece surface to identify the arrangement of the objects arranged on the workpiece surface.

[0008] The system of the present invention allows the operator to focus on repairing paint or other defects without first locating the defects for the first time. In addition, since the system identifies the placement of defects, the location and whether the defects have been repaired are also provided by visualization of the placement on the workpiece surface.

[0009] One way to achieve this is to display assembly task information on the workpiece surface. This can be accomplished by projecting a raster image onto the workpiece surface using a conventional video projector, but this may be limited by the relative brightness, accuracy and precision of the display, and the flexibility of interaction with the operator. Alternatively, a vector display of laser points projected from a laser source sequentially tracked through a pattern can provide unique capabilities. For example, if a 1m 2 The grating projector in the area provides the projection source, and the 1mW laser spot is guided to the 1mm 2 The area will require a kilowatt projection source.

[0010] Alternatively, conventional raster laser technology projects a laser image with sub-millimeter accuracy over an area covering multiple square meters. The primary requirement for clear laser projection is to provide a clear, flicker-free laser projection optimized for precise placement of the laser image on the workpiece surface. To provide the required accuracy, the position of the workpiece is typically identified with precisely measured reference targets, which the projection system can use to establish the relative position required to provide the required precision and accuracy.

[0011] The method of the present application does not necessarily require the precision and structure associated with layup template projection. The tasks of the present invention can be completed in an order selected by the operator. Often, multiple operators may act together to complete the overall task, for example, verifying that defects in the vehicle paint have been identified and repaired. Therefore, rather than performing the project steps in sequence, a large number of potential operations are presented simultaneously to allow completion in any order. However, it is still necessary to ensure that the task is completed and all fasteners have been inserted or all defects in the paint finish have been repaired. Therefore, in this case, traditional template laser projection methods are not sufficient.

[0012] As will be further explained below, it is believed that various laser projection technologies are feasible for meeting these demanding standards, even if they are not necessarily constructed to perform the process of the present invention. One such example is U.S. Patent No. 9,200,899, "Laser Projection System and Method," which discloses various techniques for positioning objects in a three-dimensional coordinate system using a collaborative photogrammetry system and a laser projector. Another is U.S. Patent No. 10,052,734, "Laser Projector with flash Alignment," which discloses a technique for quickly positioning a target to establish a geometric relationship between a workpiece and a laser projector. Still further is U.S. Patent No. 9,881,383, "Laser Projection System with Motion Compensation and Method," which discloses a technique for detecting the dynamic motion of an object in a three-dimensional coordinate system. Another is U.S. Patent No. 9,245,062, "Laser Projection System Using Variable Part Alignment," which discloses improved accuracy when projecting laser images onto workpieces that were not manufactured as designed. For the sake of brevity of this application, the contents of each of these patents are included herein by reference.

[0013] Rather than performing the traditional step-by-step process disclosed in these references, a new process for laser identification and location of various objects on a workpiece surface is disclosed. The establishment of a random sequence provides the operator with the ability to determine the order of the sequence. In addition, multiple operators are provided with the ability to perform work functions simultaneously in no particular sequential order. All of this is coordinated by the controller and object identification system of the present invention while verifying that the task has been completed and notifying the operator of such completion.

[0014] In the system of the present invention, visibility of the laser projected image becomes more relevant than clarity. Thus, even when performing tasks in a brightly lit workroom like a typical inspection room, a finely focused flicker-free projected image is not required. Since the location of the bolt to be placed is limited by the threaded hole, precise positioning is less important, while repairing a defect in a painted finish requires only positioning sufficient to help identify the defect itself, and visibility is more desirable than clarity of the projected pattern.

[0015] The visibility of a projection is typically a function of the total amount of energy divided by the apparent perimeter of the feature or laser image. If more features, e.g., a complete set of bolt locations, are presented simultaneously without flicker, the apparent brightness may decrease linearly with the total projected length of the projected profile. Traditionally, the reduction in apparent intensity is offset by increasing the instantaneous laser power, thereby increasing overall energy and visibility. From a safety perspective, both practical and regulatory concerns exist regarding the fact that heat energy from laser exposure dissipates over time. If a given area of ​​the projection is exposed to the retina of the eye for half the time, regulation limits the power to a 20% increase, while equivalent energy and visibility would require doubling the power. This reduction substantially limits practical visibility if flicker-free projection is desired.

[0016] On the other hand, if flicker-free projection is not required as in the present invention, the optimal energy dose for the desired visibility can be evaluated. In order to provide safe operating conditions, it is generally necessary to use a Class 2 laser projection system. For Class 2 operation, the eye may be exposed to a 1mW laser source for 0.25 seconds. For a time period of approximately 50 milliseconds to 0.250 seconds, for a given instantaneous laser power level, the increased exposure duration only appears to be longer in perception, without a significant increase in visibility. According to the Class 2 specification, the acceptable emission level (AEL) = 7x10-4 t0.75 = 75μJ, equivalent to a laser power of approximately 1.5mW. This energy distributed over the minimum circumference provides the best visibility, but is subject to the limitation that smaller patterns beyond a certain distance will become more difficult to distinguish based on vision limitations. Providing an indicator with optimal visibility to identify the operating position becomes a balance between the minimum size or energy concentration and the typical expected distance from the operator.

[0017] The downside of this approach can be that flicker can be quite severe. A 50 millisecond period is equivalent to a 20Hz refresh rate. Therefore, if 20 defects are displayed simultaneously, each defect will flicker once per second. While not the best result in terms of clarity, and can be very frustrating if used as an alignment reference standard, it has proven to be ideal when used as a position indicator. In practice, the flickering of the indicator also triggers the motion receptors in the visual periphery of the eye, which significantly improves the operator's ability to locate the displayed indicator. This effect is so beneficial that if too few indicators are displayed, especially a single indicator, or the exposure time of each indicator is too short, an additional blanked delay is introduced to create the optimal flashing effect, thereby maximizing operator visibility.

[0018] Thus, the systems and methods of the present invention provide the operator with maximized visible icons that indicate the task to be performed and the completion of the task. While providing only clearly visible markings at the location of the desired operation can guide the operator to the desired location, the entire process is uncontrolled unless feedback is also provided to the operator regarding the results of the task. In order to guide the current status of the processing task, feedback can be provided based on the shape and visibility of the projected icon. The shapes of different laser projected icons can be used to indicate the type of bolt to be tightened or the status of the repair. For example, an icon associated with a defect can cycle between grinding, polishing, and completion states to indicate the operator's progress in repairing a single defect. In particular, once the repair of the defect is completed, the shape of the icon can be changed to indicate completion, and the flashing can be eliminated to not draw attention to the icon. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Other advantages of the present invention will be more easily understood and the present invention will also be better understood through the following detailed description with reference to the accompanying drawings, in which:

[0020] Figure 1 A perspective view of a workshop in which repairs to vehicle bodies are performed is shown;

[0021] Figure 2 showing a side view of a workshop in which the car bodies are inspected and repaired;

[0022] Figure 3 A fixture for calibrating a three-dimensional coordinate system within a work cell is shown;

[0023] Figure 4 A handheld probe for identifying the location of a feature on a workpiece is shown;

[0024] Figure 5 shows a vehicle body located in a three-dimensional coordinate system;

[0025] Figure 6 showing a controller screen with the identified transformations;

[0026] Figure 7 showing an identification icon projected onto the vehicle body surface; and

[0027] Figure 8 A flow chart of a typical repair procedure is shown, identified by a projection icon. DETAILED DESCRIPTION

[0028] refer to Figure 1 , the system of the present invention is generally indicated by the reference numeral 10. The system 10 is integrated with a work cell, also referred to as a booth 12. In this embodiment, the booth receives a vehicle body 14 after a paint coating is applied, where the paint coating is inspected. However, it should be understood that the booth 12 can be used for any operational purpose, including body-in-white inspection, component inspection, component application, and any other purpose where inspection and identification can occur (as will be further understood below).

[0029] The system includes an optical device 16. In one embodiment, the optical device 16 is mounted at an entrance 18 of the work cell 12. However, it should be understood that multiple optical devices 16 may be included, for example, if multiple car bodies 14 are positioned within the work cell 12, or if multiple car bodies are positioned on a moving conveyor 20. The optical device 16 includes high precision imaging optics, which may be provided by Micro-Epsilon's ATENSOR-LS1 technology. The optical device 16 identifies an object (item) 22 located on a surface 24 of the car body 14. As described above, the optical device 16 detects flaws or defects (items 22, see FIG. 2 ) in the paint coating of the car body 14. Figure 1 ), such as, but not limited to, dirt, scratches, pits, sunken portions, etc., thereby eliminating the need for an operator to perform a visual or manual inspection. In one embodiment, the object 22 is identified by the optical device 16 at another location before the vehicle body 14 enters the inspection room or work cell 12. Once the object is identified, the vehicle body is placed on a transfer conveyor to move it to the work cell 12.

[0030] The optical detector 16 is electronically connected to the controller 28 for sending a signal indicating a defect (object) 22 in the paint coating to the controller 28. In addition, the optical detector 16 also sends a signal indicating the vehicle model and chassis number to the controller 22. "Electronic connection" herein refers to physical wired or wireless transmission via WIFI, Bluetooth or equivalent technology. For simplicity, physical wiring is not shown in the figure. However, the correct identification of the object 22 on the surface of the vehicle body 14 requires recalibrating the vehicle body 14 in a three-dimensional coordinate system relative to the workshop 12. The controller is electronically connected to an object recognition system 30, which includes a laser projector 32 and a photogrammetry device 34. It should be understood that any light projector or projection system that projects a beam of light can be used. However, for simplicity, "laser projector 32" will be used throughout this specification. In one embodiment, the system 10 includes a plurality of laser projectors 32 spaced around the compartment 12. Four laser projectors 32 are located above the vehicle body 14 so that the laser can be projected onto each outer surface of the vehicle body 14. Alternatively, a fifth laser projector 32 is located directly above the vehicle body 14 for providing laser projections onto horizontal surfaces of the vehicle body 14. It should be understood that any number of projectors 32 and photogrammetric devices 34 may be included to provide projections onto each surface of any target object placed within the bay 12 for inspection. Furthermore, the laser projectors 32 and photogrammetric devices 34 may be integrated into a single unit or located separately.

[0031] The system 10 of the present invention provides the ability to indicate a location or task to be performed on a workpiece (e.g., a vehicle body 14) even as the vehicle body 14 is passing through a work cell or bay 12. These include the presence of a rivet, repairing a defect that has been identified in a paint coating or substrate, placing a fastener for routing, or installing a bolt or screw in a desired location. Thus, the object recognition system 30 is also adapted to generate a three-dimensional coordinate system within the bay 12 and locate the vehicle body 14 within that coordinate system.

[0032] Traditional optical template projection aligns the projected pattern with the tool by scanning targets mounted on the tool with known positions specified in the coordinate system of the CAD model of the part or layup tool. Using a method called "resection", the scan angles for each target and the corresponding coordinates of those targets can be used to calculate the 3D position and orientation of the projector relative to the tool, which in turn is used to calculate the scan angles to project through the required 3D coordinates to track the template pattern. Each projector must have enough of these targets in the projector's field of view to be positioned relative to the tool.

[0033] For applications suitable for the method of the present invention, when inspecting the painted finish of a part, it may be impractical to apply retro-reflective targets to the vehicle body 14, let alone accurately measure the location of these targets. In addition, it is usually necessary to project the projection field onto each surface of the part from all directions, thus requiring multiple object recognition systems 30 spaced around the compartment 12.

[0034] To support this desired feature, the laser projector 32 is configured to surround the vehicle body 14 from all sides within the work booth or booth 12. Figure 1 and Figure 2 As shown, a plurality (five) of projectors 32 surround all sides of the vehicle body 14 within the work cell 12 .

[0035] In order to establish an accurate projection of the laser projector 32, the position of the laser projector 32 in the three-dimensional coordinate system is established by placing a reflective target 36 at an arbitrary position in the cubicle 12. Alternatively, the reflective target 36 can be placed at a known position in the workroom 12. In one embodiment, a plurality of photogrammetric devices 34 first locate the reflective target 36 in the workroom, and then a more accurate position is identified by scanning the reflective target 36 with a laser projector and detecting the reflected laser light using an integrated laser sensor 43. In this way, a common reference system is established in the workroom 12, and the relative positions of all laser projectors in the workroom are established in the three-dimensional coordinate system.

[0036] To allow standardization between different work cells 12 in a factory, a standardized reference system is established by placing known reference fixtures 38 at defined locations within the work cell 12 . Figure 3 One configuration of the reference element 38 is shown in FIG. In this embodiment, the reference fixture 38 takes the form of a triangular frame. However, other shapes may also suffice and are included within the scope of the present invention. A plurality of reflective reference targets 40 are spaced apart along the reference fixture 38. In this embodiment, the reflective reference targets 40 are spherical retro-reflectors that allow the laser beam to be reflected back toward the laser projector 32 from any direction toward the laser sensor 43 in a known manner.

[0037] To create an integrated 3D reference coordinate system within the work cell 12, the laser projector 32 performs laser scanning, and the 3D position of each target is measured from each of the multiple spaced-apart laser projector 32 positions, and then merged into a common reference coordinate system by the controller 28 by identifying similar subsets of the positions of the targets 36. If specified, the reference fixture 38 will act as the master device, and the positions of all targets 36 will be converted to the coordinate system of the fixture 38.

[0038] After the laser projectors 32 have measured the targets and the controller 28 has positioned them in a common three-dimensional coordinate reference system, each projector 32 can use the reference targets 36 to establish and update the position of the projector 32 within the workroom 12, and more specifically within the common three-dimensional coordinate system. If the projectors 32 have an integrated photogrammetry system 34, such as the VPS1 projector from VirtekVision International, any projector 32 can be repositioned within the workroom 12 and automatically identify a subset of visible targets 36 and recalibrate its position within the workroom 12 by imaging the targets 36.

[0039] Once the three-dimensional configuration of the work cell 12 is established, the projectors 32 can be configured to cover all surfaces of the vehicle body 14. Thus, even if a projector 32 is moved, the projector 32 is automatically recalibrated within the work cell 12 using at least one of the laser projector 32 or the photogrammetry device 34. However, in order to accurately project, the position of the vehicle body 12 or other workpiece (as appropriate) still needs to be determined. In one embodiment, this is achieved using the conventional method of placing reference targets 36 at predetermined locations on the vehicle body. In this way, the system 10 can identify the configuration of targets associated with the vehicle body 12 to provide accurate laser projection. When the configuration of the work cell 12 has been established in a three-dimensional coordinate system and each projector 32 is positioned in the coordinate system, each projector 36 does not need to have an image of all targets 36 to accurately project a laser image onto the surface of the vehicle body 14. In this case, the controller 28 has calibrated the position of each projector 36 relative to the other projectors 36, and the disposition of the vehicle body 12 can still be identified as long as a sufficient number of targets are visible to the group of projectors 32.

[0040] In some cases, it is impractical to provide a precise measurement of the location of the target 36 on the vehicle body 14. In such cases, alternative embodiments are created in which the location of the vehicle body 14 can be determined by measuring features of the vehicle body 14 itself, such as the hood edge, door frame, identified holes, etc. For systems 10 with integrated metrology determination capability, such as a VPS1 projector, an optically tracked mechanical probe 42 (such as Figure 4 4 (shown). The probe 42 includes a contact ruby ​​ball tip 44, which in one embodiment has a diameter of 6 mm. The tip 44 is connected to the probe face 46 by a shaft or grip 48. The system 10 recognizes various probe face 46 configurations by a pattern of retro-reflective dots 50 pre-arranged on the probe face 46. In this way, a given probe 42 can be used to identify the location of a given feature on the vehicle body 14. The probe 42 projector can also be used to take measurements on different surfaces of the vehicle body 14 to establish the location of the surface of the vehicle body 14. To position the probe 42, the laser projector 32 scans the face 46 of the probe 42 so that the retro-reflective dots 50 reflect the laser beam to the light sensor 43 in a known manner. In this embodiment, the operator can point the measurement cursor on the controller screen 29 to the desired location to measure the specified feature of the vehicle body 14.

[0041] However, in some cases, it is not efficient for an operator to manually identify features using the probe 42. As a further alternative embodiment, a method for automatically measuring the location of the vehicle 14 position in a three-dimensional coordinate system has been developed. In this embodiment, a series of alignment steps are performed that sequentially determine the position of the vehicle body 14. These alignment steps consist of two basic components: transformation and measurement. It is believed that the use of transformations can provide sufficient accuracy for the purpose of identifying the arranged objects 22 on the surface of the vehicle body 14, as opposed to using only fiducials. However, the use of fiducials can help with alignment accuracy (as further described below).

[0042] A "transformation" herein refers to the relationship between an initial position of the vehicle body 14 and subsequent or interval positions of the vehicle body 14. For example, a rough alignment position or first transform may establish the intended approximate position of the component, in one embodiment using the photogrammetry device 34. The derived initial position may be based on measurements taken to refine the first transform 58 based on a laser scan of the vehicle body 14 guided by the initial rough alignment position to establish a second transform 60 that is more accurate than the first transform 58. Other defined transforms may define constrained relationships, such as moving from an initial position on a lift, translating on a sliding mechanism or moving conveyor, or even pivoting on an axis, which may provide accurate measurements of the door surface when opened or closed on hinges.

[0043] The measurements are associated with a specific or first transform 58 to establish or constrain the position of the derived or second transform 60. The first transform 58 measurements are nominally specified three-dimensional coordinates based on known cell constraints, obtained by scanning with an optical tracking probe 42 by triangulating laser spot positions within the cell 12 or by triangulating visual image features such as holes using photogrammetry equipment 34. Alternatively, the measurements are defined rays that "pin" the position of the body 14, established by visual image features or obtained using a laser by detecting the silhouette of a projected light stripe relative to the boundaries of the part (e.g., Figure 2 ). In this embodiment, the projected light pattern 52 is scanned on reflective patches 54 strategically placed within the work cell to enhance contrast. In one embodiment, the reflective patches 54 are placed on the floor or on a moving conveyor and are scanned by a laser projector that first projects a light stripe or other light pattern onto an edge 56 of the vehicle body (e.g., a hood or trunk lid) and then onto the reflective patches 54. In this manner, the laser projector 32 detects the edge 56 of the vehicle body 14 for the controller 28 to calculate the first transformation 58.

[0044] Based on the defined transformations, the transformations and their associated measurements are combined in a hierarchy. If the second transformation 60 is a "child" of another transformation, it is derived from the parent or first transformation 58, subject to possible positional changes, such as translation provided by a conveyor, a lift, or rotation around an axis caused by movement around a hinge or spindle. This hierarchy provides a higher level of positioning accuracy of the vehicle body 14, even in the event of drift or other movement.

[0045] refer to Figure 6 , a rough tolerance may be applied to any measurement that explicitly allows the vehicle to move more or less unhindered within the tolerance without affecting alignment. For example, a first transformation 58 may be established that indicates the expected approximate position of the vehicle body 14 within the work cell 12 with a tolerance of a few inches. The measurements that define this position (perhaps initially measured with a probe, or based on general knowledge of how the workpieces are transported) may guide the location of a more precise second transformation 60 for actual workpiece placement. If the rough tolerance of the first transformation 58 is not exceeded, the measurements of the precise transformations are applied sequentially guided by the initial transformation with constraints on the application of each additional measurement. The rough alignment value representing the first transformation is in Figure 6 The point is represented by an outer sphere surrounding the point.

[0046] In addition, each measurement has a tolerance limit, which will limit the impact of a transformed measurement that is too far off position. For example, the controller will recognize if the scan hits the wrong surface entirely. The alignment limits are defined by the sphere representing the first transform 58. The more accurate second transform 60 is represented by a smaller sphere shown inside the larger sphere defining the first transform. If the measurement position of the second transform 60 is outside the sphere defined by this value, any additional impact on the alignment is minimized.

[0047] To illustrate the method of the present invention, a non-limiting example is now provided. Example vehicle body 14 is generally located in the workshop 12, and then based on the floor (such as Figure 5 The measurement of the vehicle body 14 is limited to the initial position (as shown by the grid shown in FIG. 1 ). The sequential scans are used to establish the location of the vehicle body 14 in the work cell 12 and, therefore, in the established three-dimensional coordinate system. The translation of the vehicle body 14 in a given direction (e.g., along the vehicle axis or sliding direction 65) is updated using a single scan at one end of the vehicle body 14, as shown in FIG. Figure 6 As shown, a computer generated scan image is displayed on the controller screen 29.

[0048] The initial position for the first transformation process is established by three probe measurements Rough PT 1, Rough PT 2, and Rough PT3, using a rough tolerance of 8 inches to allow for variations in the placement of the vehicle body 14. The derived position transformation "Initial Position" is established by measuring the floor reference plane by initially measuring the floor surface by scanning the placed target 36 by the laser projector 32. This step does not need to be repeated as each additional vehicle is introduced into the work cell 12. A series of scanning measurements are then taken to determine the orientation of the vehicle 14 on the surface. Each additional measurement further geometrically constrains the position of the vehicle body 14 within the work cell 12, and therefore within the three-dimensional coordinate system.

[0049] Once the vehicle body 14 is established in the workshop 12, a scanning measurement of the vehicle body 14 is automatically performed by the system of the present invention. Similar to nominal measurement and probe measurement, scanning measurement also uses coarse and extreme alignment values, that is, a first transformation 58 and a second transformation 60 are established. For all scanning measurement types, a range (Range) and a zone (Zone) can be specified. The range (Range) specifies where the system can search for the measurement target. The controller 29 guides the search to a specified distance in two directions from the nominal value (as can be understood from the CAD data). The zone (Zone) gives the radius around the nominal position where the measurement is acceptable. The final scan in the sequence "Scan Side 2" is shown as follows. Figure 6. In this example, the location of a specified surface point 62 is measured. A vector 64 extending outward from the surface point 62 indicates the range specified for the measurement. The region defines the acceptable measurement locations and is represented by two disks 66 located at either end of the range line. Setting tight tolerances for the regions allows very specific areas on the part to be measured.

[0050] The "slide position" transform 65 is used to determine the final translation in a single direction along the length of the body 14. All other constraints on the body 14 are specified by the "initial position", so a single scan measurement at the sliding end 68 of the body 14 can update any position in the sliding direction. Figure 5 In the non-limiting example shown, the slide can be adjusted + / - 10" from the middle position and is currently at -3.2518" as a non-limiting example.

[0051] The slide end measurement to determine the current position of the slide is defined with a drift check interval, which periodically measures the end position and adjusts the slide position as needed. The method defined herein can be used to locate any object within the work cell 12 with the necessary accuracy for a particular job purpose (e.g., repairing a paint defect). For example, Figure 5 As shown, the position of the vehicle shown in the positioning workshop 12 is based on the definition of the floor and the scanning of the surface of the vehicle; for example, highlighting the scanning measurement of the side of the vehicle.

[0052] This embodiment provides an alternative to the reference alignment currently implemented in motion compensation disclosed in US Patent Application No. 9,881,383, the contents of which are incorporated herein by reference. However, the transform and data alignment are optionally done in parallel so that both alignment methods are available.

[0053] In existing implementations, a number of digital alignment polygons are defined. The operator can start the measurement by starting a cycle through each reference polygon via a key on the computer keyboard and starting the measurement with the probe via a separate key. Optimization of the measurement process is achieved by determining the position of the body 14 in the work cell 12 to minimize the overall error between the measurement and the associated digital alignment polygon. For example, the operator can also select an arbitrary polygon instead of a single-sided polygon. Optimization is performed every time a new measurement is taken. Improvements can be made for automotive applications, where we have an initial position of the car and then it moves vertically on a lift or in a sliding direction. Recalibration is achieved by additional measurements for drift checking or motion compensation.

[0054] In validating the system 10 and method of the present invention, an initial position of the vehicle body 14 is established by specifying "coarse" datum features on the vehicle body 10, which are measured by the probe 42 as described above. The measurements are taken while the vehicle lift is in the lowered position. This measurement data is essentially "scrambled" by first optimizing only the coarse datums on the vehicle 14 (typically three datum points). The three datums establish an approximate position of the vehicle 42 in a three-dimensional coordinate system, providing enough stability for the controller 28 to calculate the geometric position. Once calculated, the system is able to infer the position of any additional measurements. Mathematical alignment is performed by assigning lower weights to measurement errors, and then more accurate measurements are added. However, if the additional measurements are significantly off, they may pull the position completely out of the initial alignment, rendering the system accuracy unusable and requiring recalibration.

[0055] In one embodiment, a more flexible fiducial identification method is provided. Two components of the proposed method include the use of transformations and the performance of fiducial measurements. A "transformation" represents the position of the vehicle body 14 in a three-dimensional coordinate system and the relative position between transformations. Thus, a "rough" alignment of the vehicle 14 in a three-dimensional coordinate system can be optimized using a first transformation 58 and represented separately from one or more second transformations 60 derived from the first transformation 58, thereby enabling relative relationships to be defined between multiple transformations.

[0056] As used herein, a fiducial measurement is a measurement of a specific fiducial feature on the vehicle body 14 and is explicitly referenced relative to an associated transformation. For example, three fiducial measurement points would be associated with a calculated initial rough alignment or first transformation. Rather than having an explicit "rough" fiducial type, tolerances (tolerances) can be applied to any fiducial that explicitly provides more or less unhindered movement within the tolerance range (with a small deviation equivalent to a small threshold, which may be optionally set to 0.010", which will center the fiducial if not refuted by other measurements).

[0057] Transformations and reference measurements are combined in a hierarchy based on previously defined transformations. If a second transformation is a "child" transformation of another or a first transformation, the second transformation is derived from the parent transformation, affected by possible transformations (e.g. conveyors or lifts), and all reference measurements and tolerances of the parent transformation must also be maintained relative to the vehicle configuration in the coordinate system.

[0058] Specific alignment process steps are mixed with the projection of icons to be displayed on the surface of the vehicle body 14. Editing of the hierarchy is achieved using the process editing function, where each step can be defined as a projection step, a transformation, or a reference measurement. Additional information required to indicate tolerances, reference type, measurement method (target or probe), measurement list, current position of the lift, etc. is present in the interface as required.

[0059] The root process of the process tree defaults to being the root transform in the hierarchy; however, any sibling transforms will have independent positions. Any base measure siblings of the transform or descendants of the transform, until another transform is reached, will be applied to constrain the transform. Any descendant transforms of the transform or its base measure will also be derived from the parent transform, and must also satisfy the requirements of any dependent base measures. Siblings of a transform are independent of each other, but all depend on a common parent transform. In a similar manner, the position of any sibling or descendant projection processing steps of a given transform will be determined by that transform.

[0060] The implementation of the invention of the present application provides for complete definition of the transformation / reference hierarchy, but is particularly convenient for measurement by using a probe. Alternatively, the method of the present invention provides for "nominal" direct input of measurements, manual measurements by selection in the image, selection with a laser, intersection with a laser ray, etc.

[0061] In one embodiment, a root transformation on the workpiece or vehicle body 14 is specified by three datum points measured with tolerances ranging from two inches to four inches. The next level of measurement includes an initial position datum that tightly constrains the position using datums established by features on the surface, and a final transformation level provides a transformation determined by sliding the workpiece horizontally, with the position determined by the endpoint datum measurement.

[0062] In the interface, identification of three coarse reference points establishes a rough position of the surface of the body 14. A derived "initial position" is established by measuring a point and two edges of the body 14. Subsequently, a slide transform is defined based on the normal of the specified plane, and the position of the body 14 is defined by the end point reference measurements. When the slide transform is actively measured, the optimized current slide position is indicated. Clearing the end probe measurements on the controller moves the slide position back to the initial position.

[0063] Two types of position processing steps are envisioned, one for feature type "None" and one for feature type "parallel plane". For feature type "None", the sibling transformation steps that follow this step define the position. For example, the default step is to define a position (Position) for feature type "None". This is typically followed by three measurement points, each with a large rough tolerance, which combine to define the approximate position of the part. For feature type "parallel plane", the position is specified to be movable in a single direction, perpendicular or orthogonal to the defined plane.

[0064] The controller screen 29 is interactive for the operator. Therefore, a location step can be added as part of the location recognition scheme. A sequence of procedures for adding a location step is listed below:

[0065] 1. Click the Create New Step button to enable the step editing controls.

[0066] 2. Type a unique, descriptive name in the Step Name box to identify the measurement point.

[0067] 3. Select the location for the type of process step. Controls become visible based on the selected feature type.

[0068] 4. Select the feature type (None or Parallel plane).

[0069] a. If the operator selects “None”, proceed to step 5.

[0070] b. If the operator selects "Parallel Planes", the following steps are performed:

[0071] c. Define the parallel planes that will be used for this process step.

[0072] d. Specify the minimum and maximum range of motion for the plane. The motion is in the direction perpendicular to the plane.

[0073] 5. Select the step in the Process list after which the new step should be added, either as a sibling or a child.

[0074] 6. Add the new step as a sibling or substep by clicking the appropriate button.

[0075] The measurement process steps are used to measure a physical portion of the vehicle body 14 in order to align the portion with the mathematical model. These steps enhance the ability of the laser projection to be accurate. The method of the present invention includes four types of measurement features: nominal point, nominal line, surface point and plane. Nominal features use measurement points to align with specific locations. For surface points, only the proximity of the measurement point to the surface defining the point is used to establish alignment. For planes, alignment is achieved by measuring three points on the plane.

[0076] Three types of measurement methods are included: Nominal, Probe, and Scan. Nominal and Probe are set up to identify all feature types on a workpiece, but Scan can be used to identify surface points only.

[0077] Nominal and probe measurements are used to measure specific locations on a portion of the vehicle body 14. Nominal measurements require the user to manually specify the location of a point or plane. Nominal measurements are best used when the location required for the measurement is known relative to the three-dimensional coordinate system of the cell 12. For example, a location on a fixture that is part of a mathematical model or a reference plane that is a floor. For a nominal point, nominal line, or surface point, a measurement is added by manually entering the x, y, z coordinates of the point. For a plane, the values ​​for three points on the plane must be entered.

[0078] Probe measurements use a probe to measure one or more points, as described above. With nominal or probe measurements, you can specify coarse alignment tolerances and alignment limits. For coarse alignment values, the measurement position can be anywhere within the sphere defined by the value without any effect on the alignment. As described above, coarse alignment values ​​are Figure 6 is shown by a sphere surrounding the point.

[0079] For the alignment limit, if the measurement position is outside the sphere defined by this value, any additional effect on the alignment is minimal. The alignment limit is also represented as a sphere. It should also be understood that two alignment values ​​can be used simultaneously.

[0080] Scan measurements are performed automatically by the system. However, only surface points can use the Scan measurement type. Like Nominal and Probe measurements, Scan measurements also use Coarse and Extreme alignment values. For Scan measurements, the operator can select the Scan measurement type: Standard, Drift check, or Nominal.

[0081] For all scanning measurement types, Range and Zone options are available. Range specifies where the system can search for measurement points. The search or scan is performed at a specified distance in both directions from the nominal value. Figure 6 The range is shown by the long line 64 through the centre of the surface point location. The area gives the radius around the nominal location that will receive the measurement. It is shown by the two disks at either end of the range line.

[0082] The size of the Zone measurement type is based on the surface being measured. If a narrow surface is being measured, the value must be small enough to accept measurements only on the surface at that point. If the surface is large and flat, a larger Zone value may be acceptable. For the Standard Scan measurement type, measurements are taken when a step is selected. For the Drift Check Scan measurement type, measurements are taken at regular intervals. The time (in seconds) between measurements is specified by the Interval. Drift checks can be optionally performed at regular intervals without additional operator input. This is particularly useful when the vehicle body is movable while located in the inspection chamber 14.

[0083] When the placement of the vehicle body 14 is determined within the three-dimensional coordinate system, objects 22 placed on the vehicle body surface can now be identified to the operator through laser scanning, eliminating the need for the operator to visually search for the object 22. While any light source is capable of projecting the icon 70 onto a surface with a certain geometric accuracy, the laser projector 32 is particularly suitable when it comes to providing the dual function of providing accurate measurement of objects, surfaces and edges. Projecting the icon 70 onto the vehicle surface is best illustrated in FIG. Figure 7 As mentioned above, once the object is located in the three-dimensional coordinate system, the object recognition system provided by Micro-Epsilon works in conjunction with the laser projector.

[0084] refer to Figure 8 , a step-by-step flow chart is shown, wherein, first, the laser projector 32 projects an indicator icon 72 in the form of an arrow or chevron pattern pointing to the object 22 (in this case, a paint defect). Due to the rapid scanning nature, multiple defects can be identified simultaneously. Once the operator indicates that the object 22 has been identified (in one embodiment, by placing a reflector in the laser beam to return the laser beam to the laser sensor 43), the projector 32 indicates the process to be performed, such as grinding with the start repair icon 72. Upon completion, the projector 32 detects the reflection again and scans the grinding complete icon 76 onto the surface at the location of the identified object. The projector then scans the start polishing icon 78 (or any other necessary task). When the projector detects that polishing is complete, the repair complete icon 80 is scanned onto the surface. When multiple defects are identified that require multiple operators to perform repairs, the completion of the vehicle repair process is signaled only when all icons are visible. Of course, the operator can indicate to the projector by movement or other reflective elements that additional repairs are required, which is indicated by scanning the additional repair icon 84 at the defect location.

[0085] This process can still be confusing if there is no easy means for the operator to specify the change of state of the display position. As mentioned above, defects can be arbitrarily distributed on the paint surface, requiring one or more operators to determine that, for example, the defect in the upper right corner of the rear passenger door has been repaired. To achieve this goal, the internal sensor 43 used in the laser projector to locate the retro-reflective mark 36 to establish the projector position is used as a feedback mechanism. When the laser beam passes through a piece of retro-material, a signal is recorded that is associated with the location icon through which the laser passed.

[0086] For example, by placing a piece of reflective material on the back of a glove that is moved into the path of the laser of the tracking indicator, the system can signal the controller 28 that the icon has transitioned to the next process state. Note that this operation is in the context of a specific location icon 70, so the relevant location does not need to be identified from the outside, and any number of operators can interact with the system simultaneously in this manner. The process status can be highlighted or summarized on a remote or local interface or controller screen 29 to dynamically display the progress of the task. Statistics can be tabulated to determine the cause of the defect. Using feedback of this nature can provide a foolproof method of ensuring that the entire process is completed, for example, by changing the background color of the interface to clearly indicate the completion status. In addition, for process control purposes, it is possible to track the time taken for various steps in the process, associate the difficulty of performing specific types of tasks or repairs, and evaluate the efficiency of operators or the benefits of using different numbers of operators in a work team.

[0087] It will be appreciated by those skilled in the art that the invention of the present application can be used in a variety of manufacturing environments. For example, the invention can be used in fixed work sales. In addition, the invention can be used in a fixed work cell where the vehicle or other object being processed can be moved in a vertical direction to assist the operator in performing the process. Further, the invention can be used in a cell with a moving assembly line where the vehicle, and sometimes the operator, moves continuously.

Claims

1. A method for identifying the geometric position of an object on a workpiece surface, wherein an optical device identifies the object on the workpiece surface, the method comprising the following steps: Provide a controller; providing an object recognition system including a light projector and a photogrammetry system; generating a three-dimensional coordinate system within a work cell using at least one of the light projector and the photogrammetry system; identifying a position of the workpiece surface in the three-dimensional coordinate system using at least one of the light projector and the photogrammetry system; the controller calibrating the position of the workpiece surface in the three-dimensional coordinate system with the identified object by identifying a first transformation indicative of an expected approximate position of the workpiece and a second transformation established by refinement of the first transformation guided by the expected approximate position of the workpiece, wherein the second transformation is more accurate than the first transformation; The controller calculates the geometric position of the object on the surface of the workpiece identified by the optical device in the three-dimensional coordinate system; and The controller sends a signal to the light projector to project a light beam onto the workpiece surface to thereby identify the arrangement of the object located on the workpiece surface.

2. The method according to claim 1, wherein: The step of identifying the arrangement of the object located on the workpiece surface is further defined by identifying the position of the object on the workpiece surface using the light beam projected by the light projector.

3. The method according to claim 1, wherein: The step of identifying the arrangement of the objects located on the workpiece surface is further defined by identifying a work function that has been performed on the objects identified on the workpiece surface.

4. The method according to claim 1, wherein: The second transform is a "child" transform of the first transform and is derived from the first transform.

5. The method according to claim 1, wherein: The step of calibrating the position of the workpiece surface in the three-dimensional coordinate system is further defined by at least one of the light projector and the photogrammetry system identifying features of the workpiece surface and calibrating the features in the three-dimensional coordinate system.

6. The method according to claim 5, wherein: The step of identifying a feature on the workpiece surface is further defined by identifying the feature with a handheld probe and reflecting a light beam from a light source from the probe to the photogrammetry system or a light sensor located at the light source.

7. The method according to claim 1, wherein: The step of providing a light projector and a photogrammetry system is further defined by providing a plurality of light projectors and a plurality of photogrammetry systems.

8. The method according to claim 1, wherein: The step of generating a three-dimensional coordinate system within the cell is further defined by placing a reference object at a known location within the cell and positioning the reference object using at least one of the light projector and the photogrammetry system.

9. The method according to claim 1, wherein: The step of identifying the arrangement of the objects on the surface of the workpiece is further defined by identifying paint defects on a painted surface.

10. The method according to claim 9, wherein: The step of identifying the arrangement of the objects located on the surface of the workpiece is further defined by identifying whether a repair of the paint defect has been performed.

Images