Classification of Laser Speckle Curves for Verifying Part Placement in Assembly Tasks

Abstract

The present disclosure relates to the classification of laser speckle curves for verifying part placement in an assembly task. A method for verifying the placement of a workpiece in an assembly task by scanning a coherent light source, such as a laser, over a surface and characterizing the detected interference speckle pattern to discern the position of the placed workpiece from the surface on which it is placed. Such discernment is possible even if the characteristic features of the workpiece and the background are less than the resolution of the scanning system. Additionally, by classifying the associated speckle response, characteristics of the workpiece, such as the orientation of fibers in the material, can be detected.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Patent Application No. 63 / 047,394, filed Jul. 2, 2020, the content of which is incorporated herein in its entirety. Technical Field

[0003] The present invention generally relates to a method for inspecting the surface of an assembly to verify the accuracy of a manufacturing process. More specifically, the present invention relates to an improved method for identifying the correct placement and orientation of components or workpieces on a working surface. Background Art

[0004] For certain types of assembly, it may be difficult to verify the placement of components using conventional imaging systems. For example, when manufacturing composite parts, many layers are sequentially placed on a tool surface. A typical workpiece, ply, component, or fiber - impregnated tape may be composed of carbon fiber material impregnated in a carrier resin, which is placed on a more or less identical layer before it until a composite material of a predetermined thickness is achieved. Each workpiece is typically placed by a robotic fiber - tape laying machine or manually positioned guided by a projected laser pattern that serves as an optical template to direct the assembly. However, for either type of placement, the position of each workpiece and the orientation of the fibers within the material must be confirmed to ensure the strength and structural integrity of the produced component. Currently, this is achieved by manual inspection, comparing the placement of each workpiece to the projected laser template. Unfortunately, this inspection method is rather slow and prone to human error as it relies on human interpretation.

[0005] The need to inspect the assembly process is not limited to laminated composites but is required for many assemblies. Typically, during the assembly process, two components that are difficult or impossible to visually distinguish are mated, which presents a difficult or impossible challenge to verifying the correct placement and orientation during mating. In such cases, manual inspection is not feasible when consistent results are required.

[0006] To reduce the need for manual inspection, it would be desirable to automate the inspection process, for example, by using an inspection camera to measure the workpiece position and fiber orientation. However, the tools used to manufacture components can be very large, and the features of the workpieces can be very small (e.g., a 20 - foot tool while the material is composed of carbon fibers with a diameter of only one - thousandth of an inch). As a result, any ordinary inspection camera or device will generally not have the ability to directly resolve the features of individual workpieces to verify their placement.

[0007] In addition, carbon fiber materials respond in a very directional manner, either reflecting any ambient light in a mirror-like fashion (direct reflection) or absorbing any remaining light in the dark fibers. Similar difficulties exist with other materials that have similar surface characteristics when assembled. Each of these material characteristics greatly increases the difficulty of successfully automating the inspection process. Known camera-based inspection systems have proven ineffective in obtaining accurate and consistent inspection results in these situations. Therefore, it would be desirable to automate the inspection process in a manner that provides consistent and accurate inspection results of the work surface after the workpiece or component has been mated to the work surface to verify correct placement and orientation. Summary of the Invention

[0008] To overcome problems associated with inspecting two elements, such as a component or workpiece and a work surface, the invention of the present application presents a method for evaluating interference speckles generated by a coherent light source. In one embodiment, a method for verifying the placement of a workpiece within an assembly task includes placing an impregnated laminate workpiece onto a work surface. In another embodiment, the method is used to verify the placement of a component on a work surface when the component is difficult to distinguish from the work surface. The coherent light source is scanned across the surface of the placed workpiece and along the work surface that is set as the background of the placed workpiece. The optical sensing system records the optical characteristics of the interference speckles generated by the scattered coherent light from the coherent light source, and in one non-limiting embodiment, the coherent light source is a laser. The speckle response for distinguishing between the surface of the workpiece and the background is classified to determine whether the workpiece or component has been correctly placed and oriented.

[0009] To overcome the difficulties of known camera-based inspection methods, the method of the present invention scans a coherent light source over a portion of the work surface and monitors the characteristics of the light reflected from that surface. The coherent light (such as, for example, a laser beam) provides very strong illumination of the workpiece or component and the work surface source, thereby overcoming any poor reflectivity of the surface. Therefore, the resulting interference characteristics of the returned light or speckles can distinguish the microstructure of the surface of the workpiece or component being inspected from the work surface. The method and system of the present invention can be used to identify the alignment of any material, whether or not visually distinguishable. This includes the placement of trusses and pegboards, fabrics, metal components, mating metal structures, and the alignment of any two components that can be assembled or mated in a manner that requires verification that the assembly has been correctly performed. In another complementary embodiment, a calibration laser that is currently used to project a directional template onto the work surface for accurate placement can also provide a reference for manual inspection to perform an inspection scan that provides the necessary accuracy to verify placement. Brief Description of the Drawings

[0010] Figure 1Shows the reconstruction of the expected image according to the irradiation characteristics of the laser source;

[0011] Figure 2 Shows the variable speckle sensor response in a narrow bandwidth;

[0012] Figure 3 Shows the variable speckle sensor response at a wide bandwidth;

[0013] Figure 4 Shows the high-contrast speckle pattern of detuned laser scanning on an aluminum substrate;

[0014] Figure 5 Shows the low-contrast speckle pattern of detuned laser scanning on standard paper;

[0015] Figure 6 Shows the Figure 4 Deconstructed composite image of an imperceptible scratch on an aluminum substrate;

[0016] Figure 7 Shows the scanning of a workpiece by a coherent light source of a laser projector with a coherent light source;

[0017] Figure 8 Shows the laser line projection aligned parallel to the expected lamina boundary; and

[0018] Figure 9 Shows an alternative embodiment of the scanning of a workpiece by a coherent light source of a laser projector with a coherent light source having a beam splitter and a sensor. Detailed Description

[0019] The invention of the present application uses coherent light to distinguish between two different components, such as a working surface and a lamina, a workpiece, or a component part. When coherent light is reflected from a surface, the roughness or texture of the surface produces a varying reflection, which produces an interference pattern. The interference pattern is produced due to the roughness and slope of the surface. For example, two different materials (such as plastic and fabric) produce different interference patterns when reflecting coherent light.

[0020] The interference pattern is produced by speckle, which is an optical characteristic that produces image variations. When a laser is used to produce coherent light, this phenomenon is sometimes referred to as laser speckle. When produced by a laser, the speckle pattern typically appears in the diffuse reflection of monochromatic light. The speckle pattern varies with the type of surface or substrate from which the coherent light is reflected, such as paper, white paint, or other rough surfaces. The reflectivity varies with the surface roughness.

[0021] Reference Figure 1, showing an exemplary reconstruction of the expected image relative to the surface onto which the coherent light is projected. The X-axis indicates millimeters, while the Y-axis indicates intensity. This response depends on the nature of the illuminated surface, the characteristics of the coherent light source or laser, and the response of the camera optics. Although the reflectivity, coherent light source, and sensor output are consistent, the response is inconsistent and is related to the physical characteristics of the illuminated surface.

[0022] Figure 3 Represents the effect of slightly moving the laser speckle pattern on a uniform surface, where the X-axis also indicates millimeters and the Y-axis indicates intensity. This is exemplary when the laser illumination is over an area larger than the area imaged by the camera. Thus, the camera images an area, for example, smaller than the laser beam diameter, resulting in an image presented as a fine speckle pattern. In this arrangement, the transition of the laser beam from one surface to a new surface including a texture transition causes a sudden change in the laser speckle pattern. When scanning moves from one surface to another, the variable speckle pattern response is shown to vary with the surface change.

[0023] A laser projection system for projecting a laser template onto a work surface, such as that disclosed in U.S. Patent No. 9,200,899, is particularly suitable for this process of the present application due to the focusing of a fine laser spot less than one millimeter, the content of which is incorporated herein by reference. Although the view of the overview camera is rather coarse, typical pixels corresponding to a 1 / 8 to 1 / 4 inch area also cause the optical characteristics of the camera lens to blur the image of the laser spot over multiple pixels. This situation is shown in Figure 2 , showing a variable sensor response to a narrow bandwidth, where the X-axis indicates millimeters and the Y-axis indicates intensity. Thus, the image now appears as a Gaussian or approximately Gaussian-shaped spot within the image. However, due to optical wavelength interference, the position of the image may appear to move irregularly. The system of the present invention utilizes this irregular movement and variation that occurs at very slight offsets as small as 0.3 mm, thereby revealing variations on a scale as small as the orientation of individual fibers, even when the camera and laser projector are set at large offset distances.

[0024] Considering these technical elements, if the laser beam scans crosswise to the elongated direction fibers disposed in a tape or laminate, the interference changes rapidly, while scanning generally parallel to the orientation of the fibers maintains a much greater similarity and less interference to the response speckle pattern. Then, scanning the laser in a changed direction establishes and identifies the orientation of the individual fibers disposed in the laminate, even though neither the laser nor the camera has a resolution to directly image the features of the material. It should be understood that this process is not limited to identifying the orientation of fibers, but any distinguishable surface feature can be used to determine the orientation of components placed on the work surface.

[0025] In this example, the interaction of the coherent light generated by the laser with the material surface can be examined through various modes. In one embodiment, the mode includes causing the laser to scan through the identification area at high speed to generate field illumination. Alternatively, a slower scan speed can be used to detune the laser focus, resulting in a large laser spot on the surface. In either case, a tan image of the laser speckle generated by the laser beam in the local area can be generated. As Figure 4 shown, the laser speckle generated on the 6061 aluminum plate produces a high-contrast speckle pattern, which is mainly the result of fine, almost imperceptible scratches or micro-scratches on the aluminum surface. In addition, any material can be distinguished from another material based on surface characteristics, anomalies, and specific features. Even when a workpiece or component mates with another workpiece, component, or working surface that is of the same material stock, the system 10 of the present application is capable of distinguishing them separately by relying on the laser speckle pattern, as further explained below.

[0026] As a comparison, Figure 5 shows the laser speckle as a result of scanning a standard white paper with the coherent light generated by the laser beam. The fine structure of the fabric of the fibers that make up the paper exhibits a much lower speckle contrast. However, even when a low-contrast speckle pattern is generated, a large amount of detail can be determined from the camera image of the speckle pattern.

[0027] The spatial deconstruction of the speckle characteristics by the controller provides details, thus giving the controller the ability to distinguish one component or workpiece from another component or workpiece and even from the working surface. As Figure 5 shown, a schematic diagram of the deconstruction of the image of the speckle pattern is shown. In this example, the controller identifies the strongest speckles in the image of the bottom or quarter of the laser speckle pattern on the Figure 4 6061 aluminum sample. After identifying the strongest speckles, the controller reduces these speckles to a single line of data that reveals the X / Y coordinates of the speckles in the sensed image. The controller also deconvolves each speckle to indirectly identify the location of fine scratches on the material surface that are invisible to the laser sensor or camera sensor, which will be further described below. As used herein, deconvolution refers to the process of reversing the optical distortion that occurs in an optical imaging instrument to produce a clear image of, for example, laser speckle. In one embodiment, deconvolution is implemented in the digital domain through a software algorithm that uses the Fournier transform as part of various image techniques. However, alternative methods of implementing laser speckle image deconvolution are also within the scope of the present invention. It should be understood that the process of deconvolution can be used to sharpen the surface image, which may otherwise be unclear due to the rapid movement, jitter, or dynamic movement of the camera or laser projector during image capture, including laser speckle of a wide spectrum.

[0028] In view of the above inventive principles, systems and methods of various embodiments of the present invention that utilize an enhanced scanning system generally shown at Figure 7 will now be described in detail. In a first embodiment, the coherent light source is a laser 12 that projects a green laser beam 14 at 532 nm. The laser beam 14 generated by the laser 12 is redirected in a conventional manner by a first scanning mirror 16 driven by a first galvanometer motor 18 and a second scanning mirror 20 driven by a second galvanometer motor 22. The scanning mirrors 16, 20 direct the laser beam 14 toward a work surface 24 of an assembled workpiece 26 to be characterized in terms of placement and orientation.

[0029] A camera 28 is synchronized with the movement of the galvanometer motors 18, 22 to track the scanning laser beam as it scans the workpiece 26 and the work surface 24. In a typical scanning configuration, the camera 28 monitors the intensity of the return beam 30 as the laser beam 14 scans across the workpiece 26, correlating the detection time with the scanning position.

[0030] To effect verification of the placement and orientation of the workpiece 26, reference is made to Figure 8 . When mated with the work surface 24, the laser line 32 scans parallel to the expected boundary of the workpiece 26. In this embodiment, the outline of the workpiece 26 is known, particularly when a laser template has been projected onto the work surface 24 to guide the placement of the workpiece 26, as described in U.S. Patent Nos. 9,200,899 "LASER PROJECTION SYSTEM AND METHOD", 9,245,062 "LASER PROJECTION SYSTEM USING VARIABLE PART ALIGNMENT", 9,442,075 "GALVANOMETER SCANNED CAMERA WITH VARIABLE FOCUS AND METHOD", and 9,881,383 "LASER PROJECTION SYSTEM WITH MOTION COMPENSATION AND METHOD", the contents of each of which are incorporated herein by reference. When the laser template is projected to guide the placement of the workpiece 26, the controller 34 has determined the boundary of the workpiece 26 on the work surface by identifying the position of the work surface 24 in a three-dimensional coordinate system. Even if the workpiece 26 has been misplaced, the controller 34 knows where the workpiece 26 should be to project the laser line 32. Thus, to sample the laser line 32, the laser beam 14 is scanned toward the boundary of the workpiece 26.

[0031] When the laser beam 14 generated by the laser 12 scans from the workpiece 26 to the working surface 24, the laser speckle pattern suddenly changes, indicating different surfaces of the workpiece 26 and the working surface 24. As Figure 8 shown, the workpiece 16 is shown as a fiber-impregnated composite tape, while the working surface 24 is shown as a previously applied fiber-impregnated composite tape to which the workpiece 16 is applied under the guidance of a laser template. In this embodiment, the fibers disposed in the workpiece 16 have a different orientation from the fibers disposed in the working surface 24, thus presenting a pattern misalignment. Scanning the laser beam 14 to intersect the workpiece 16 (in the direction of arrow 36) across the expected boundary establishes the actual placement of the workpiece by changing the laser speckle characteristics. The boundary between two different speckle characteristics defines the actual boundary of the placed workpiece 26 on the working surface 24. The actual boundary is then compared with the desired boundary visually defined by the laser template during placement. It should be understood that, for clarity, Figure 8 the images in

[0032] are enlarged because the distance between the laser lines 32 is only in pixels and the fibers are not perceptible to the human eye.

[0033] It should also be understood that the fibers disposed in the workpiece 26 and the working surface 24 can be oriented in the same direction, and the orientation of the sampling line 32 is changed to establish the path of the fibers and identify the boundary of the placed workpiece 26. Scanning the laser beam 14 in a direction perpendicular to the fibers provides the greatest speckle interference variation, while scanning the laser beam 14 in the same direction as the fibers produces the most consistent speckle behavior. It is believed that scanning the laser beam in both directions can provide the best results.

[0034] Alternatively, it may be desirable to operate the system 10 with an initially closely supervised assembly process that is used to train a pattern of fiber characteristics for each assembly step. In this case, monitoring may detect any deviation from the earlier placement caused by accidental movement of the earlier placed workpiece 26. Thus, collecting placement and orientation data during the supervised assembly process provides a reference standard for verifying assembly during general production. Further verification is performed when the computer-aided design data includes fiber workpiece 26 or laminate and orientation information to allow image comparison.

[0035] As described above, when a laser projector used to project an optical template is used to guide and supervise the assembly task, the included laser scanner (lasers 10, scanning mirrors 16, 20, and galvanometer motors 18, 22 in this embodiment) is adapted to scan the work surface 24 and the workpiece 26 being inspected. Thus, both the assembly guidance and verification processes are performed by the same system 10, which in this embodiment simplifies the verification placement of the workpiece 26 on the work surface 24. Prior to placing the workpiece 26, boundary samples adjacent to the laser template of the background work surface 24 identify background speckle characteristics and profiles, also referred to as background signatures. In this embodiment, the background signature can be subtracted from the speckle characteristics and profiles measured after placing the workpiece 26. After placing the workpiece 26, the placement and orientation are verified by confirming that the previously identified background speckle characteristics and profiles remain unchanged for the post-placement boundary samples outside the template or the proper placement location of the workpiece. Additionally, verification of the workpiece within the template or proper placement is achieved by identifying alternating speckle characteristics and profiles. When sampling the laser speckle prior to placing the workpiece 26, very sensitive speckle characteristics of the background work surface 24 can be obtained. Thus, precise placement of the workpiece 26 can be established by scanning across the edge of the workpiece 26 in a localized sample area, or by scanning a sample area parallel to the expected boundary position that separates the acceptable placement tolerance, which can be as small as 0.5 mm. When using the speckle characteristics measured prior to placing the workpiece 26, only the samples of the workpiece 26 after placement should vary (be different from the previous samples).

[0036] Figure 9 An alternative system generally designated 110 is shown. In this embodiment, the coherent light source is a laser 112 that projects a green laser beam 114 at 532 nm. The laser beam 114 generated by the laser 112 is redirected in a conventional manner by a first scanning mirror 116 driven by a first galvanometer motor 118 and a second scanning mirror 120 driven by a second galvanometer motor 122. The scanning mirrors 116, 120 direct the laser beam 114 onto a work surface 124 having an assembly workpiece 126 to be characterized for placement and orientation.

[0037] In this embodiment, the return beam 130 travels along the same path as the projected laser beam 114 to the optical beam splitter 128, which allows the return beam 130 to pass through and reach the light sensor 132. The sensor 132 includes collection optics and sensor electronics, such as, for example, at least one of a photodiode, an avalanche photodiode, and a photomultiplier tube, to provide the necessary speckle detection sensitivity to identify different laser speckle characteristics. However, as long as speckle sensitivity is achieved, other types of optical or laser sensors can also be used for the same purpose. In this embodiment, it is contemplated that a conventional laser projector as described herein can be employed. Thus, with the aid of mirrors 116, 120 and galvanometer motors 118, 120, synchronization of the detection of laser speckle characterization with the movement of the laser beam 114 is easily achieved. A conventional laser scanning system for optical stencil protection is thus re-purposed to provide the function of laser speckle characterization. As in the previous embodiment, when the projected laser stencil is used to guide where the workpiece 126 is to be placed, the controller 134 has determined the boundaries of the workpiece 126 on the work surface by identifying the position of the work surface 124 in a three-dimensional coordinate system.

[0038] The invention has been described in an illustrative manner, and it is to be understood that the terminology used is intended to be of a descriptive nature rather than of a limiting nature. Clearly, many modifications and variations of the present invention are possible in light of the above teachings. Thus, it is to be understood that in the claims, the appended reference numerals are for convenience only and are not limiting in any way, and that the invention may be practiced in a manner different than specifically described.

Claims

1. A method for verifying the placement of a workpiece within an assembly task, comprising the steps of: Projecting a laser template onto a work surface before placing the workpiece onto the work surface and scanning the background adjacent to the template; Placing the workpiece onto the work surface; Scanning a coherent light source across the surface of the placed workpiece and along the work surface that is set as the background of the placed workpiece, thereby illuminating the placed workpiece and the work surface with coherent light; Providing an optical sensing system to record the optical characteristics of interference speckles generated by scattered light from the coherent light; And Classifying the speckle response to discriminate between the surface of the workpiece and the background to verify the placement of the workpiece, wherein the method further comprises the steps of: identifying at least a portion of the background work surface and a sampling area on the placed workpiece.

2. The method according to claim 1 further comprises the following steps: Scanning the background with coherent light before placing the workpiece onto the work surface and storing an image of the background before classifying the speckle response to discriminate between the surface of the workpiece and the background.

3. The method according to claim 1 further comprises the following steps: Decomposing the speckle feature space into individual speckle images.

4. The method according to claim 3 further comprises the following steps: Deconvolving each of the speckle images to reveal deviations in two dimensions.

5. The method according to claim 1, wherein The step of scanning a coherent light source across the surface of the placed workpiece and along the background work surface that defines the background of the placed workpiece is further defined as scanning a coherent laser source to irradiate the placed workpiece and the background work surface with a laser beam.

6. The method according to claim 1 further comprises the following steps: Modifying the sampling area according to the orientation of the fibers provided in the workpiece.

7. The method according to claim 1, wherein, The step of scanning a coherent light source across the workpiece is further defined as scanning the coherent light from the coherent light source relative to the workpiece in two directions.

8. The method according to claim 1, further comprising the steps of: a laser projector scanning the laser template on the work surface and using the laser template as a boundary for scanning the coherent light source.

9. The method according to claim 1, wherein, The step of discriminating between the surface of the workpiece and the background to verify the placement of the workpiece is further defined as identifying the orientation of the scanning of the workpiece relative to the background.

10. A method for identifying the precise placement of a workpiece within an assembly task, comprising the steps of: Identifying a predetermined position of the workpiece on a work surface by projecting a laser template onto the work surface so as to trace the contour of the predetermined position of the workpiece on the work surface, as part of the assembly task; Placing the workpiece at the predetermined position on the work surface; Scanning coherent light from a coherent light source over the workpiece and at least a portion of the work surface to which the workpiece has been assembled; Providing an optical sensing system to sense a first speckle feature of the coherent light reflected from the workpiece and to sense a second speckle feature of the coherent light reflected from the work surface when the coherent light is scanned over the workpiece and the work surface; and Signaling the first speckle feature and the second speckle feature to a controller to distinguish between the first speckle feature and the second speckle feature so as to determine whether the workpiece has been placed at the predetermined position on the work surface, wherein the method further comprises the steps of: the controller identifying at least a portion of the work surface and a sampling area on the workpiece to provide the optical sensing system towards the sampling area.

11. The method according to claim 10, wherein, The step of identifying the predetermined position of the workpiece is further defined as identifying the orientation of the workpiece.

12. The method according to claim 10, wherein The step of scanning coherent light from a coherent light source is further defined as the coherent light source including a laser projector.

13. The method according to claim 10 further comprises the following steps: Signal a second speckle characteristic to the controller before placing the workpiece in a predetermined position.

14. The method according to claim 10, wherein The step of scanning coherent light from a coherent light source is further defined as scanning the coherent light in a first direction and a second direction over at least a portion of the workpiece and the working surface.

15. The method according to claim 10 further comprises the following steps: The controller modifies the sampling area according to the orientation of the fibers disposed in the workpiece.

16. The method according to claim 10 further comprises the following steps: Deconvolve the first speckle characteristic and the second speckle characteristic into individual speckles to identify the Gaussian intensity curve of each speckle, thereby distinguishing the workpiece and the working surface.

17. The method according to claim 10 further comprises the following steps: The controller uses a laser template as a boundary to scan the coherent light source and sense the first speckle characteristic and the second speckle characteristic.

18. The method according to claim 10 further comprises the following steps: Scan the coherent light source in multiple directions along the workpiece and the working surface, thereby identifying the orientation of the fiber material disposed in the workpiece and the working surface.

Images