Video overlay display registered in virtual space

Abstract

The present disclosure relates to a video overlay display registered in virtual space. A method of identifying to an operator the precise assembly of a component onto a workpiece includes using a laser projector and a headset or goggles. The laser projector projects a laser image relative to the workpiece. The headset or goggles include an imaging device and a display element. The positions of the laser projector, the imaging device, and the display element are registered in a common coordinate system relative to the workpiece. The imaging device generates an image of the field of view of the workpiece that appears in the display element when the operator views through the display element. A computer-generated image of the component is overlaid on the image of the workpiece, the component image being disposed at a geometrically precise location on the image of the workpiece. The image of the workpiece that appears in the display element is registered to a three-dimensional coordinate system defined relative to the workpiece.

Description

[0001] Prior application

[0002] This application claims priority to U.S. Patent Application No. 15 / 058,867, filed on March 2, 2016, which claims priority to U.S. Provisional Patent Application No. 62 / 126,756, filed on March 2, 2015, the contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates generally to laser projection and video systems for industrial environments. More specifically, the present application relates to using enhanced video overlays of components on a workpiece in an image generated with the aid of a laser projection system. Background Art

[0004] Manufacturers of complex workpieces and assemblies that require high assembly accuracy are continually seeking to improve processes using assembly aids. While conventional manufacturing processes have historically relied on mechanical fixtures that register with the workpiece at fixed locations as assembly aids, these mechanical fixtures have proven difficult to use and do not provide adequate accuracy when complex equipment requires reduced tolerances. Alternatively, various reference features or templates applied to the workpiece are also used to manually measure the attachment location. Expenses, labor-intensive requirements, and error-proneness remain challenges facing manufacturing companies.

[0005] Figure 1 A schematic diagram of a workpiece 10 is shown. The workpiece 10 includes a component (or subassembly) 12 that mates with a surface 16 at a geometrically important location. As described above, the use of a fixture enables accurate positioning of the component 12 to the workpiece 10. When the fixture is large or heavy, the accuracy is particularly insufficient, making it difficult for the operator to properly position it on the workpiece 10.

[0006] Recently, optical templates have been projected directly onto the workpiece, thereby providing an optical image to locate where to place the component on the workpiece 10. Figure 2 1, wherein a workpiece 10 is subjected to a laser projector 12 that projects a template 14 onto a predetermined surface 16 of the workpiece. One such example of a laser projector 12 for projecting a template 14 onto a workpiece 10 is disclosed in U.S. Patent No. 9,200,899 "LASER PROJECTION SYSTEM AND METHOD," the contents of which are incorporated herein by reference. In this example, the position of the projector 12 relative to the workpiece, as well as the three-dimensional geometry of the workpiece and the contour of the attachment location are predetermined. With these elements, the outer shape of the attachment contour is accurately projected onto the desired workpiece surface 16 in the manner of an optical template 14.

[0007] Systems for projecting the template 14 onto the workpiece have proven to be highly accurate. However, there are limitations that rely solely on laser projection of the template 13 that have limited the wider use of optical templates in the market. For example, it is sometimes difficult or impossible to position the workpiece relative to the laser projector within a three-dimensional coordinate system. In addition, arbitrary assembly tasks that traditionally use mechanical fixtures placed in arbitrary three-dimensional positions are not always uniquely suited for laser projection of templates 14. In some cases, a support surface is available, but the shape template can only be accurately projected onto the first surface of the workpiece. In addition, when it is necessary to verify correct part selection, placement, or orientation, three-dimensional parts fixed to the workpiece prevent the projection of patterns that "look like" parts.

[0008] The projection of the laser template 14 is typically flattened to be received on the surface of the workpiece. Currently, laser projectors are unable to provide an accurate focus point at a location spaced from the surface 16 of the workpiece. This prevents projecting the true location of a part from a computer-aided design model that is three-dimensional or includes geometrically significant features of interest that would float in space above the surface 16 of the workpiece 10.

[0009] It would therefore be desirable to provide a method for overlaying an image of a component to be assembled in a workpiece that provides a three-dimensional characterization of the component relative to the workpiece. Summary of the invention

[0010] A method for identifying precise assembly of a component onto a workpiece includes projecting a laser image relative to the workpiece using a laser projector. An imaging device generates a video image of the workpiece displayed on a display device. A display device generates a video image of the workpiece generated by the imaging device. The laser projector projects a plurality of laser images relative to the workpiece, and the display device registers a field of view of the imaging device relative to the workpiece based on the plurality of laser images. The display device generates a virtual image of the component on the video image of the workpiece. The virtual image of the component is disposed at a geometrically precise position on the video image of the workpiece.

[0011] Another embodiment includes using a laser projector and a headset to identify to an operator the precise assembly of a component onto a workpiece. The laser projector projects a laser image relative to the workpiece. The headset includes an imaging device and a display element. The positions of the laser projector, the imaging device, and the display element are registered in a common three-dimensional coordinate system relative to the workpiece. The imaging device generates an image of the field of view of the workpiece that appears in the display element when viewed by the operator through the display element. A computer-generated image of the component is overlaid on the image of the workpiece, the component image being disposed in a geometrically precise position on the image of the workpiece. The image of the workpiece that appears in the display element and the computer image of the component are registered to a three-dimensional coordinate system defined relative to the workpiece.

[0012] The present invention is a significant improvement over the use of laser projectors as an assembly aid. For the first time, the benefits of laser projection relative to a workpiece are combined with an imaging device and a display device, wherein an operator can view a virtual image of the component to be assembled to the workpiece. The precise measurements that can be made by the laser projector provide the ability to generate a three-dimensional image of the component with a geometrically precise position and orientation on the workpiece, enabling an operator to properly position the component and verify that important features of the component are set in a geometrically precise position spaced from the surface of the workpiece. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Other advantages of the present invention will be readily appreciated as they may be better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0014] Figure 1 Represents workpieces and parts of workpieces;

[0015] Figure 2 represents a prior art laser projector that projects a template of a part onto a workpiece;

[0016] Figure 3 indicates a computer-aided design overlay using laser alignment;

[0017] Figure 4 A computer-aided design overlay utilizing an arbitrary laser alignment reference is shown;

[0018] Figure 5 shows a laser alignment reference projected onto a workpiece using an optical measurement system;

[0019] Figure 6 An alternative embodiment of the method of the present invention using a headset in the form of augmented reality goggles is shown; and

[0020] Figure 7 A headset in the form of augmented reality goggles for use by an operator is shown as another alternative embodiment of the present invention. DETAILED DESCRIPTION

[0021] Reference Figure 3 , a laser projection and video overlay assembly for implementing the method of the present invention is generally shown at 20. The workpiece 10 defines a surface 16, wherein the components 12 are assembled on the surface 16 (such as Figure 1 ). A laser projector 22 projects a laser template 24 onto the surface 16 of the workpiece at a geometrically relevant location.

[0022] The tablet computer 28 includes an imaging device 30 and a display device 32. As will be appreciated by one of ordinary skill in the art, the imaging device on the tablet computer 28 is in the form of a camera, and the display device 32 is in the form of a video screen. Also as will be appreciated by one of ordinary skill in the art, the imaging device 30 is disposed on a side of the tablet computer 28 opposite the display device 32. The description of a tablet computer is merely exemplary, and other devices are also within the scope of the present invention, including but not limited to smartphones, virtual reality devices, headsets including virtual reality, or augmented reality goggles; detachable cameras and video screens, and any device or combination of devices capable of both generating and displaying an image of the workpiece 10. Although "tablet computer" is used throughout the specification, it should be understood that each of these devices is within the scope of the present invention.

[0023] In this embodiment, the laser template 24 is located within the field of view of the imaging device 30 along with the workpiece 10. This creates the illusion that the display device 32 provides a window to the scene outside the tablet computer 28. In addition, in this embodiment, the tablet computer 28 includes a processor that is capable of executing the necessary algorithms to monitor the projected template 24 and calculate the associated field of view or viewpoint of the imaging device 30. Once the registration of the imaging device 30 to the field of view is determined, a virtual overlay 34 based on computer-aided design ("CAD") is placed on the live video stream of the workpiece 10. In addition, even if the tablet computer 28 is moved, the processor continuously updates the geometric position of the workpiece 10 while updating the video overlay 34 based on the CAD model. In this way, a photorealistic rendering of the component 12 set at the desired location of the assembled component 12 is generally represented in the display device 32. The video overlay 34 of the present invention is not constrained by the projected template on the surface 16 of the workpiece 10.

[0024] Another embodiment is generally Figure 4 34 of the display device 32. During assembly, when the component 12 is placed in the attachment position as indicated by the laser template 24, the template 24 projected by the laser projector 22 may be obstructed. Therefore, the accuracy and availability of the overlay 34 may be reduced. In this embodiment, the laser projector 22 is programmed to project an arbitrary laser spot 36 toward the workpiece 10. The imaging device 30 observes the laser spot 36, and the processor aligns the field of view of the imaging device 30 to accurately depict the overlay 34 on the display device 32. An arbitrary projection pattern of at least four laser spots 36 is considered to provide the most accurate overlay 34. However, projecting additional laser spots 36 provides beneficial redundancy and improves the dynamic repositioning of the video overlay 34. In addition, the continuous random projection of arbitrary laser spots 36 reduces the possibility of operator obstruction or obstruction of the assembled component 12 when mating with the workpiece 10.

[0025] It may be desirable to project the laser spot 36 not only onto the surface 16 of the workpiece, but also onto the surrounding surfaces to further facilitate setting the precise position of the workpiece 10 in the three-dimensional coordinate system. In this alternative embodiment, to handle the operator's movement of the tablet 28, the overlay 34 on the image of the workpiece 10 shown on the display device 32 is also rapidly and dynamically recalculated by the processor.

[0026] Additional improvements are desirable when the surface of the environment in which the workpiece 10 is disposed is unknown. In this case, Figure 5 As best represented in , it is desirable to use a measuring device 38. In one such example, the measuring device 38 includes optical coordinate measurement that triangulates the image of each laser spot 36 using cameras 40 spaced a known distance apart. In this way, the measuring device 38 identifies the precise location of the surface 16 of the workpiece 10 to be mated with the component 12. Additional improvements utilize placing a retroreflective target at a known location on the workpiece 10 or by measuring reference features of the workpiece 10 using a handheld probe, as disclosed in U.S. Pat. No. 9,200,899, the contents of which are incorporated herein by reference. Thus, the projected laser spot 36 can be effectively arbitrarily positioned so that it falls on a known or unknown workpiece 10 surface or other surface in the overall environment, which will provide the necessary visibility within the field of view of the imaging device 30.

[0027] The augmented or virtual reality provided by the present invention creates the illusion of a window or portal to an existing environment, which is augmented onto a live video stream by a CAD graphic. Although tablet computers and smartphones have been disclosed above, Figure 6 Another embodiment shown includes a video goggles 42 with an integrated imaging device 44. Many of these devices available on the market today include processing power and the necessary sensors (including accelerometers and MEMS gyroscopes) to quickly register video images and process CAD data to accurately generate augmented reality, as described above. This provides the ability to bring in a three-dimensional graphical representation of a CAD-generated part 12 that responds to high-frequency movement of a tablet 28, projector 22, or other device so that the virtual part 12 appears to be part of the environment at a geographically accurate location on the workpiece 10.

[0028] The above method provides many benefits over traditional laser template projection. Flicker limitations are eliminated, and a high level of extensive detail (including fiber orientation) or even special instructions are provided, all of which are overlaid on the image generated by the display device 32. In addition, the visibility of the shape is not limited by the reflectivity of the background material. In this way, the desired shape is always the most obvious feature in any combination of colors, flickers or other highlights to convey the characteristics of the template shape through the display device 32. When the application is specially adapted to overcome the potential embarrassment of remote viewing and the resolution of the video image, it is most effective to use the enhanced video image generated by the display device 32. For example, with appropriate imaging device position and orientation, direct laser projection for ply layup is extremely effective.

[0029] When additional precision may require limiting the field of view to a very localized, typically featureless area of ​​the workpiece 10, the IRIS offered by Virtek Vision International, Inc. TM A further improvement can be achieved by a spatial positioning system. A spatial positioning system enables any spot projected by the laser projector 22 measured by a measuring device 38 or spatial positioner to be referenced to register the video streams with high accuracy.

[0030] One example includes attaching a rectangular component 12 to a workpiece 10. In this example, the laser projector 22 projects both the template 24 and an arbitrary laser spot 36 onto the workpiece 10 and the surrounding environment. By registering the imaging device 30 according to the arbitrary laser spot 36, the virtual component 12 is overlaid on the actual attached portion in the video stream on the display device 32 based on the CAD data, as long as the laser spot 36 remains in the field of view of the display device 32. Of course, the arbitrary laser spot 36 is also located by a spatial positioning system, which allows the laser or laser to quickly reposition the reference position to keep the arbitrary laser spot 36 in the field of view of the imaging device 30. In this way, the movement of the tablet, smart phone or goggles becomes irrelevant because the registration is maintained to achieve accurate enhancement.

[0031] It should be understood by those skilled in the art that the above description, while disclosing an apparatus and method for accurately positioning a component 12 on a workpiece 10, includes broader aspects of the apparatus and method. For example, the laser projection and video overlay assembly 20 disclosed above provides the ability to identify the precise location of a drilled or machined hole (not shown) into a workpiece. In addition, the laser projection and video overlay assembly 20 identifies to the operator whether the correct component 12, fastener, etc. has been assembled to the workpiece 10.

[0032] Another aspect of the present invention is the ability to block the laser spot 36 or other laser image projected by the laser projector 22. In some cases, the laser spot 36 may cause confusion or otherwise be undesirable when shown in the display device 32. One example is if the virtual overlay 34 is used for entertainment purposes. In this example, the impact of the virtual overlay 34 is reduced. In this case, the tablet 28 is programmed to block the laser spot 36.

[0033] The use of goggles 42 or a headset including goggles presents another challenging aspect, namely, generating a video overlay of the part 12 on the workpiece 10 at a geometrically precise location defined relative to the three-dimensional coordinate system of the workpiece 10. "Goggles" and "headset" are used interchangeably throughout the specification, and it should be understood that the term "goggles" includes devices worn on the eyes of the operator, as well as any device worn on the operator's head that provides a hands-free viewing feature or display device 45. Because it is impossible to calibrate the viewing angle of an operator wearing goggles 42, a unique method of registering the video overlay 34 of the part 12 on the workpiece 10 has been developed. In addition, the position of the goggles 42 placed on the operator's head is prevented from shifting and destroying the accuracy of the video overlay 34 of the part when the operator is aware of it.

[0034] As explained in detail above, the goggles 42 (and more specifically, the integrated imaging device 44) are aligned with the workpiece 10 by laser projection of a laser image 36 generated by the laser projector 22. Once the integrated imaging device 44 is aligned with the workpiece 10 by detecting the pattern of the laser image 36 projected onto the workpiece 10 toward the workpiece 10, a video overlay 34 of the part 12 is generated by the goggles display element 45. Although the video overlay 34 generated by the goggles display element 45 is precisely aligned with the workpiece 10 in the goggles display element 45, the position of the video overlay 34 may be skewed when perceived by an operator. However, the operator's perception is sufficient to locate the place where the part 12 is placed on the workpiece 10 at a generally precise location defined by the computer-generated graphic design. Additionally, for further accuracy, the goggles 42 include an environmental tracking feature, such as such sold by Microsoft Corporation. Provided by the system.

[0035] The environmental tracking features include one or more sensors that cooperate with the imaging device 44 to generate a three-dimensional range image and track the three-dimensional configuration of the environment and inertial monitoring to monitor the dynamic movement and orientation of the goggles 42. For further accuracy, the imaging device 44 can include more than one camera (even four cameras) to generate an accurate three-dimensional range image. In addition, dynamic movement is unpredictable movement of the goggles 42, as expected by the movement of the operator's head. However, real-time updates of the three-dimensional range image and inertial monitoring are sufficient to determine the approximate or approximate position of the part. In other ways, the virtual field of view 46 removes the necessity of accurately positioning the goggles 42 relative to the workpiece 10 because the video stream of the field of view includes the laser image 36, which provides a reference for the position of the video overlay 34 relative to the virtual workpiece 48.

[0036] Once approximate placement of the component 12 on the workpiece 10 is achieved, the operator triggers a virtual field of view 46 or a mixed reality field of view of the workpiece 10 and the component 12. The virtual field of view 46 is a live stream, i.e., a real-time video image, of the workpiece 10 generated by the display element 45 as viewed by the operator while wearing the goggles 42. In one embodiment, the virtual field of view 46 is disposed in a space between the operator and the actual workpiece 10 in the operator's line of sight. In another embodiment, the virtual field of view 46 is disposed in a space offset from the workpiece 10.

[0037] When the workpiece is represented in the virtual field of view 46, the video overlay 34 of the component 14 in the virtual field of view 46 is precisely registered on the virtual workpiece 48. By registering the video overlay 34 of the component 14 on the virtual workpiece 48 as displayed in the virtual field of view 46, operator perceived inaccuracies are completely eliminated and very tight assembly tolerances can be achieved. The video overlay 34 is an electronically generated image of the component 12 at a geometrically precise location on the virtual workpiece 48. The video overlay 34 provides a visual indication of where the component 12 is to be placed during the assembly process.

[0038] After the part 12 is placed, the video overlay provides the operator with information about whether the part 12 has been properly placed on the workpiece 10. By aligning the goggles 42 according to the laser spot 36 and the environmental tracking features, the optimal placement of the virtual field of view 46 is automatically determined to most closely match the operator's viewpoint without requiring the operator to hold the actual device for trial and error. In addition, the virtual field of view 46 does not obstruct the projected pattern of the laser spot 36, further facilitating the alignment of the goggles 42 with the workpiece 10. The real-time video stream of the workpiece 48 also includes a video image of the laser image 36, enabling the video overlay 34 to be aligned with the virtual workpiece 48 as shown in the virtual field of view 46.

[0039] Additional alignment feedback may be obtained by generating a CAD image of the part 12 in the virtual field of view 46. In the display, the CAD image may be "ghosted" at the appropriate alignment location on the virtual workpiece 46. However, it is believed that virtual alignment of the video overlay 34 on the virtual workpiece 48 may be achieved by detecting the laser image 36 alone and without the need for a CAD overlay.

[0040] The environment tracking feature of the goggles 42 provides assistance to the laser projector 22 to project the laser spot 36 into the field of view of the goggles 42. As described in more detail above, the environment tracking feature updates the field of view of the goggles 42 (and thus the operator's field of view) in real time and signals the field of view to the laser projector 22. The laser projector 22 uses the information received from the environment tracking feature to continuously update the position and orientation of the integrated imaging device 44 of the goggles 42 to optimize the projection of the pattern of the laser spot 36. In one embodiment, the environment tracking feature is a computer interpretation of the scene observed by the operator.

[0041] To provide further clarity to the operator, the virtual field of view 46 is cropped and resized to the operator's actual field of view of the workpiece 10. Thus, in size, the image of the workpiece 48 is the same size as the actual workpiece 10 as perceived by the operator, such that edges and related features are equally spaced.

[0042] Once the operator places the component 12 onto the workpiece 10, proper placement is verified by a controller or computer (not shown) for generating a virtual image. The overlay of the component 34 defined by the computer design of the component / workpiece assembly 20 is compared or aligned with the actual placement of the component 10. The exact position can be verified by the operator visually comparing or by the computer comparing the edges, fiducials and related features of the overlay 34 of the component and the placed component 12. The precise registration of the virtual workpiece 48 in the virtual field of view 46 is detected by providing the operator with an indication of where the component 12 should be placed on the workpiece 10 through the imaging device 44. The frame capture of the placement of the component 12 on the workpiece 10 can be recorded as providing evidence of proper component 12 placement in a database. In addition, in another embodiment, the image is processed by a computer to compare the expected position of the designed component 12 with the observed image features of the component.

[0043] Obviously, many modifications and variations of the present invention are possible in light of the above teachings. The foregoing invention has been described in accordance with relevant legal standards; therefore, the description is merely exemplary and not limiting in nature. Variations and modifications to the disclosed embodiments may be apparent to those skilled in the art and do fall within the scope of the present invention. Therefore, only by studying the following claims can one determine the scope of legal protection afforded this invention.

Claims

1. A method for indicating to an operator the precise assembly of a component onto a workpiece, comprising the following steps: providing a laser projector for projecting a laser image relative to a workpiece; providing an operator with a headset, the headset including an imaging device and a display element; detecting a laser image by the imaging device, and registering the positions of the laser projector, the imaging device, and the display element in a three-dimensional coordinate system relative to the workpiece; The imaging device generates a virtual image of a field of view of the workpiece that appears in the display element when viewed by an operator through the display element; overlaying a computer-generated image of the component on a virtual image of a field of view of the workpiece generated by an imaging device, wherein the computer-generated image of the component is disposed at a geometrically precise location on the image of the workpiece; The image of the workpiece appearing in the display element is registered to a three-dimensional coordinate system defined relative to the workpiece.

2. A method as claimed in claim 1, comprising the step of generating an image of the field of view of the workpiece at a position arranged between the workpiece and the display element.

3. The method of claim 1, further comprising the step of providing said headset with an environment tracking feature.

4. The method of claim 3 further comprising the step of the environment tracking feature signaling to the laser projector the position and field of view of the imaging device of the headset.

5. The method of claim 1, wherein registering the positions of the laser projector, the imaging device, and the display element in a three-dimensional coordinate system relative to the workpiece is further defined as projecting a laser image onto the workpiece by the laser projector and detecting a pattern of the laser image by the imaging device.

6. The method of claim 1 further comprising the step of updating the image of the workpiece displayed by the display element in real time to maintain the computer generated part image overlaid at a dimensionally accurate geometric location relative to the workpiece.

7. The method of claim 1 further comprising the step of cropping and scaling the image of the field of view appearing in the display element to match the operator's geometric field of view of the workpiece.

8. The method of claim 1 further comprising the step of said display element generating a video overlay of the component at an initial location on the workpiece prior to aligning the computer generated image of the component at a geometrically precise location on the image of the workpiece.

9. The method of claim 1 further comprising the step of verifying correct placement of the component on the workpiece by comparing the placement of the component on the workpiece to a computer generated image of the component disposed at a geometrically precise location of the image of the workpiece.

10. The method of claim 9, wherein: The step of verifying proper placement of the component on the workpiece is further defined by aligning the image of the component placed on the workpiece with the computer generated image of the component overlaid on the image of the workpiece.

Images