Real-time quality control during manufacturing using augmented reality

Abstract

Various techniques for providing real-time quality control during manufacturing using augmented reality are described herein. In particular, in one embodiment, the techniques herein project a three-dimensional (3D) virtual shape / model onto a work-in-progress part in the real world to see defects in real time, thereby allowing corrections during manufacturing. The embodiments herein generally consist of the following steps: first calibrating the "virtual world" to the real world, then calibrating the physical product to the virtual world, and finally projecting information, such as various design information and / or quality control information, onto the physical product.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 640,877, filed by Borke et al. on March 9, 2018, entitled “REAL-TIME QUALITY CONTROL DURING MANUFACTURING USING AUGMENTED REALITY,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to augmented reality, and more particularly to real-time quality control during manufacturing using augmented reality. Background Art

[0004] Quality control (QC) is a process by which an entity reviews the quality of specific (e.g., all) factors involved in production. In general, although quality can be based on "softer" factors (e.g., work management, performance, ability, integrity, etc.), inspection is the main component of quality control, in which the physical product is visually inspected. Thus, for example, a product inspector is typically provided with a list and description of unacceptable product defects (e.g., cracks or surface blemishes). Early forms of quality control were, for example, drawings of required items, where the item was rejected if it did not match the drawing. However, more complex manufactured products made it difficult to make parts exactly the same as their descriptions, leading to the introduction of tolerances, where it was understood that as long as the product's parts were measured within the tolerances, the product would function adequately. Based on this understanding, the "quality" of many manufactured products can be determined by using devices such as plug gauges and ring gauges.

[0005] However, as manufacturing processes become more complex, larger in scale, and often with lower tolerances, traditional methods of performing quality control are inefficient. For example, traditional quality control techniques often detect defects after the product has been fully manufactured, resulting in wasted time and materials. Summary of the invention

[0006] According to embodiments herein, various techniques utilize augmented reality to provide real-time quality control during manufacturing. In particular, in one embodiment, the techniques herein project a three-dimensional (3D) virtual shape / model onto a work-in-progress part in the real world to see defects in real time, thereby allowing corrections during manufacturing. Embodiments herein generally consist of the following steps: first calibrating the "virtual world" to the real world, then calibrating the physical product to the virtual world, and finally projecting information, such as various design information and / or quality control information, onto the physical product.

[0007] Specifically, in one embodiment herein, the technology herein calibrates a virtual world to a real world having one or more projectors and one or more sensors; calibrates a physical object in the real world to a virtual object in the virtual world based on the one or more sensors; compares the physical object with the virtual object based on the one or more sensors; determines information about the physical object based on the comparison; and projects the information about the physical object onto the physical object using the one or more projectors.

[0008] Other specific embodiments, extensions or implementation details are described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The embodiments herein may be better understood by referring to the following description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements, and in which:

[0010] Figure 1A-1B An example of calibration showing real-time quality control during manufacturing using augmented reality;

[0011] Figures 2A-2C Another example showing calibration using augmented reality for real-time quality control during manufacturing;

[0012] Figure 3 Showing examples of using augmented reality for real-time quality control during manufacturing;

[0013] Figure 4 A schematic block diagram showing an example computing device; and

[0014] Figure 5 Example simplified procedure showing real-time quality control during manufacturing using augmented reality. DETAILED DESCRIPTION

[0015] As discussed above, quality control for complex processes can be difficult and / or often inefficient. For example, for additive manufacturing, such as 3D printing or manual application of layers of material (e.g., fiberglass, carbon fiber, or other composite materials), the process may take days or even weeks. If at the end of the process it is determined that the product was improperly manufactured (e.g., the wrong shape, too much or too little material in specific or all areas, etc.), the finished product must be corrected or discarded entirely. If excess material needs to be removed (e.g., trimmed to match a desired depth / size / shape, etc.), correcting the defect (even if possible) may be wasteful in terms of wasted material and time, in addition to the fact that additional material is added, wherein the costs incurred may include the need for adhesives, pretreatment, etc. Moreover, when trimming excess material, the correction must also ensure that too much material is not removed.

[0016] Therefore, according to embodiments herein, a technique is provided for projecting a three-dimensional (3D) virtual shape / model (e.g., a computer-aided design or "CAD" file) onto a work-in-progress part in the real world to see defects in real time, thereby allowing corrections in manufacturing. As described below, embodiments herein generally consist of the following steps: first calibrating the "virtual world" to the real world, then calibrating the physical product to the virtual world, and finally projecting information, such as various design information and / or quality control information, onto the physical product.

[0017] In operation, the present disclosure first provides a calibration technique to map the virtual simulation space to the real world space. Figure 1A-1B In the system 100, one or more projectors 110 may be configured to project onto a given area (e.g., a clean room, a manufacturing floor, or elsewhere). In addition, one or more sensors 120, such as an example infrared (IR) camera (or laser, lidar, camera, etc.), may be configured to be located at a known position (e.g., locked / stationary, otherwise determined, for example, by "inside-out tracking," as will be understood by those skilled in the art) and arranged to see a portion of the projection area. (Note that during calibration, the sensors may typically be "locked" relative to the projector to determine the sensor position relative to the projector). However, by using techniques that track the position of the sensor (relative to the projector) over time, the sensor position may be movable accordingly.

[0018] According to the techniques herein, at least one projector then projects colored dots 115 at known projector-related landmarks, e.g., five equally spaced points from top to bottom on the centerline of the projection. A reference object 130 (e.g., a block) may also be placed below the projection so that one of the colored projection points is aligned with a specific location of the object 130 (e.g., the center of the top of the block). Using blob detection (an image processing technique that defines "blobs," or grouped sets of pixels in an image given a set of parameters, as is understood by those skilled in the art), the sensor 120 detects the block and stores its virtual reference point (e.g., center point) relative to the sensor and the location of the colored dots relative to the projector as anchor points. This step is repeated multiple times (e.g., four times - twice at two different colored projection points) to obtain multiple (e.g., four) anchor points.

[0019] Generally, at least four anchor points should be measured. For example, these four points are used to draw two lines; one line connects each pair of virtual points that share a color point. The intersection of the formed lines will produce the virtual position of the projector relative to the sensor. Using these lines, the virtual boundary values ​​of the real-world projection area (top, bottom, left, right, near plane, and far plane) are calculated and used to create the perspective projection transformation matrix using the following formula:

[0020]

[0021] Here, assuming a virtual position and rotational coordinate system relative to the projector, n corresponds to the "near" boundary plane of the projection volume, f corresponds to the "far" boundary plane of the projection volume, l corresponds to the left boundary plane of the projection volume, r corresponds to the right boundary plane of the projection volume, t corresponds to the top boundary plane of the projection volume, and b corresponds to the bottom boundary plane of the projection volume.

[0022] Second, orientation techniques are defined herein to match a 3D virtual object (e.g., a CAD file) to a real-world object with unknown orientation, scale / size, and position. (For example, in addition to rotational orientation, the object may be on its side while the corresponding CAD file is in a vertical position.) An example orientation technique is as follows, with general reference to Figure 2A-2B Example 200, with a virtual object 210 and a physical object 220:

[0023] 1. Place three points (actual markers 225 ) on the real world object 220 .

[0024] 2. On the 3D virtual object, set a virtual marker 215 that exactly matches the location of the real-world point 225 (take the cursor in the virtual world and mark the same point on the CAD file as the point marked on the real-world object, for example, an easily recognizable feature such as a corner or an actual marker).

[0025] 3. Measure the height of the real world point 225 from the ground (ie, if the point is on the floor, its height is 0). These are the Y-axis values ​​for each of the three points.

[0026] 4. Due to the above real-world to virtual conversion, the system herein now has the virtual heights of these three points. Using these virtual heights, the technology herein can then determine the pitch and roll of the virtual model / object 210 (eg, CAD model) that matches the actual object 220.

[0027] 5. As Figure 2BAs shown in , the software can then project a crosshair 230 on the real world object 220. The user (or robot) can then drag and match the crosshair projection to match the points / markers "1" and "2" of the real world object, providing the X and Z virtual point values.

[0028] 6. Thus, the virtual x, y, and z of the two real-world markers, and the virtual y of the third marker, are known. From this, the techniques herein can determine the yaw (e.g., by drawing a line between the first two CAD model markers, and another line between the "point cloud" points just calculated, the angle difference between these two lines is the yaw rotation, e.g. Figure 2B As shown in (eg, based on angles a, b, c, and d, as will be understood by one skilled in the art).

[0029] 7. Now, the system herein calculates the distance between the two CAD points (virtual markers 215) and the two point cloud points (actual markers 225). If the two distances are not the same, the CAD file can be scaled to match the point cloud.

[0030] 8. Finally, the system herein obtains the position of the marker 1 of the point cloud and transforms the CAD so that the two markers 1 match. The CAD, point cloud, and real world object are now matched, that is, they have the same orientation, scale, and position.

[0031] like Figure 2A As further described in , a transfer attribute transformation can be performed from a real point on a real world object to a virtual point in virtual space, which can be used to orient / position / scale the virtual representation of the object. First, the technology in this article projects the virtual space onto the real world space and moves the virtual point (which can be actually seen due to the projection) to cover / align / match the corresponding physical feature. This converts the physical position to a virtual position, and the virtual point now represents the virtual position of the physical feature (used to orient / position / scale the 3D model). In other words, this process is about converting from physical to virtual using a projector (overlaying the virtual space on the physical space) as the conversion device. To illustrate this point, Figure 2C A virtual mark / point 215 is shown provided to the projector 110 from the orientation program 240 and projected onto the physical box 220, wherein the sensor first determines that the projected mark is not aligned with the actual mark 225 (e.g., Figure 2C ), the virtual marker can then be moved to align properly (as shown in the "Misaligned" section on the left of Figure 2C) to align the projected virtual marker with the actual marker. Note that in certain embodiments, the software may move the virtual / projected marker to adjust the virtual model to match the real world model, while in alternative or additional embodiments, the actual object may be repositioned to align with the projected marker.

[0032] about Figure 2B , the calibration process also involves the conversion from physical to virtual space. However, in this step (which is technically Figure 2A Previously) attempted to create Figure 2A The required "conversion device". That is, the technology herein uses the projector 110 to project virtual reference points, which our sensors 120 (which measure physical space) can read and use to map (orient / scale / position) the physical space into the virtual space. Once the physical space is perfectly represented in the virtual space, the physical measurements can be converted into Figure 2A In particular, points 1-6 (measurements) are shown to 1) find the position of the projector in the virtual world and thus 2) align the physical / virtual space using the perspective projection matrix transformation.

[0033] Therefore, according to one or more embodiments herein, and generally referring to Figure 3 , quality control measures can now be taken by comparing points in the point cloud on the physical object 320 (e.g., each measurable point) to its matching point in the 3D virtual object (CAD) 310. If the distance is zero, a first color 330 (e.g., green) can be projected, indicating that the real world object correctly matches the CAD file. If the distance is less than X, a second color 340 (e.g., yellow) can be projected to highlight that there are minor differences. If the distance is greater than X, a third color 350 (e.g., red) can be projected to highlight that the difference is large. Other color and tolerance systems can be developed, such as under and over tolerance (e.g., insufficient or excessive material), missing or extra components (e.g., missing fasteners, extra shims, etc.), different degrees / ranges of tolerance (e.g., color shading), blinking / flashing displays, dynamic displays (e.g., indicating that a part or material needs to be moved from a current location to a new location), etc. In this way, the techniques herein generate real-time topological maps to highlight defects during processes such as 3D printing, composite materials, etc., and optionally provide further information such as orientation, context, etc.

[0034] Advantageously, the techniques herein provide real-time quality control during manufacturing using augmented reality. In particular, as described above, the techniques herein provide a visually beneficial quality control system that requires only one or more projectors, one or more sensors (e.g., IR sensors), and appropriate control software that projects (e.g., consistently projects) accurate indicia in real-time as to whether a particular aspect of a manufactured product is correct and / or particularly within a given tolerance. In particular, if an object is moved during manufacturing, then accordingly, by using the above-described orientation techniques, the system herein can also easily recalibrate itself to match the new orientation.

[0035] Although example embodiments are shown and described, it should be understood that various other modifications and changes may be made within the spirit and scope of the embodiments herein. For example, the embodiments described herein are primarily described with reference to a projector that projects actual colors on a manufactured object. However, other embodiments may project other information, such as text, instructions, images, etc. In addition, although the projector is shown as projecting an image on an object ("image projector"), other embodiments may use glasses, headphones, or other visual aids to illustrate quality marks (e.g., only for the user, not projected on the product itself). That is, herein, headphones, tablets, phones, and other devices may be configured as projectors to allow users to overlay digital / virtual models onto real-world objects (augmented reality, virtual reality, etc.).

[0036] Moreover, the embodiments herein may generally be performed in conjunction with one or more computing devices (e.g., personal computers, laptops, servers, specially configured computers, cloud-based computing devices, etc.), which may be interconnected via various local and / or network connections. Various actions described herein may be specifically associated with one or more of these devices, but any reference herein to a particular type of device is not meant to limit the scope of the embodiments herein.

[0037] For example, Figure 4 A schematic block diagram of an example computing device 400 that may be used for one or more embodiments described herein is shown. The example device may include at least one network interface 410, one or more audio / video (A / V) interfaces 415, at least one processor 420, memory 430, and user interface components 470 (e.g., keyboard, monitor, mouse, etc.) interconnected by a system bus 480, and a power supply 490. Other components may be added in the embodiments herein, and the components listed herein are merely examples.

[0038] The network interface 410 includes mechanical, electrical, and signal circuits to transmit data over a link coupled to a computer network. The A / V interface 415 includes mechanical, electrical, and signal circuits to transmit data to / from one or more A / V devices (e.g., cameras, IR sensors, projectors, displays, etc.). The memory 430 includes a plurality of storage locations addressable by the processor 420 to store software programs and data structures associated with the embodiments described herein. The processor 420 may include hardware elements or hardware logic suitable for executing the software programs and operating the data structures 439. The operating system 432 (portions of which typically reside in the memory 430 and are executed by the processor) functionally organizes the machine by invoking operations that support software processes and / or services executed on the machine. These software processes and / or services may include an example calibration process 434 and a real-time quality control projection process 436.

[0039] Those skilled in the art will appreciate that other processors and memory types (including various computer-readable media) may be used to store and execute program instructions related to the techniques described herein. Additionally, while the description illustrates various processes, it is expressly contemplated that the various processes may be implemented as modules that are configured to operate in accordance with the techniques herein (e.g., in accordance with the functionality of similar processes). Additionally, while the processes are shown independently, those skilled in the art will appreciate that a process may be a routine or module within another process.

[0040] As an example, certain aspects of the techniques described herein may be performed by hardware, software, and / or firmware, such as in accordance with the various processes and components described herein, which may include computer-executable instructions executed by the processor 420 and / or associated hardware components to perform functions related to the techniques described herein.

[0041] By generally referring to the above technology, Figure 5 An example simplified process for real-time quality control during manufacturing using augmented reality according to one or more embodiments described herein is shown. Process 500 may begin at step 505 and proceed to step 510, where, as described in more detail above, the techniques herein calibrate a virtual world to a real world having one or more projectors and one or more sensors (e.g., infrared (IR) sensors, light radar (LIDAR) sensors, lasers, cameras, etc.). Note that, as described above, the one or more sensors may be fixed to known locations in the real world, or may be movable in the real world (such that the method will further include determining the location of each of the one or more sensors in the real world, for example, by inside-out tracking).

[0042] Additionally, as described above, calibrating the virtual world to the real world may include projecting a plurality of points (mapped in the virtual world) into the real world, sensing positions of the projected plurality of points in the real world, and associating the sensed positions with the mapped points in the virtual world. As described above, one or more of the plurality of points may be projected onto one or more reference objects in the real world, and may be distinguished from one another in the plurality of points.

[0043] The techniques herein also calibrate the physical object in the real world to the virtual object in the virtual world based on the one or more sensors in step 515. As described, such calibration can be based on performing orientation techniques to determine the orientation, scale, and position of the physical object relative to the virtual object based on the one or more sensors.

[0044] In step 520, the techniques herein may then compare the physical object to the virtual object based on the one or more sensors to determine information about the physical object based on the comparison in step 525. For example, the information may include design information (what the object should look like), manufacturing instruction indications (e.g., next step, location of a component, etc.), or quality control information. With respect to determining the quality control information, this may be performed as described above based on the differences between the virtual object and the physical object based on the comparison.

[0045] In step 530, the technology herein may then project the information about the physical object onto the physical object using the one or more projectors. For example, the quality control information may include color-coded indications of quality, indications of one or both of too little or too much material, etc. Text, images, motion, animation, etc. may also be projected for quality control, design, manufacturing instructions, etc.

[0046] The simplified procedure 500 ends at step 535, and in particular, it is optional to continue detecting defects / differences, redirecting the object (for example, detecting that the physical object has moved in the real world, and recalibrating the physical object in the real world to the virtual object in the virtual world based on one or more sensors), etc.

[0047] It should be noted that although specific steps within process 500 may be optional as described above, Figure 5 The steps shown in are only examples for illustration, and specific other steps may be included or excluded as needed. In addition, although the specific order of these steps is shown, this order is only an example, and any suitable arrangement of these steps may be used without departing from the scope of the embodiments herein.

[0048] The above description relates to specific embodiments. However, it is clear that other changes and modifications may be made to the embodiments to achieve some or all of their advantages. For example, it is expressly contemplated that the specific components and / or elements described herein may be implemented as software, which is stored on a tangible (non-temporary) computer-readable medium (e.g., hard disk / CD / RAM / EEPROM / etc.), which has program instructions executed on a computer, hardware, firmware, or a combination thereof. Accordingly, this description is for example only and does not limit the scope of the embodiments herein. Therefore, the appended claims are intended to cover all such changes and modifications that fall within the true spirit and scope of the embodiments herein.

Claims

1. A method for providing real-time quality control, the method comprising: calibrating the virtual world to a real world by a process, the real world having one or more projectors and one or more sensors; By the process, a physical object in the real world is calibrated to a virtual object in the virtual world based on the one or more sensors; By the process, the physical object is compared to the virtual object based on the one or more sensors; By the process, determining information about the physical object based on the comparison, wherein the information about the physical object includes quality control information, the quality control information including an indication of one or both of too little or too much material; and Through the process, the information about the physical object is projected on the physical object using the one or more projectors.

2. The method of claim 1, wherein: This information includes design information.

3. The method of claim 1, wherein: This information includes manufacturing instruction identification.

4. The method of claim 1, wherein: The one or more sensors are selected from the group consisting of an infrared (IR) sensor; a light radar (LIDAR) sensor; a laser; and a camera.

5. The method of claim 1, wherein: The one or more sensors are locked to known locations in the real world.

6. The method of claim 1, wherein: The one or more sensors are movable in the real world, the method further comprising: A location of each of the one or more sensors in the real world is determined.

7. The method of claim 6, wherein: Determining the position includes inside-out tracking.

8. The method of claim 1, wherein: Calibrating the virtual world to the real world includes: Projecting a plurality of points mapped in the virtual world into the real world; sensing positions of a plurality of points projected in the real world; and The sensed position is associated with the mapped point within the virtual world.

9. The method of claim 8, wherein: One or more of the plurality of points are projected onto one or more reference objects in the real world.

10. The method of claim 8, wherein: One or more of the plurality of points may be distinguishable from one another in the plurality of points.

11. The method of claim 1, wherein: Calibrating the physical object in the real world to the virtual object in the virtual world includes: An orientation technique is performed to determine the orientation, scale, and position of the physical object relative to the virtual object based on the one or more sensors.

12. The method of claim 1, further comprising: detecting that the physical object has moved in the real world; as well as Based on the one or more sensors, the physical object in the real world is recalibrated to the virtual object in the virtual world.

13. The method of claim 1, wherein: The one or more projectors are selected from the group consisting of an image projector; a headset; a tablet; a phone; and smart glasses.

14. A tangible, non-transitory computer readable medium providing real-time quality control, the tangible, non-transitory computer readable medium comprising instructions executable by a processor to perform a process comprising: calibrating the virtual world to a real world having one or more projectors and one or more sensors; calibrating the physical object in the real world to the virtual object in the virtual world based on the one or more sensors; comparing the physical object to the virtual object based on the one or more sensors; determining information about the physical object based on the comparison, wherein the information about the physical object includes quality control information, the quality control information including an indication of one or both of too little or too much material; and The information about the physical object is projected onto the physical object using the one or more projectors.

15. The computer-readable medium of claim 14, wherein: Calibration includes: Projecting a plurality of points mapped in the virtual world into the real world; sensing positions of a plurality of points projected in the real world; and The sensed position is associated with the mapped point within the virtual world.

Images