SYSTEM AND METHOD FOR INSPECTING A WORK SURFACE - Patent application

Abstract

A method for assessing a surface is presented, the method including imaging the surface with an imaging system. The imaging includes positioning a camera of the imaging system proximate to the surface. The imaging also includes moving the imaging system and the surface relative to one another such that a distance between the imaging system and the surface is substantially maintained. The imaging also includes acquiring image data of the surface, the image data being acquired in a near-dark field mode or a dark field image mode. Additionally, the method includes analyzing the image data to detect a topography and / or appearance of the surface. The method also includes generating an assessment for the surface based on the detected topography and / or surface appearance.

Description

[Background technology]

[0001] Many industrial processes involve applying materials to a work surface. It is often critical that the material is applied evenly and without trapped air bubbles, wrinkles, or foreign bodies. In surface preparation applications, such as painting applications (e.g. primer polishing, clear coat defect removal, clear coat polishing, etc.), adhesive dispensing, film wrapping applications, or material removal systems, there is a need for technology suitable for the use of abrasive articles and / or robotic inspection and repair. Summary of the Invention

[0002] A method for evaluating a surface is presented, the method including imaging the surface with an imaging system. The imaging includes positioning a camera of the imaging system proximate to the surface. The imaging also includes moving the imaging system and the surface relative to one another such that a distance between the imaging system and the surface is substantially maintained. The imaging also includes acquiring image data of the surface, the image data being acquired in a near-dark field mode or a dark field image mode. The method also includes analyzing the image data to detect a topography and / or an appearance of the surface. Additionally, the method includes generating an assessment for the surface based on the detected topography and / or surface appearance. [Brief description of the drawings]

[0003] The drawings are not necessarily drawn to scale and like parts may be designated with the same numerals in different drawings. These drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the present document.

[0004] [Figure 1] 1 illustrates a film wrapping process in which embodiments of the present invention are useful.

[0005] [Figure 2A]1 illustrates the operation of a line scan array imaging system in accordance with embodiments described herein. [Figure 2B] 1 illustrates the operation of a line scan array imaging system in accordance with embodiments described herein. [Figure 2C] 1 illustrates the operation of a line scan array imaging system in accordance with embodiments described herein. [Figure 2D] 1 illustrates the operation of a line scan array imaging system in accordance with embodiments described herein. [Figure 2E] 1 illustrates the operation of a line scan array imaging system in accordance with embodiments described herein.

[0006] [Figure 3A] A line scan array imaging system for curved surfaces is shown. [Figure 3B] A line scan array imaging system for curved surfaces is shown.

[0007] [Figure 4] 10A-10C illustrate methods of preparing and evaluating a surface after application of a material according to embodiments described herein.

[0008] [Diagram 5] 1 illustrates a surface with detected defects that are addressed in one embodiment.

[0009] [Figure 6A] 1 illustrates layers of a respirator that can be assembled and tested using the systems and methods described herein. [Figure 6B] 1 illustrates layers of a respirator that can be assembled and tested using the systems and methods described herein. [Figure 6C] 1 illustrates layers of a respirator that can be assembled and tested using the systems and methods described herein. [Figure 6E]1 illustrates layers of a respirator that can be assembled and tested using the systems and methods described herein. [Figure 6F] 1 illustrates layers of a respirator that can be assembled and tested using the systems and methods described herein.

[0010] [Figure 7A] 1 illustrates a glass welding helmet that can benefit from the systems and methods described herein. [Figure 7B] 1 illustrates a glass welding helmet that can benefit from the systems and methods described herein.

[0011] [Figure 8A] 1 illustrates a prosthetic fitting that can benefit from the systems and methods described herein. [Figure 8B] 1 illustrates a prosthetic fitting that can benefit from the systems and methods described herein.

[0012] [Figure 9A-1] Tissue samples before and after negative pressure treatment are shown. [Figure 9A-2] Tissue samples before and after negative pressure treatment are shown. [Figure 9B-1] Tissue samples before and after negative pressure treatment are shown. [Figure 9B-2] Tissue samples before and after negative pressure treatment are shown. [Figure 9B-3] Tissue samples before and after negative pressure treatment are shown. [Figure 10] 1 illustrates a microreplicated surface that can be inspected using the systems and methods described herein.

[0013] [Figure 11] 1 illustrates an adhesive dispensing operation that can benefit from the systems and methods described herein.

[0014] [Figure 12]1 illustrates an example of a topography detection system according to embodiments described herein.

[0015] [Figure 13] 1 illustrates a surface inspection system according to embodiments described herein.

[0016] [Figure 14] 1 illustrates a driven robotic unit to which a topographical mapping unit can be attached according to embodiments described herein.

[0017]

[0018] [Figure 15] 1 illustrates a method for evaluating a surface according to embodiments described herein.

[0019] [Figure 16] 1 shows the architecture of a defect inspection system.

[0020] [Figure 17] 1 illustrates an example of a computing device that can be used in the embodiments illustrated in the previous figures. [Figure 18] 1 illustrates an example of a computing device that can be used in the embodiments illustrated in the previous figures. [Figure 19] 1 illustrates an example of a computing device that can be used in the embodiments illustrated in the previous figures.

[0021] [Figure 20A] 1 shows an example of a surface treatment and associated calculations. [Figure 20B] 1 shows an example of a surface treatment and associated calculations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Recent advances in imaging technology and computing systems have allowed many surface preparation processes to be performed at high speeds. Adhesives can be prepared, dispensed, and cured automatically. However, defects are often not detected during the process, and only during manual inspection after the process is complete. However, while some defects are tolerable, others are not, and may result in defective products that may or may not be repairable. Also, many defects can be corrected by in-process inspection. A system capable of rapid inspection is needed to improve product acceptance rates without slowing down the process.

[0023] As automated imaging of work surfaces improves, the ability to automatically process work surfaces can improve as well. It is desirable to be able to detect and repair defects with minimal manual intervention. However, as described herein, many work surfaces present challenges for imaging and quality control. These include curvature, sharp surface features (corners, bends, grooves, or irregularities that deviate significantly from a flat surface in the area requiring inspection), etc. Similarly, obtaining a high-precision image of a post-processed work surface is difficult. It is important to quantify, classify, and characterize defects on processed surfaces. To obtain high-precision images and perform quantitative evaluation, a more detailed understanding of the physical location of the surface in relation to the imaging system is required.

[0024] Presented herein are embodiments of imaging systems and methods of use that may be beneficial for many processes. The imaging systems described herein may be used on curved, flat, or irregular surfaces. Some embodiments include systems that use a known expected topography (e.g., from a captured CAD model or other known information). In some embodiments, the systems include a distance sensor or distance sensor array that aids in mapping the topography of the work surface.

[0025] As described herein, the term "work surface" is used broadly to refer to a surface that undergoes a process that adds or removes material from the surface. Similarly, the term "material" is used broadly and may refer to, for example, solid materials (e.g., wrappings or film layers), liquid materials (e.g., adhesives), hardenable materials, 3D printed filaments or structures, etc. The systems and methods described herein are not limited to the exemplary use cases presented herein.

[0026] The term "defect" as used herein refers to an irregular surface that interferes with visual aesthetics, interrupts the pattern of a surface, or a foreign object trapped in or on a surface. A defect may be removable after imaging and prior to application of material, may be repairable after material application, or may cause the work surface to be unsuitable for its intended use.

[0027] FIG. 1 shows a wrap 120 applied to the driver's side of a vehicle 100. Wraps 120 are becoming increasingly popular as a means of advertising on vehicles. The goal is to apply a flat, smooth film that looks like paint. Wraps 120 may include a thin film layer that is intended to be applied directly to the surface of the vehicle so that the desired message appears painted.

[0028] However, as shown, the door of the vehicle 100 has a curvature 110 and the wrap 120 spans the gap 130 between the driver door and the rear passenger door. Although the location of the vehicle can be automatically detected and the wrap 120 can be automatically applied, and the curvature 110 is available from a CAD model of the vehicle 100, a system or method is needed to inspect the surface to which the wrap 120 is applied. Inspection before applying the wrap 120 may be useful to detect and remove dirt or foreign objects before applying the wrap 120. Inspection after application may be useful for a quality check, for example, to identify and remove air bubbles or wrinkles.

[0029] As described herein, the inspection may be performed immediately before or after repair using, for example, an imaging system that may be mounted on a moving robotic arm, an Unmanned Aerial Vehicle (UAV), or may have its own locomotion mechanism.

[0030] In some embodiments, a CAD model may not be available or the exact location of the vehicle 100 may be unknown. In that case, it may be appropriate to first identify the topography 110 of the vehicle 100 and use that information to apply the wrap 120. In some embodiments, multiple passes are performed. For example, a first pass may identify a particular area (e.g., abnormal topography, suspected defects, etc.) and a second pass may provide more detailed information about the topography or an image of the defect. The additional information may be useful in providing information to the defect repair technician or system. In other embodiments, a second or third pass after the repair is performed to verify that the defect has been repaired and to verify how the repair has affected the surface, i.e., whether the surface is now acceptable.

[0031] Currently, many of the quality control processes involved in aesthetics, such as vehicle wraps, are performed by human technicians and are therefore subjective. Using a mobile vision system provides a more objective quality control criterion. In some embodiments, the system outputs location information of the detected defects, a classification of the defect (e.g., trapped foreign object or air bubble) and / or a severity of the defect. Based on information regarding the number of defects detected, the system may also provide an overall assessment of the acceptability of the graphic film installation.

[0032] 2A-2G illustrate the operation of a line scan array imaging system, such as is suitable for inspecting a vehicle 100 before or after application of a wrap 120. FIG. 2A shows a line scan camera array system 200 with a line scan array 210 behind a lens 212. The array system places a light source 220 behind a knife edge 222 and is aimed at a surface 202 so that it appears dark or gray overall. The array 210 captures a linear sequence of images, which are combined to form an image of the surface, as shown in FIGS. 2D and 2E. As the line scan array 210 passes over defects or obstructions on the surface, it refracts the light differently. If something on the surface scatters or refracts the reflected light, the image will appear darker (if it refracts toward the knife) or lighter (if it refracts away from the knife). The images in FIGS. 2D and 2E show this effect on a large defect 250 and a more subtle defect 260.

[0033] Figure 2D shows a defect on a surface detected using a line scan array. The bright areas shown in Figure 2D are caused as the system passes over the defect on the surface. Line scan arrays such as those shown in Figures 2A-C are very sensitive to light refraction. A robotic system may be useful to control a line scan array system as it is capable of precise movement and control.

[0034] Especially for highly reflective surfaces, e.g., shiny paintwork, mirrors, etc., the angle of the light source and the camera relative to the sample is a key parameter for characterizing the surface, just as a human observes a surface at different angles to see different reflections of the surface. Line scan arrays such as system 200 offer the added advantage of being able to adjust sensitivity by adjusting how close the imaging is to the knife edge. However, it is expressly envisioned that the systems and methods described herein will be useful for a variety of surfaces with different textures and reflectivities. Line scan arrays can be tuned to specific wavelengths to achieve maximum edge definition accuracy.

[0035] Line scan array systems also work for both specular and matte surfaces. Imaging systems that can quantify surface parameters, such as slight changes in height that indicate potential deviations from the expected topography, can help fine-tune the automated defect removal process. It is desired to polish as much as possible to remove defects, polish enough to achieve the required surface finish, and manage device settings such as applied force, dwell time, and speed of movement to reduce haze and scratches. The systems and methods described herein provide useful feedback to improve robotic control.

[0036] 2A-2E show example configurations of a line-scan array imaging system that may be useful for imaging defects in a near-dark-field imaging mode. In some embodiments, different configurations are used in the near-dark-field imaging mode.

[0037] In the embodiments described herein, it is contemplated that three passes are performed over the surface: a first pass to obtain an initial topography of the surface, a second pass to image in dark field mode, and a third pass to image in near dark field mode. However, as described herein, in some embodiments, the first pass may occur before processing operations. It is also contemplated that near dark field imaging may occur before dark field imaging. Depending on the application, more or fewer passes may be performed than those shown or in different configurations.

[0038] Once an image is captured using the imaging device 220, different analysis techniques can be applied to better characterize and quantify the defect information. For example, deflectometry can be used to detect quantitative height value information, whereas a line scan image array alone can only provide qualitative data of defect height. However, line scan image array data appears to be consistent with human visual perception. Deflectometry is particularly useful for highly reflective images, as it can generate sufficient fringe patterns.

[0039] 3A and 3B show a line scan array imaging system for curved surfaces. As shown in FIG. 1, many vehicles have curved surfaces. However, to obtain high-precision images with a line scan array and perform image processing and quantification, it is necessary for the sensor mechanism to be at a known position (distance and angle) from the reflecting points on the surface. Although it may be possible to have access to a 3D model (e.g., a computer-aided design (CAD) model), such a model may be inaccurate or insufficient to locate the reflecting points with sufficient accuracy. With a basic understanding of the surface topography, it is desired to provide a line scan array with a distance sensor to obtain a very accurate topographical surface of the vehicle. However, it is expressly contemplated that in some embodiments, a system such as system 300 may be used to obtain an initial topography.

[0040] It is also necessary that the line scan array be placed at the correct angle relative to the surface being imaged. It is desired that there be an angle between the line scan array and the light source perpendicular to the surface. In some embodiments described herein, a distance sensor is placed that passes first through the work surface to obtain accurate distance and curvature information, followed by a second pass by the line scan array. In the second pass, the line scan array can be moved to achieve a position perpendicular to the surface at each point being inspected. In other embodiments, a distance sensor is placed in front of the line scan array. Based on feedback from the distance sensor, the position of the line scan array relative to the work surface is adjusted on the fly.

[0041] 3A shows a schematic diagram of an imaging system 300 for imaging a surface 302. A line scan array 310 is positioned behind a lens 320 such that the array 310 captures an image of the surface 302 at a point 304 perpendicular to the surface 302 and at a right angle to a light source 340.

[0042] The imaging system 300 also includes a distance sensor or distance sensor array. Since many vehicles have surfaces with curvature in multiple directions, it is important to obtain at least distance information for the distance the length of the array 310 traverses. As noted above, in some embodiments, the distance sensor moves separately from the system 300 and is shown as sensor location 330b. In some embodiments, sensor location 330b represents the real-time location of the sensor relative to the system 300, as the sensor array moves across the surface 302 in front of the system 300, as shown by arrow 306. Sensor location 330b illustrates an embodiment in which the sensor array moves independently of the system 300. However, it is expressly contemplated that the sensor array may be mechanically coupled. However, in some embodiments, sensor location 330b illustrates the sensor array moving on a first pass before the system 300 moves along the path 306.

[0043] In some embodiments, the sensor array is mechanically coupled to the system 300, shown as sensor location 330a, such that the sensor array moves along a path 306 at a fixed location relative to the system 300. The entire system 300, with the sensor array located at sensor location 330a, moves across the surface 302 on a first pass for the range sensor to capture the exact topography of the surface 302, and then on a second pass the system 300 captures an image of the surface 302.

[0044] As shown in the transition from Figure 3A to Figure 3B, the orientation of the system 300 changes to maintain a perpendicular angle to the point 304 being imaged. Based on the information from the position sensor array, a robotic arm, UAV, or other movement mechanism of the system 300 rotates and moves the system 300 to maintain a desired distance and orientation relative to the surface 302. For surfaces with zero Gaussian curvature, e.g., cylindrical surfaces, one sensor array is required. However, for non-zero Gaussian curvature surfaces, e.g., spherical surfaces, multiple sensor arrays may be used.

[0045] 4 illustrates a method for preparing and evaluating a surface after application of a material according to embodiments described herein. Method 400 may be used to evaluate a surface undergoing processing or any suitable surface where imaging is beneficial. The surface may be flat, curved, irregular, or have features such as corners, localized hills or valleys.

[0046] At block 410, the surface is prepared for operation. This includes positioning the surface relative to the imaging system, as shown in block 402. This may include moving the imaging system to the surface or positioning the surface in a location known to the imaging system. In some embodiments, the imaging system is equipped with a movement mechanism capable of moving in three dimensions to get closer to the surface and follow the topography of the surface. In some embodiments, the imaging system is mounted on the end of an arm of a driven robotic unit. In some embodiments, the imaging system is mounted on a remote controlled UAV. Surface preparation may also include cleaning the surface, as shown in block 404, and other preparation steps, as shown in block 406.

[0047] In block 420, the surface is prepared, which may include applying material as shown in block 412, removing material as shown in block 414, or other operations as shown in block 416. It is contemplated that in some applications the surface will not be prepared and will only be imaged and inspected.

[0048] At block 430, the surface is inspected. In some embodiments, the surface is inspected in real time as information is captured by the imaging system. In other embodiments, the surface analyzer may not complete the analysis until the relevant surface area has been completely imaged. The captured images may be processed in a manner described herein or other suitable manner to detect defects and topography.

[0049] At block 440, the surface is evaluated. Based on an analysis of the captured images, an assessment of the suitability of the surface is made. If the surface is satisfactory and approved (e.g., in FIG. 1, air bubbles are small enough or in an inconspicuous location), it is discarded as shown at block 432. Alternatively, in the case of FIG. 1, if the wrinkles are unacceptably large or the film is stretched improperly, it is discarded as shown at block 434. Alternatively, it may be repairable as shown at block 450 (e.g., air bubbles are detectable but repairable). The method returns to block 420 as shown in FIG. 4, and if the problem is resolved by applying 412 or removing 414 additional material, the surface is re-trimmed.

[0050] FIG. 5 illustrates a surface where a detected defect is addressed in one embodiment. Surface 500 is an Embossit® dressing, with an air bleed liner 510. Liner 510 and dressing 500 are transparent. As shown, the transparent liner 510 is only present on a portion of the dressing. Typically, only strips remain at the interface between the handling bar and the polyurethane film. Currently, defective products are identified manually. However, it would be useful to have a system that can automatically sort dressings where the liner has not been fully removed. The current production rate for such dressings is about 80 feet per minute. In some embodiments, a topography detection system may be fixed above the moving product line. The illustrated dressing is about 1 millimeter high and on the order of a few millimeters wide. The liner is about 0.1 millimeters high. The change in height due to the presence of the liner may be detectable. Detection may sound an alert to remove the remaining liner on the spot in some embodiments. In other embodiments, the location and product number or other suitable identifiers are recorded and provided to a repair technician or automated repair system.

[0051] Surface 500 includes several intended features 502. Liner 510 should be removed from paper handle 512. A topography detection system such as described herein can detect the change in height due to the presence of liner 510 or detect the presence of a liner based on the difference in reflectivity between liner 510 and paper handle 512.

[0052] 6A-6E show a respirator assembly that may be tested using the systems and methods described herein. The respirator is designed to fit over the wearer's nose and mouth and seal tightly against the wearer's face, with all air passing through layers designed to filter different target contaminants from the surrounding environment, allowing the user to inhale contaminant-free air. The respirator includes an assembly 600 made up of multiple layers. As shown in FIG. 6, these layers include a hydrophobic layer 610, one or more electrostatic layers 620a-b, and a biocompatible layer 630 that contacts the wearer's skin. Adjacent layers must be effectively sealed to allow air to flow along paths 642-644-646-648 from the outside air 640 to the interior region 650 and back along paths 652-654-656-658. If the seal is incomplete, air will be forced out (or drawn in) through the incomplete seal. This will expose the user to unfiltered air that leaks through gaps between the layers.

[0053] FIG 6B shows a CAD model that may be the basis for a 3D printed ventilator mold. To form mold portion 660, a 3D printing system deposits polymer layers to form a mold shaped to cover and protect the mouth and nose. Respiratory portion 660 has multiple attachment points 662 to receive fasteners. As shown in FIG 6C, 3D printing may produce a mold that includes poor attachment areas 664, such as molding defects or tears.

[0054] FIG 6C shows a CAD model of multiple components that form the respirator. Part 672 is molded as described in connection with FIGS. 6A-6B or in other suitable manner. Parts 674, 676, and 678 are all 3D printed components. Filter 679 is placed into component 676 and secured by component 678. If all 3D printed components are defect free, air will pass through filter 679, removing contaminants and preventing exposure to the wearer.

[0055] However, if any of the components 672-678 have gaps or pinholes, if the components 672-678 have defects that cause them not to seal completely, or if there are other defects, such as defects in the attachment points 662, the wearer of the respirator may inhale contaminated air. The systems and methods described herein can help detect these gaps along the curved surfaces of a 3D printed respirator so that the defects can be eliminated through design changes or changes to process parameters.

[0056] The need for perfect seals between layers 610, 620, and 630 presents a challenge in 3D printing of a ventilator. Using a vision system such as that described herein, the seals can be inspected as the layers are deposited or on the completed ventilator to detect gaps.

[0057] Similarly, other molded parts where there are quality control concerns could benefit from imaging done during or after production. It may be possible to detect weak or thin areas that could potentially fail.

[0058] 7A and 7B illustrate a glass welding helmet that may benefit from the systems and methods described herein. The welding helmet 700 has a glass shield 750. The shield 750 is a glass shield with a curvature 752. The technique for forming the curved glass shield 750 is sensitive to contaminants. The shield 750 has a protective film applied to its surface, and any foreign matter on the glass surface 754 may cause air bubbles to appear in the film. A topographical imaging system may be able to detect the foreign matter before the film is applied, or identify the air bubbles or imperfections. An installer may be able to take action, for example, by creating a pinhole to remove the air bubbles and allow trapped air to escape.

[0059] Although an air bubble on the shield of a welding helmet is presented as an example, it is expressly contemplated that the systems and methods described herein are also useful for detecting defects in molded or formed parts, such as cracks, scratches, or minor imperfections. The quality of an applied film can also be evaluated using the topography detection systems described herein.

[0060] 8A and 8B show a prosthetic fitting that may benefit from the systems and methods described herein. The examples described so far describe processes in which material is added or removed from a surface and then inspected. However, it is expressly envisioned that a vision system such as that described herein may be useful in other applications involving irregular surfaces.

[0061] Prosthetic limbs are uncomfortable if they do not fit well. A poorly fitting prosthesis can cause the wearer to experience abrasions, blisters, rashes, or pain. Worse yet, poorly fitting prostheses can cost thousands of dollars per patient in medical costs and can be life-threatening if new wounds develop.

[0062] The topography of the amputation site changes over time as remodeling takes place, and a previously well-fitting prosthesis may become uncomfortable over time. Currently, prosthetic molds are created using plaster casting, which is time-consuming. There is a need for a solution that can rapidly scan the limb stump, process the collected image data to obtain the 3D topography of the limb stump, and form a prosthesis that fits the detected topography. It may also be possible to better design a tight fit that distributes weight to the appropriate areas (e.g., differences in residual bone and soft tissue).

[0063] FIG. 8A shows a residual limb patient 800 being fitted for a prosthesis. The amputation process may leave the end of each limb with a unique topography 820, including features such as folds 810 where the skin is sewn together. Because of this, prosthetic limbs are often expensive and require customization to be comfortable. A mobile imaging system may capture the topography 820 and use it as the basis for an internal topography 830 of the prosthesis 850. Using a topography imaging system to obtain an accurate 3D topography of the limb 800, the features 810 are fully captured and the interior 830 of the prosthesis may be shaped to fit the patient comfortably. The prosthesis 850 may be 3D printed, for example, based on the topography 820, or molded or formed in other suitable ways.

[0064] While prosthetic fitting as shown in FIG. 8 is described as an example, other personalized medical applications of the topography sensing system described herein are expressly envisioned. For example, acquiring the topography of a patient's body may aid in custom wound care treatments or custom IV fixation mechanisms.

[0065] Another example is scanning the plantar topography to create custom orthopedic shoes or select the best fitting orthopedic shoe from available models to improve patient comfort, which is especially useful for patients with diabetic foot ulcers.

[0066] Additionally, while the system described herein is described as being useful for the manufacture of personalized medical devices, it is expressly envisioned that it will also be used to scan custom surfaces and compare the captured topography to an expected topography for rapid quality checking or verification that the correct device is being provided to the correct individual. If no match is found, the device has been mislabeled.

[0067] Figures 9A and 9B show tissue samples before and after negative pressure treatment. Negative pressure wound therapy (NPWT) is a method to draw fluid and infection from the wound to promote healing. NPWT promotes healing by removing healing inhibitors, increasing blood flow, stimulating vascularization and granulation tissue, and applying mechanical stress to the wound bed. Figures 9A-1 and 9A-2 show actual overhead (9A-1) and schematic cross-sectional (9A-2) views of tissue before NPWT, where the native tissue appears flat. In Figures 9B-1 and 9B-2, a "dome" appears in the native tissue several hours after NPWT. Figure 9B-3 shows a finite element analysis model showing the contours of the skin after NPWT.

[0068] The systems and methods described herein can be used to measure the change in topography before and after NPWT to understand how the topography has changed. With reference to FIG. 9A-2 and FIG. 9B-2, tissue 2010 has a surface path length 2012 across the surface of tissue 2010. This corresponds to the surface area of ​​the patient's skin. After NPWT, the path length 2022 of treated tissue 2020 has increased due to swelling of different areas of tissue 2020. Skin strain measurements can be determined by comparing path lengths 2012 and 2022. For example, if path length 2012 is 2 cm and path length 2022 is 2.5 cm, then swelling has caused a 40% strain.

[0069] Currently, tissue strain cannot be easily measured without performing a biopsy. Computer modeling (such as that shown in FIG. 9B-3) is often used as a surrogate. The systems and methods described herein may provide a less invasive way to more accurately understand a patient's tissue strain. This may be used to replicate a patient's treatment and help improve wound care.

[0070] FIG. 10 illustrates a high-definition surface that may be inspected using the systems and methods described herein. The surface 900 is formed with a number of structures 952, as seen in the enlarged portion 950. The structures 952 may have channels 954 or spacing between them. High-definition structures have an expected relationship between adjacent structures 952, with equal spacing 954 and geometric arrangement, as shown in the image 900. In some applications, it is particularly important that the structures 952 and / or spacing 954 are accurate and error-free. FIG. 10 illustrates a TRIZACT® polished surface where the precise placement of the structures 952 and channels 954 improves performance.

[0071] While different high definition technologies may have different tolerances than others, all may benefit from quality inspection using the imaging systems described herein to detect expected height variations (e.g., channel 952) as well as unexpected variations (trapped foreign objects, excess abrasive slurry, etc.).

[0072] Images of microreplicated surfaces are processed to obtain a binary pattern based on expected height and density, and defects may be detected that do not fall within the expected height or density range.

[0073] High Resolution (MR) applications rely on the presence of large-scale 3D features that are difficult to see but must be consistently present. Missing, damaged, or poorly formed microscopic features can degrade the intended macroscopic performance of MR materials.

[0074] The topography detection system described herein can be used to scan the undulating substrate on which the MR is applied or formed, which produces unique image patterns that can be interpreted using Fourier transform analysis and / or machine learning.

[0075] Patterns from a perfect product can be fingerprinted to provide a known good sample, or such a standard can be calculated using theoretical assumptions, after which defects and drift / deviations from the ideal perturb the image pattern.

[0076] The systems and methods described herein contemplate different sensitivity ranges depending on the reflectivity of the surface and the camera optics. For visually obvious features, it may be possible to detect topographical changes as small as 0.01 mm in size. If necessary, sensitivity can be improved by changing the camera settings relative to the depth setting.

[0077] Quality indicators can then be formed based on the degree of perturbation. Machine learning and Fourier analysis of the pattern perturbations may be used to characterize MR defects, anomalies, and damage throughout the material's lifecycle, from manufacturing to end use.

[0078] A number of different exemplary surfaces that may be inspected with the systems and methods described herein are described, however, these are by way of example only, and it is expressly envisioned that many other industries could benefit from similar surface inspection systems, such as, for example, idlers and structured rolls used in roll-to-roll processing, molded objects with or without thermoformed layers, inspection of windshields and their laminations, embossing rolls, turbine blades with protective films, etc.

[0079] FIG. 11 illustrates an adhesive dispensing operation that may benefit from the systems and methods described herein. Dispensing of adhesive depends on several variables, including temperature, dispensing speed, speed of movement of the dispenser relative to the transport, and proper mixing conditions. A lot of effort goes into designing and troubleshooting the dispenser 1020 so that adhesive 1022 is consistently dispensed onto the substrate 1010. However, as shown in dispensing operation 1000, it is possible to use an image system 1050 to follow the dispenser 1020 to ensure that the proper amount of adhesive is dispensed. While in FIG. 10 the image system 1050 is shown moving separately from the dispenser 1020, it is expressly contemplated that in some embodiments the light source and line scan array move as one unit following the dispenser at a set distance.

[0080] Some areas may have too much adhesive 1028 dispensed and other areas may not have enough adhesive 1026. In some applications, there is tolerance for variation in the adhesive dispensed. However, in other applications, gaps in the adhesive may cause a poor seal and lead to the work surface 1010 being scrapped. Therefore, having a system 1050 that can inspect the adhesive in real time during dispensing and curing allows the problem to be fixed before the adhesive cures by adding more adhesive or smoothing out excess adhesive, thereby reducing waste.

[0081] Similarly, while a flat surface 1010 is shown for convenience, it is contemplated that curved surfaces may require more or less adhesive. In FIG. 10, the imaging system 1050 is shown moving behind the adhesive dispenser, but in some embodiments, it is possible for the imaging system 1050 (or a second system) to move in front of the dispenser 1020. A forward positioned imaging system can provide accurate upcoming topography information so that the dispenser can be adjusted at the appropriate speed.

[0082] 12 illustrates an imaging system according to embodiments described herein. The imaging system 1100 is controlled by a controller 1150 and may receive instructions from an operator. For example, instructions may be received using a keyboard as shown. However, in some embodiments, the system 1100 is automatically controlled by the controller 1150 based on information received, for example, from distance / position sensors or other sources.

[0083] The line scan array 1120 images a surface 1140, which in some embodiments moves relative to the system 1100. However, it is expressly contemplated that in some embodiments the work surface may be fixed and the system 1100 may move. The light source 1110 is directed towards the surface 1140 and light is reflected towards the line scan array 1120.

[0084] An orientation adjustment component 1130, such as the illustrated curved rail, may be used to change the orientation of the system 1100 and the surface 1140 while maintaining a desired orientation between the light source 1110 and the line scan array 1120. This may be useful to maintain the perpendicular perpendicular orientation formed by the light source 1110 and the line scan array 1120 when there is curvature in the surface 1140. In the illustrated embodiment, the orientation adjustment component 1130 operates independently to change the angle of the light source 1110 and the imaging device 1120 relative to the surface 1140. This may be preferred when the optimal placement to reveal and characterize defects varies based on the optical properties of the surface, the angle of incident light, and the position of the camera.

[0085] FIG. 13 shows a driven robot unit that can be equipped with a topography mapping unit according to embodiments described herein. The driven robot arm 1200 has a number of different pivot points 1220, 1230, 1240, which allow for free movement. Depending on the design (robot 1200 is shown by way of example, not limitation), each of these pivot points may allow for movement in one, two, or three degrees of freedom. Furthermore, the robot arm 1200 is fixed at point 1210, or point 1210 shows a connection to a movement mechanism (e.g., wheels or a rail system allowing movement in one, two, or three dimensions). In the illustrated embodiment, a topography detection system 1250 is mounted at the end of the arm. However, it is expressly contemplated that in other embodiments, the system 1250 may be located elsewhere (e.g., behind or in front of the distribution system).

[0086] The robotic unit 1200 may use a topographical mapping system to traverse the work surface. As noted above, in some embodiments, a line scan array may have a narrow range for high precision mapping. As such, the robotic unit 1200 may guide the system 1250 over the surface in a grid pattern until the entire region of interest is adequately mapped.

[0087] FIG. 14 illustrates a surface imaging system according to embodiments described herein. The surface imaging system 1300 may be used to capture an image of a work surface 1390. The work surface 1390 may be a flat surface, a curved surface, an irregular surface, or a surface with features such as corners, depressions, hills and valleys. The work surface 1390 may have curvature in one or more directions. The surface inspection system 1300 may be useful for imaging the work surface 1390 before or after surface treatment. The surface inspection system 1300 illustrates several components that may be useful for inspecting the work surface 1390 using the imaging system 1310. However, it is expressly contemplated that other components 1308 may be useful in some embodiments.

[0088] The surface inspection system 1300 includes an imaging system 1310 that captures an image of a work surface 1390. The image is captured by a line scan array 1312. A lens 1314 may be used to focus a camera of the line scan array 1312. The line scan array 1312 is directed toward the work surface 1390 such that light from a light source 1316 is reflected from the work surface 1390 to the line scan array 1312. A knife edge 1318 is positioned in front of the light source 1316. The imaging system 1310 may also include other features such as a second lens 1314 and a second light source 1316. In some embodiments, the imaging system 1310 includes a movement mechanism that allows the imaging system 1310 to move relative to the work surface 1390 and maintain a perpendicular position relative to the perpendicular angle formed by the line scan array 1312, the work surface 1390, and the light source 1316. The movement mechanism 1322 can rotate the imaging system 1310, raise or lower the imaging system 1310 with respect to the work surface 1390, or otherwise adjust the relative position of the imaging system 1310 and the work surface 1390. In some embodiments, the movement mechanism 1322 can be part of or coupled to a robotic arm. The imaging system 1310 can capture images of the work surface 1390 that can be stored or processed, for example, by a surface analyzer 1350. The imaging system can also include other components 1324.

[0089] In some embodiments, images captured by the imaging system 1310 are communicated to other devices using the image communication device 1302. For example, the captured images may be sent to a storage component or provided on a display for review.

[0090] In some embodiments, the surface inspection system 1300 includes a distance sensor 1304. The sensor 1304 may be a distance sensor array in some embodiments. The distance sensor array 1304 is coupled to the imaging system 1310 and in some embodiments moves with the imaging system 1310. The imaging system may move in front of the imaging system 1310, move with the imaging system 1310, or move behind the imaging system 1310. In other embodiments, the distance sensor array 1104 moves independently of the imaging system 1310. The distance sensor array 1304 passes over the work surface 1390 using, for example, a movement mechanism 1306, which may be coupled to a movement mechanism 1322 or may be separate. The distance sensor array 1304 captures detailed topographical information of the work surface 1390 and uses that information to enable the imaging system 1310 to move over the work surface 1390 and take very accurate images from desired directions.

[0091] The distance information captured from the distance sensor array 1304 is provided to a path planner 1330, which calculates a path for the imaging system 1310 to travel across the work surface 1390. A topography receiver 1332 receives the distance information and provides the information to the path planner 1330. Based on the topography of the work surface, a path generator 1340 generates a path for the imaging system 1310 to travel. The path includes a position 1342 of the imaging system 1310 relative to the work surface 1390 and an angle 1344 required for the imaging system 1310 to maintain a perpendicular position relative to the work surface 1390. The position 1342 refers to the spatial position required to maintain a desired distance between the imaging system 1310 and the work surface 1390.

[0092] In some embodiments, the movement mechanism 1306 is a robotic arm and the imaging system 1310 is attached to an end effector of the robot. In some embodiments, the sensor array 1304 includes three or more distance sensors. Suitable sensors include, for example, the LM series high precision measurement sensors from Banner Engineering and the CL-3000 series confocal displacement sensors from Keyence.

[0093] The sensors of the sensor array 1304 are positioned along the field of view of the camera and provide a sparse 3D distance map.

[0094] Many surfaces are curved in at least two directions. Therefore, using a standard line array, only the central portion of the image may be valid. With appropriate image processing, the valid area can be identified, for example using information from the image itself or 3D surface mapping. However, it is explicitly envisioned that in some embodiments a 3D camera scanning system will be used to completely map the entire surface. Such systems are available from companies such as Cognex, Keyence, and LMI.

[0095] From the 3D map, a path for the robotic unit with the imaging system can be planned. The path is calculated so that at each point the imaging system is perpendicular to the surface. The robotic arm can precisely control the angle and distance to ensure high quality imaging.

[0096] In some embodiments, the path planner 1330 is configured to allow the imaging system 1310 and the distance sensor array 1304 to make a single pass over the work surface 1390. For example, the topography receiver 1332 receives feedback from the distance sensor array 1304 in near real-time, and the path generator 1340 generates a path and instructs the motion mechanism 1322 to change the position 1342, angle 1344, or velocity 1346 of the imaging system 1310 along the path. The distance sensor feedback is provided and a path is generated and communicated to the motion mechanism 1322 to adjust the imaging system 1310 in the time it takes to move the distance between the distance sensor array 1304. For example, if the distance sensor array 1304 is coupled to the imaging system 1310 with a separation of 3 inches, the information is communicated and the path is returned and the imaging system is adjusted in the time it takes for the imaging system to move 3 inches.

[0097] In another embodiment, a two pass system is used, where in the first pass the range sensor array 1304 acquires topographical information which is provided to the path planner 1330, which generates a path and communicates it to the moving mechanism 1322, and during the second pass the position 1342 and angle 1344 of the imaging system 1310 are implemented.

[0098] The path planner 1330 may also include other functions 1336 .

[0099] The images captured by the imaging system 1310 are provided to a surface analyzer 1350, which in some embodiments performs an analysis on the surface parameters of the work surface 1390, for example to determine if the surface has unexpected height variations (e.g., too much or too little surface treatment, etc.). The surface analyzer 1350 also provides relative height information so that the topography of the surface can be mapped. The images are received by an image receiver 1352. In some embodiments, image information may be received in near real-time from the line scan array 1312, and the image receiver 1352 may assemble an image from the received array signals. Once images of the work surface 1390 are collected, they may be displayed by a human operator or automatically analyzed for quality control concerns.

[0100] The defect detection device 1356 may identify defects in the work surface 1390 based on the images from the image receiving device 1352. A defect may be too much material present (e.g., trapped foreign objects, too much adhesive dispensed, etc.) or too little material (e.g., scratches, cracks, too little adhesive dispensed, etc.).

[0101] The defect identifier 1358 may identify the detected defects as specific types of defects, for example scratches versus cracks, based on information from the defect detector 1356 and the surface analyzer 1350.

[0102] The defect evaluator 1364 may evaluate the severity of the detected defect and whether it can be mitigated, e.g., repaired by de-airing air bubbles, adding additional adhesive to fill gaps, etc.

[0103] Based on the results of the analysis unit 1350 and the defect assessment, the surface characterization unit 1354 may output an indication of the quality of the processed working surface 1390. For example, the surface characterization unit 1354 may determine that the surface 1390 is unacceptable but repairable.

[0104] The defect correction acquisition device 1362 may acquire a defect mitigation procedure if it is determined that the defect can be mitigated. For example, the location of the detected air bubble may be output and provided to a bleeding system that punctures the air bubble based on the detected location. Or, the location of the over-dispensed adhesive may be provided to a wiping unit that smooths or removes the excess adhesive.

[0105] The surface analysis device may include other features or functionality 1366 .

[0106] Information from the analyzer 1350 may be provided to the controller 1360 to adjust one or more parameters to reduce future defects for the next operation. For example, it may be determined that multiple gaps in the dispensed adhesive indicate a higher than expected viscosity and that the metering rate should be increased to compensate.

[0107] The controller 1360 may provide control signals to components of the surface inspection system 1300, such as instructing the motion mechanism 1322 to adjust the position or angle of the imaging system 1310, instructing the imaging system 1310 to begin capturing images, and instructing a distance sensor to capture topographical information.

[0108] The systems and methods are described herein for a scenario in which the work surface is stationary during topography and imaging collection. However, it is expressly contemplated that the systems and methods described herein are also applicable when the work surface 1390 is moving. The imaging system 1310 is also moving, either in the same direction as the work surface 1390 or in a different direction, or the imaging system 1310 may be stationary. Thus, when the work surface 1390 is moving, the work surface 1390 may include a movement mechanism 1394, such as a conveyor belt or wheels, and may also include a stabilizer 1392 to stabilize the work surface 1390 during imaging.

[0109] In some embodiments, custom imaging lenses with telecentric imaging are used to improve operation due to their compact design.

[0110] In some embodiments, the light source is a diffuse LED light with a knife edge. In other embodiments, the light source includes a small LCD display with individually addressable pixels, which may allow sensitivity to be changed without mechanical adjustment. In some embodiments, the knife edge includes an automatic height adjustment mechanism.

[0111] 15 illustrates a method for assessing a work surface according to embodiments described herein. Method 1400 may be used with any of the systems described herein or other suitable systems for capturing and analyzing images of a work surface. The target work surface may be a flat surface, a curved surface, or a surface that includes features such as hills, valleys, corners, depressions, etc.

[0112] At block 1410, a topography of the work surface is acquired. In some embodiments, this includes acquiring a 3D model, e.g., a CAD model 1402. However, the CAD model 1402 may not be completely accurate with respect to the topography of the surface, such as when paint is not uniform or bumps occur due to trapped foreign objects. Also, the vehicle may not be perfectly aligned or positioned in space. Thus, to obtain a high quality image of the surface, in some embodiments, a sensor array 1404 must be used to obtain an accurate topography of the surface, especially since many surfaces have curvature in multiple directions and have corners, recesses, raised features, etc. For example, many vehicles have surfaces that are curved in at least two directions (so-called "complex curves"), and limb stumps may have asymmetric curvatures due to healing. Also, adhesive may not be distributed perfectly smoothly. In some embodiments, the method 1400 proceeds in a multiple pass process 1414, since the topography must be acquired before imaging can occur. However, it may be possible to obtain accurate topography simultaneously 1412 while imaging is taking place, or using an imaging system following the sensor array 1404. Other embodiments 1416 are also contemplated.

[0113] In other embodiments, the exact topography can be obtained, for example using an imaging system that also detects the topography on the surface. Any other suitable system may be used to obtain the topography of the surface.

[0114] At block 1420, an image is captured. In some embodiments, a line scan array 1422 is used to capture the image at a known distance from the surface. In some embodiments, a 3D camera 1424 is used. For curved surfaces, the imaging device needs to be at a known distance from the surface at all times during the scan. Thus, in some embodiments, capturing the image at block 1420 involves moving the imaging device along a path such that a set distance and / or orientation is maintained relative to the surface being imaged. Other suitable imaging devices may also be used.

[0115] In block 1430, the captured images are processed to obtain information about the surface. For example, deviations from expected topography are detected, and height changes that are greater or smaller than expected are detected. The detected deviations are classified, for example, too much or too little material is present. Images of the detected deviations are presented for inspection by a human operator. Other processing may occur to generate other useful views or to address detected defects.

[0116] At block 1440, the suitability of the surface is evaluated, such as by providing the image to a human operator who determines whether the surface is satisfactory, whether it can be repaired, whether parameters for future operations should be adjusted, or other appropriate action may be taken.

[0117] Alternatively, the repair may be quantitatively assessed by image analysis equipment. Identified defects are assessed for severity to determine whether they can be mitigated, for example repaired by releasing air from bubbles. Defects are assessed automatically, either defect by defect, for example where one large defect on an entire surface can be as problematic as ten small defects, or on an entire surface. When material is removed, for example air is released from bubbles, the remaining defects are inspected. Other properties may also be quantified.

[0118] In some embodiments, at least some steps of method 1400 are completed automatically. For example, in a two pass system, the second imaging pass may be automatically initiated once the topography is acquired and a pass is planned for the imaging system. Also, processing of the image may be performed as soon as the image data is received from the line scan array system or in-situ. Once the image is processed by the imaging system, the working surface is evaluated. Instructions to the components to perform each step or analysis shown in FIG. 11 may be provided by the robot controller. The instructions include movement instructions (direction, speed, orientation, etc.) of different components.

[0119] Method 1400 may need to be performed multiple times during a surface preparation operation. For example, a typical defect repair process for a vehicle may include (1) defect location and pre-inspection, (2) sanding, (3) wiping, (4) polishing, (5) wiping, and (6) final inspection. Imaging may be required in steps (1) and (6), and based on the imaging in (1), a polishing recipe may be selected to address the particular defect. Similarly, in the air bubble example of FIG. 1, instead of steps 2-5y, an intermediate step may be to drill out the air bubble and smooth the area.

[0120] After intermediate surface preparation operations, the surface area may be imaged again to detect, characterize and quantify defect residues and determine whether additional repairs are required.

[0121] FIG. 16 illustrates an architecture of a surface inspection system. Surface processing system architecture 1500 illustrates one example of an implementation of a surface inspection system 1510. For example, the surface inspection system 1500 can provide computation, software, data access, and storage services without the end user having to know the physical location or configuration of the system providing the services. In various embodiments, a remote server can provide the services using an appropriate protocol over a wide area network such as the Internet. For example, a remote server can provide applications over a wide area network that are accessed through a web browser or other computing component. The software or components and corresponding data illustrated in FIGS. 1 through 15 can be stored on a remote server. Computing resources in a remote server environment may be consolidated or distributed at a remote data center location. The remote server infrastructure may appear to the user as a single access point, but provide services through a shared data center. Thus, the components and functions described herein may be provided from a remote location using a remote server architecture. Alternatively, they may be provided by a traditional server and installed directly on a client device or provided in another manner.

[0122] In the example shown in Figure 16, some items are similar to those shown in previous figures. Figure 16 specifically illustrates that surface inspection systems 1510 can be located at a remote server location 1502. As such, computing devices 1520 access these systems through the remote server location 1502. An operator 1550 can also access a user interface 1522 using the computing devices 1520.

[0123] FIG. 16 illustrates that some elements of the system described herein are located at a remote server location 1502, while other elements are contemplated not to be located there. For example, storage 1530, 1540, or 1560, robotic system 1570 may be located at a different location than location 1502 and accessed through a remote server. Regardless of where they are located, they may be accessed directly from computing device 1520, accessed over a network (either a wide area network or a local network) using system 510, provided as a service hosted at a remote site, or accessed by a connected service. Data can be stored in almost any location and accessed intermittently or transferred to interested parties. For example, a physical carrier, or in addition to an electromagnetic carrier, may be used.

[0124] It should also be noted that elements of the systems described herein, or portions thereof, may be located in a variety of devices, including servers, desktop computers, laptop computers, embedded computers, industrial controllers, tablet computers, or other mobile devices, such as palmtop computers, mobile phones, smartphones, multimedia players, and personal digital assistants.

[0125] The systems and methods can help gather data for the controller to learn to better detect, classify, and respond to detected topography. A non-exhaustive list of machine learning techniques that may be used on the data obtained here includes support vector machines (SVM), logistic regression, Gaussian processes, decision trees, random forests, bagging, neural networks, deep neural networks (DNN), linear discrimination, Bayesian models, k-nearest neighbors (k-NN), and gradient boosting algorithms (GBA). During operation, the learned behavioral model can better characterize the topography and determine appropriate next steps based on that characterization.

[0126] As described herein, in different embodiments, the system can adjust sensitivity based on camera specifications, aperture settings, lens stack specifications, and illumination. Depth of field is adjusted based on the effective diameter and focal length of the lens. Depth of field refers to object space, and depth of focus refers to image space. The field of view is the point where parallel rays converge after passing through the lens.

[0127] As described herein, the system maintains a constant distance from the surface during imaging / topography mapping, although in some embodiments there is some tolerance for changes in distance due to mechanical instability, vibration, imprecision, speed of movement, etc. The amount of movement allowed may vary based on the desired sensitivity of the resulting topography measurements.

[0128] Additionally, while the systems and methods are described as measuring topography to detect surface irregularities, it is envisioned that the systems and methods can also detect and measure features that change the physical appearance of a surface without changing the topography, such as surface gloss, glass transparency, surface haze, etc.

[0129] The systems and methods described herein explicitly contemplate line-scan imaging systems, although it is expressly contemplated that other imaging devices may be suitable, such as area-scan arrays and 3D imaging cameras.

[0130] 17-19 show examples of computing devices that can be used in the embodiments shown in the previous figures.

[0131] Figure 17 is a simplified block diagram of one example of a handheld or mobile computing device that may be used as a user or client handheld device 1616 (e.g., as computing device 1520 of Figure 16). For example, a mobile device may be deployed in an operator compartment of computing device 1520 for use in generating, processing, or displaying data. Figure 18 is another example of a handheld or mobile device.

[0132] 17 provides a general block diagram of components of device 1616, some of which may be executed or interacted with. A communication link 1613 is provided in device 1616 that allows the handheld device to communicate with other computing devices and, in certain embodiments, provides a channel for receiving information automatically. Examples of communication link 1613 include wireless services used to provide cellular access to networks, as well as protocols that provide local wireless connections to networks.

[0133] In another example, the application is received on a removable Secure Digital (SD) card and connected to the interface 1615. The interface 1615 and communication links 1613 are in communication along a bus 1619 with a processor 1617 (which may also embody a processor) and are connected to memory 1621 and input / output (I / O) components 1623, a clock 1625 and a position system 1627.

[0134] In one embodiment, I / O components 1623 are provided to facilitate input and output operations and the device 1616 may include input components such as buttons, touch sensors, light sensors, microphones, touch screens, proximity sensors, accelerometers, orientation sensors, etc. Output components may include a display device, speakers, or a printer port, etc. Other I / O components 1623 may also be used.

[0135] The clock 1625 may embody a real-time clock component that outputs the date and time and provides timing functions for the processor 1617.

[0136] As shown, the location system 1627 includes a component that outputs the current geographic location of the device 1616. This may include, for example, a Global Positioning System (GPS) receiver, a LORAN system, a dead reckoning system, a cellular triangulation system, or other positioning systems. It may also include mapping or navigation software that generates maps, navigation routes, and other geographic features.

[0137] The memory 1621 stores an operating system 1629, network settings 1631, applications 1633, application configuration settings 1635, data store 1637, communication drivers 1639, and communication configuration settings 1641. The memory 1621 includes all types of volatile and non-volatile computer readable memory devices. It may also include computer storage media (described below). The memory 1621 stores computer readable instructions that, when executed by the processor 1617, cause the computer to perform steps or functions according to the instructions. The processor 1617 may be activated by other components to facilitate their functionality.

[0138] 18 shows that the device is a smartphone 1701. The smartphone 1771 has a touch-sensitive display 1773 that displays icons or tiles or other user input mechanisms 1775. The mechanisms 1775 may be used by a user to run applications, make calls, and perform data transfer operations. Typically, the smartphone 1771 is built on a mobile operating system and offers more advanced computing capabilities and connectivity than a traditional mobile phone.

[0139] It should be noted that other types of device 1716 are contemplated.

[0140] FIG. 19 is a block diagram of a computing environment that can be used in the embodiments illustrated in the previous figures.

[0141] 19 is an example of a computing environment in which elements of the systems and methods described herein, or portions thereof (for example), may be deployed. With reference to FIG. 19, an example of a system for implementing some embodiments includes a general-purpose computing device in the form of a computer 1810. Components of the computer 1810 include a processing unit 1820 (which may embody a processor), a system memory 1830, and a system bus 1821, which couples various system components including the system memory to the processing unit 1820. The system bus 1821 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus, using a variety of bus architectures.

[0142] Computer 1810 typically includes a variety of computer readable media. Computer readable media is any media that can be accessed by computer 1810, including both volatile and non-volatile media and removable and non-removable media. Exemplary computer readable media include computer storage media and communication media. Computer storage media includes any other medium that can be used to store information and that can be accessed by computer 1810, other than a modulated data signal or carrier wave. Computer storage media may include volatile and non-volatile memory, CD-ROMs, DVDs, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computer 1810. Communication media may include any information-providing media that embodies computer-readable instructions, data structures, program modules or other data in a transport mechanism. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a way as to encode information in the signal.

[0143] The system memory 1830 includes computer storage media in the form of volatile and / or nonvolatile memory such as read only memory (ROM) 1831 and random access memory (RAM) 1832. A basic input / output system (BIOS) 1833, which typically contains the basic routines that help to transfer information between elements within the computer 1810, is stored in the ROM 1831. The RAM 1832 typically contains data and / or program modules that are immediately accessible to or presently being operated on by the processing unit 1820. By way of example, FIG. 19 illustrates operating system 1834, application programs 1835, other program modules 1836, and program data 1837.

[0144] The computer 1810 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example, FIG. 19 shows a hard disk drive 1841, a non-volatile magnetic disk 1852, an optical disk drive 1855, and a non-volatile optical disk 1856, which read from or write to non-removable, non-volatile magnetic media. The hard disk drive 1841 is typically connected to the system bus 1821 via a non-removable memory interface, such as interface 1840, and the optical disk drive 1855 is connected to the system bus 1821 via a removable memory interface, such as interface 1850.

[0145] Alternatively, or in addition, the functions described herein may be performed, at least in part, by one or more hardware logic components, including, by way of example and without limitation, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOC) systems, complex programmable logic devices (CPLDs), and the like.

[0146] The drives and their associated computer storage media, discussed above and illustrated in Figure 19, are provided for storing computer-readable instructions, data structures, program modules, and other data for the computer 1810. In Figure 19, for example, hard disk drive 1841 is illustrated as storing operating system 1844, application programs 1845, other program modules 1846, and program data 1847. These components can either be the same as or different from operating system 1834, application programs 1835, other program modules 1836, and program data 1837.

[0147] A user may enter commands and information into the computer 1810 through input devices such as a keyboard 1862, a microphone 1863, and a pointing device 1861 (e.g., a mouse, trackball, or touch pad). Other input devices (not shown) may include a joystick, game pad, satellite receiver, scanner, or the like. These and other input devices are typically connected to the processing unit 1820 through a user interface 1860, but may be connected by other interface or bus structures. A visual display 1891 or other type of display device is also connected to the system bus 1821, such as via a video interface 1890. Computers may also include output peripheral devices, such as speakers 1897 and printer 1896, which may be connected through an output peripheral interface 1895.

[0148] The computer 1810 operates in a network environment with one or more remote computers, such as a remote computer 1880, using logical connections, such as a local area network (LAN) or a wide area network (WAN).

[0149] When used in a LAN networking environment, the computer 1810 is connected to a network interface or adapter 1870 via the LAN 1271. When used in a WAN networking environment, the computer 1810 typically includes a modem 1872 or the like for establishing communications over the WAN 1873 (eg, the Internet). In a networked environment, program modules may be stored in the remote memory storage device. FIG. 18 shows that remote application programs 1885 reside on the remote computer 1880.

[0150] A method for evaluating a surface is presented that includes imaging the surface with an imaging system. Imaging includes bringing a camera of the imaging system into proximity with the surface. Imaging also includes moving the imaging system and the surface relative to one another such that a distance between the imaging system and the surface is substantially maintained. It also includes capturing image data. The image data is captured in a near-dark field mode or a dark field image mode. The method also includes analyzing the image data and detecting a topography and / or appearance of the surface. The method also includes generating an assessment for the surface based on the detected topography and / or surface appearance.

[0151] The method may be implemented such that the camera includes a line scan array or an area scan array.

[0152] The method may be implemented such that the imaging system includes a light source, and for a near-dark field mode, the light source and the line scan array are arranged in a first configuration relative to the surface, and for a dark field mode, the light source and the line scan array are arranged in a second configuration.

[0153] The method may be implemented to include again moving the imaging system and the surface relative to one another and capturing second image data of the surface, the second image data being captured in a near-dark field mode or a dark field image mode and may be captured in a different mode than the image data.

[0154] The method may be implemented such that the second imaging system is located on a powered robotic arm.

[0155] The method may be implemented such that the second imaging system is onboard an unmanned aerial vehicle (UAV).

[0156] The method may include displaying the captured image data or the detected topography on a display component.

[0157] The method may include comparing the detected topography to an expected topography.

[0158] The method may be implemented such that the comparison includes detecting a change in height of the topography.

[0159] This method may be implemented such that the change in height exceeds a threshold of acceptable deviation.

[0160] The method may be implemented such that the predicted topography is based on a model of the surface.

[0161] The method may be implemented such that the predicted topography is based on a previously detected topography.

[0162] The method may be implemented such that the surface includes curvature.

[0163] The method may be implemented such that the surface includes edges.

[0164] The method may be implemented such that a surface contains the dispensed material.

[0165] The method may be implemented such that the imaging system includes a distance sensor that moves in front of the camera and detects the distance between the distance sensor and the surface.

[0166] The method may be implemented such that the imaging system further includes a controller that receives the detected distance and adjusts a position of the imaging system such that the imaging system maintains the separation distance from the surface.

[0167] The method may be implemented such that the controller stores the detected distances and the analysis involves analyzing the detected distances over time and reconstructing the detected topography.

[0168] A method of dispensing adhesive is presented that includes dispensing the adhesive onto a surface. The adhesive is dispensed at a rate and temperature. The method also includes imaging the surface with an imaging system. Imaging includes bringing a camera of the imaging system proximate to the surface, moving the imaging system and the surface relative to one another such that a distance between the imaging system and the surface is substantially maintained, and capturing image data of the surface. The image data is captured in a near-dark field mode or a dark field image mode. Imaging also includes analyzing the image data and detecting a topography of the surface. The method also includes generating an assessment related to the surface based on the detected topography.

[0169] The method may be implemented such that the imaging step occurs prior to the dispensing step.

[0170] The method may be implemented to adjust the speed based on the detected topography.

[0171] The method may be implemented such that the imaging step occurs after the dispensing step.

[0172] In this manner, the topography may be indicative of the dispensed adhesive.

[0173] The method may be implemented to include generating a quality indication for the dispensed adhesive, which may include an indication that there are gaps in the dispensed adhesive or that too much adhesive has been dispensed.

[0174] The method may be implemented to include adjusting the rate based on the quality indication.

[0175] A surface evaluation system is presented that includes an image capture system for capturing an image of a surface. The image capture system includes a light source, an image capture device configured to capture a near-dark field or dark field image of the surface, and a movement mechanism for moving the image capture device relative to the curved surface. The movement mechanism maintains a distance between the image capture system and the surface substantially fixed while the image capture device moves relative to the surface. The system also includes a surface evaluation device for generating a surface quality indication based on the captured image. The system also includes a process parameter adjustment device for adjusting a process parameter based on the surface quality indication.

[0176] The system may be implemented to include a view generating device that generates a view of the surface based on an image captured by an image capture device, and a display component that displays the view.

[0177] The system may include the detected dents in the surface quality indication.

[0178] The system may be implemented to include a dent assessment device that provides an indication of the local location of the dent and the severity of the dent.

[0179] The system may include detected flaws in the surface quality indication.

[0180] The system may be implemented to include a wound assessment device that provides an indication of the local location of the wound and the severity of the wound.

[0181] The system may have a surface quality representation indicating an air bubble and the process parameter may be a location of the air bubble. The system also includes a repair command generator that generates repair commands that include the location of the air bubble.

[0182] The system includes a surface quality indicator that indicates the amount of material added or removed from the surface, and may be the location of excess or deficient material due to process parameters.

[0183] The system may be implemented to include a storage component configured to store the surface quality representation.

[0184] The system may include a path generating device that receives a surface representation, where the surface representation is a representation of a curved surface, and based on the surface representation, the path generating device generates a path for a movement mechanism to maintain relative positions of the image capture device, the light source, and the curved surface with respect to one another.

[0185] The system may be implemented such that the image capture device, the surface, and the light source form a right angle at a point on the surface being imaged.

[0186] The system may include a topography in the surface representation that is generated based on sensor information from a range sensor array.

[0187] In this system, a distance sensor array may be coupled to a moving mechanism, which moves in front of the image capture device relative to the curved surface, and a path generating device may generate a path on the fly and provide the path to the moving mechanism.

[0188] The system may have an image capture device that is a line scan array.

[0189] The system may include a 3D camera in which the image capture device is a 3D camera.

[0190] The system may include a lens between the image capture device and the light source.

[0191] The system may include a knife edge between the image capture device and the light source.

[0192] The system may be such that the surface is curved and maintaining the distance involves adjusting the position of the imaging system along the curvature of the surface.

[0193] A robotic surface inspection system is presented that includes an imaging system configured to capture a representation of a surface. The imaging system includes a light source, a knife edge disposed in front of the light source, and an image capture device disposed such that light from the light source passes in front of the knife edge and reflects off the surface to the image capture device. The positions of the light source and the image capture device are fixed relative to one another during the imaging operation. The system also includes a motion mechanism that moves the imaging system relative to the surface such that a fixed distance and orientation is maintained between the surface and the imaging system during the imaging operation. The system also includes a surface topography system that includes a distance sensor array that moves relative to the surface, a topography generator that generates a topography based on sensor signals from the distance sensor array, and a controller that generates motion commands to a powered robotic arm to maintain a relative position between the imaging system and the surface. The controller generates motion commands based on the generated topography.

[0194] In this system, the surface may be stationary and the imaging system may move relative to the surface.

[0195] In this system, the imaging system may be stationary and the surface may move relative to the imaging system.

[0196] The system may include an orientation forming a right angle between the image capture device, the surface, and the light source.

[0197] The system may be implemented such that in a first movement sequence, the range sensor array captures topographical information, and in a second movement sequence, the imaging system captures image information.

[0198] In this system, both the surface topography system and the imaging system may be active during a movement sequence, the topography generating device generates a topography in situ, and the controller generates movement commands in situ based on the topography information received from the topography generating device in substantially real time.

[0199] The system may also include a surface assessment device that provides an image-based indication of surface quality.

[0200] The system may be implemented to include a display component that displays the image.

[0201] The system may be implemented to include a storage component for storing the images.

[0202] The system may have a curved surface.

[0203] This system may have surfaces with curvature in two directions.

[0204] A method for generating a surface topography is presented that includes moving a line scan array imaging system relative to a surface. A distance between the line scan array imaging system and the surface is maintained. The method also includes imaging the surface using the line scan array imaging system to generate an image of the surface. The imaging system moves along an imaging path relative to the surface. The imaging path maintains a substantially constant distance between the line scan array imaging system and the surface. The method also includes detecting changes in distance along the imaging path using a distance sensor associated with the line scan array imaging system. The method further includes processing the changes in distance to generate a surface topography.

[0205] The method may further include generating position adjustment commands, where the surface is curved, to adjust a position or orientation of the line scan array imaging system to maintain a constant distance based on the change in the detected distance.

[0206] The method may include coupling a line scan array imaging system to a range sensor.

[0207] In the method, the position adjustment command may be generated by the controller. The line-scan array imaging system may be attached to a motion mechanism, and the motion mechanism may perform the position adjustment command.

[0208] The method may include adjusting an orientation of the line-scan array imaging system such that the motion mechanism maintains the orientation of the line-scan array imaging system relative to the curved surface.

[0209] The method may include having a moving mechanism adjust the height of the line-scan array imaging system.

[0210] In this method, imaging may be the first imaging and the system may image the surface a second time.

[0211] The method may include performing a surface treatment operation between the first and second imaging.

[0212] In this method, the imaging path may be the first imaging path and the second imaging may follow the second imaging path. The second imaging path may be different from the first imaging path.

[0213] In this method, the image may be a first image and a second image may be produced by imaging a second time. Areas of the surface may appear in both the first image and the second image.

[0214] In this method, the image may be a first image and the second imaging may produce a second image. The first image may include the first surface region and the second image may include the second surface region. The first and second surface regions may not overlap.

[0215] The method may involve the imaging system being in a dark field configuration.

[0216] In this method, the imaging system may be in a near dark field configuration.

[0217] The method may include communicating the image or the processed image to a display component, where the image or the processed image is displayed.

[0218] The method may include communicating the image or the processed image to a storage component, where the image or the processed image is stored in a retrievable format.

[0219] The method may include having the imaging system mounted on a robotic arm, the imaging system being moved along an imaging path by the robotic arm.

[0220] In this method, the imaging system may be mounted on the UAV.

[0221] In this method, the imaging path may be generated by the controller based on the acquired topography of the curved surface.

[0222] In this method, the topography may be provided to the controller from the topography detected by the range sensor array as it moves across the curved surface.

[0223] In this method, a range sensor array may be attached to the robot arm, the range sensor array may move in front of the imaging system, and a controller may generate an imaging path on the fly based on a sensor signal from the range sensor array.

[0224] The method may include changing a relative position of the imaging system with respect to the curved surface as the imaging path is performed by the robotic arm.

[0225] The method may include changing a relative orientation of the imaging system with respect to the robot arm as the imaging path is performed by the robot arm.

[0226] The method may include changing a relative orientation of the imaging system with respect to the robotic arm, comprising maintaining a relative orientation of the imaging system with respect to the curved surface while performing the imaging pass.

[0227] The method may include having a range sensor array attached to the second robotic arm.

[0228] In this method, a range sensor array may be attached to a robot arm, and during a first pass over a curved surface the range sensor array may detect the topography, and during a second pass an imaging system may image the curved surface. EXAMPLES

[0229] Example 1: Quantifying defect repair Figure 20A shows images and quantification of the four defect areas taken after repair. As shown in the figure, all three defects (A, B, and D) have visible defects and height characterization has been done based on the captured images. Defect C has been sufficiently removed to be invisible. As shown here, the size of the defect can be measured in the x and y directions, but the height of the defect (z direction) can only be assessed qualitatively. This is useful in post-inspection processes where the system only needs to determine whether the repair is successful or not. Also, as shown in Figure 20B, the light intensity profile (a') obtained from the defect is proportional to the height of the defect (a). In Figure 20B, all the defects shown in Figure 20A are characterized using a 3D non-contact laser profilometer to measure the height of the defect. This shows that the line scan array method can be used to estimate the height of the defect if the correlation between a and a' is known (e.g., by a calibration procedure).

[0230] FIG. 20B shows further characterization of the defect.

Claims

1. 1. A method for evaluating a surface, the method comprising: imaging the surface with an imaging system, the imaging comprising: positioning a camera of the imaging system proximate to the surface; moving the imaging system and the surface relative to one another such that a distance between the imaging system and the surface is substantially maintained; acquiring image data of the surface, the image data being acquired in a near-dark field mode or a dark field image mode; imaging, analyzing the image data to detect a topography and / or appearance of the surface; generating an assessment for the surface based on the detected topography and / or surface appearance; A method comprising:

2. The method of claim 1 , wherein the camera comprises a line scan array or an area scan array.

3. 3. The method of claim 2, wherein the imaging system includes a light source, the near-dark field mode includes the light source and the line scan array in a first configuration relative to the surface, and the dark field image mode includes the light source and the line scan array in a second configuration.

4. again moving the imaging system and the surface relative to one another; acquiring second image data of the surface; and Further comprising:

4. The method of claim 1, wherein the second image data is acquired in the near-dark field mode or the dark field image mode, such that the second image data is acquired in a different mode than the image data.

5. The method of any one of claims 1 to 4, further comprising comparing the detected topography with an expected topography.

6. The method of claim 5 , wherein the comparing comprises detecting a change in height of the topography.

7. The imaging system comprises: A method according to any preceding claim, comprising a distance sensor moving in front of the camera to detect a distance between the distance sensor and the surface.

8. The imaging system comprises: a controller that receives the detected distance and adjusts a position of the imaging system such that the imaging system maintains a separation distance from the surface; The method of claim 7 , wherein the controller stores the detected distances and the analyzing comprises analyzing the detected distances over time and reconstructing a detected topography.

9. 1. A method of dispensing adhesive, the method comprising: dispensing an adhesive onto a surface, the adhesive being dispensed at a constant rate and temperature; imaging the surface with an imaging system, the imaging comprising: positioning a camera of the imaging system proximate to the surface; moving the imaging system and the surface relative to one another such that a distance between the imaging system and the surface is substantially maintained; acquiring image data of the surface, the image data being acquired in a near-dark field mode or a dark field image mode; imaging, analyzing the image data to detect a topography of the surface; generating an assessment for the surface based on the detected topography; Includes, a method.

10. The method of claim 9 , wherein the topography is indicative of the dispensed adhesive.

11. 11. The method of claim 10, further comprising generating a quality indicator of the dispensed adhesive, the quality indicator comprising an indication of a gap in the dispensed adhesive or an excess of the dispensed adhesive.

12. The method of claim 11 , further comprising adjusting the rate based on the quality indicator.

13. 1. A surface evaluation system comprising:

1. An image capture system for acquiring an image of a surface, the image capture system comprising: A light source; an image capture device configured to capture a near-dark field or dark field image of the surface; a translation mechanism configured to translate the image capture device relative to a curved surface while maintaining a substantially fixed distance between the image capture system and the surface as the image capture device translates relative to the surface; an image capture system including: a surface assessment device that receives the captured images and generates a surface quality indicator based on the captured images; a process parameter adjusting device for adjusting a process parameter based on the surface quality index; Including, the system.

14. The system of claim 13 , wherein the surface quality indicator is indicative of an air bubble, the process parameter is a location of the air bubble, and the system comprises a repair command generator that generates a repair command that includes the location of the air bubble.

15. The system of claim 13 , wherein the surface quality indicator is indicative of an amount of material added to or removed from the surface, and the process parameter is a location of excess or deficiency of the material.

16. The system of any one of claims 13 to 15, further comprising a path generation device that receives the surface indication, the surface indication being an indication of a curved surface, and based on the surface indication, the path generation device generates a path for the image capture device, the light source, and a moving mechanism that maintains the relative positions of the curved surface.

17. The system of claim 16 , wherein the image capture device, the surface, and the light source form a right angle at a point on the surface being imaged.

18. The system of any one of claims 13 to 17, wherein the image capture device is a line scan array or a 3D camera.

19. The system of any one of claims 13 to 18, further comprising a lens between the image capture device and the light source.

20. The system of any one of claims 13 to 19, further comprising a knife edge between the image capture device and the light source.

21. The system of any one of claims 13 to 20, wherein the surface is curved and maintaining the distance comprises adjusting a position of the imaging system according to a curvature of the curved surface.

22. 1. A robotic surface inspection system comprising: an imaging system configured to acquire surface indicia, the imaging system comprising: A light source; a knife edge disposed in front of the light source; an image capture device arranged such that light from the light source passes in front of the knife edge and reflects off the surface to reach the image capture device, the light source and the image capture device being fixed in position relative to each other during an imaging operation; and a translation mechanism for moving the imaging system relative to the surface so as to maintain a fixed distance and orientation between the surface and the imaging system during the imaging operation; an imaging system comprising:

1. A surface topography system comprising: a range sensor array that moves relative to the surface; a topography generating device that generates a topography based on a sensor signal from the distance sensor array; a surface topography system comprising: a controller that generates movement commands to a drive robot arm such that the imaging system maintains a relative position with respect to the surface being imaged as the imaging system and the surface move relative to one another, the controller generating the movement commands based on the generated topography; Including, the system.

23. The system of claim 22 , wherein the orientation includes the image capture device, the surface, and the light source forming a right angle.

24. 24. The system of claim 22 or 23, wherein the surface topography system and the imaging system are both active during a movement sequence, the topography generation device generates the topography in-situ, and the controller generates the movement commands in-situ based on topography information received from the topography generation device in substantially real time.

25. The system of any one of claims 22 to 24, wherein the surface is curved.

26. 1. A method for generating a surface topography, the method comprising the steps of: moving a line scan array imaging system relative to a surface, wherein a distance between the line scan array imaging system and the surface is maintained; imaging the surface using the line scan array imaging system to generate an image of the surface, wherein the imaging system moves along an imaging path relative to the surface, the imaging path maintaining a substantially constant distance between the line scan array imaging system and the surface; detecting distance changes along the imaging path with a range sensor associated with the line scan array imaging system; processing the distance variations to generate the surface topography; A method comprising:

27. 27. The method of claim 26, wherein the surface is curved, and further comprising generating position adjustment commands based on the detected change in distance to adjust a position or orientation of the line scan array imaging system such that the constant distance is maintained.

28. the imaging is a first imaging, 28. The method of claim 26 or 27, comprising re-imaging the surface.

29. 30. The method of claim 28, further comprising performing a surface treatment operation between the first imaging and the second imaging.

30. 30. The method of claim 28, wherein the imaging path is a first imaging path, the second imaging occurs along a second imaging path, and the second imaging path is different from the first imaging path.

31. 30. The method of claim 28, wherein the image is a first image and the second imaging produces a second image, and an area of ​​the surface is visible in both the first and second images.

32. 30. The method of claim 28, wherein the image is a first image and the second imaging produces a second image, the first image including a first surface area and the second image including a second surface area, and the first and second surface areas do not overlap.

33. The method of any one of claims 26 to 32, wherein the imaging system is in a dark field configuration.

34. The method of any one of claims 26 to 33, wherein the imaging system is in a near darkfield configuration.

35. A method according to any one of claims 26 to 34, wherein the imaging system is mounted on a robotic arm and the imaging system is moved along the imaging path by the robotic arm.

36. The method of claim 26 , wherein the imaging system is mounted on an unmanned aerial vehicle.

Images