Surface modification system and method

Abstract

A defect tracking system includes an image capture device configured to capture an image of a surface. The system also includes a defect flag receiver configured to receive the defect indication. The defect indication includes a defect on the surface. The system also includes a defect mapper configured to map defects to the surface. The system also includes a user interface generator configured to generate a defect map for showing the detected defects. The system also includes a display component configured to display the defect map.

Description

Background Technology

[0001] Many vehicles, such as cars, trucks, boats, and airplanes, require parts replacement during their lifespan due to failure or damage. The repair industry handles a large volume of both new and used parts during vehicle repair. Within the crash repair industry, complex repairs are performed daily in repair facilities. The repair process is paused when incorrect, damaged, or faulty parts or materials are delivered until the correct parts and materials are obtained. Summary of the Invention

[0002] A defect tracking system includes an image capture device configured to capture an image of a surface. The system also includes a defect marker receiver configured to receive a defect indication, which includes a defect on the surface. The system further includes a defect mapper configured to map defects onto the surface. The system also includes a user interface generator configured to generate a defect map illustrating the detected defects. Finally, the system includes a display component configured to display the defect map. Attached Figure Description

[0003] In accompanying drawings that are not necessarily drawn to scale, similar figures may describe similar components in different views. The drawings are generally shown by way of example but are not limited to the various embodiments discussed in this document.

[0004] Figure 1 This is a schematic diagram of a robotic surface modification system useful in one embodiment of the present invention.

[0005] Figure 2 The embodiments described herein are illustrated in the process of repairing vehicles.

[0006] Figures 3A to 3D An example is provided of the real-time marking and tracking of defects during vehicle repair according to the implementation scheme described herein.

[0007] Figures 4A to 4D An example is illustrated of a system for real-time marking and tracking of defects during vehicle repair, according to an embodiment of this document.

[0008] Figures 5A to 5D An example is illustrated of a system for accurately applying body filler to a vehicle according to an embodiment of this document.

[0009] Figure 6 An example is provided of a method for determining repair operations according to the implementation scheme described herein.

[0010] Figure 7 An example user interface for repair technicians is illustrated according to the implementation scheme described herein.

[0011] Figure 8 An example of a digital defect tracking system implemented using a projection system according to the embodiment described herein is illustrated.

[0012] Figure 9 An example of a defect tracking system based on the implementation scheme described herein is presented.

[0013] Figure 10 It is a defect tracking system architecture.

[0014] Figures 11 to 12 An example of a mobile device that can be used in the implementation shown in the previous figures is illustrated.

[0015] Figure 13 This is a block diagram of a computing environment that can be used in the embodiments shown in the previous figures. Detailed Implementation

[0016] In the crash repair industry, complex repairs are performed daily in repair facilities. Some repairs require surface modification—for example, removing bumps from a surface—while others require significant time and materials—such as smoothing or filling dents. Completing repairs in a timely manner presents a significant challenge for technicians. Since many repairs are paid for by insurance companies (which typically offer fixed compensation per repair operation), it is crucial to complete the repair correctly the first time, as insurance usually only covers the initial repair and not any additional work to correct errors.

[0017] Finding and tracking numerous surface defects on vehicles with specular reflective surfaces is challenging. Furthermore, once defects are identified and their locations recorded, operators typically spend additional time relocating them before repair. Defects can be physically marked, for example, with crayons or other tools, but these markings are usually removed during the repair process, making relocation during quality inspection difficult. Additionally, many repair operations require multiple passes—for example, the first sanding operation removes the markings, making it more difficult to find defects when performing the next, finer sanding operation (e.g., moving from 240 grit to 320 grit). This also makes measuring surface bluntness difficult before moving to even finer grits.

[0018] In some cases, the repair may need to be redone due to actions taken during the repair process. For example, burning may occur if too much heat is generated during a grinding operation. Dent repair may result in applying too much or too little filler, leading to extra time for reapplying filler or causing excessive material to be ground away.

[0019] Furthermore, in some cases, such as due to damage during transport or rust removal, it may be necessary to repair the potential replacement part itself. While the replacement part can usually be returned, the time spent returning and receiving the new replacement part may exceed the timeframe for a given repair. Therefore, the repair shop may consider repairing the replacement part more valuable than returning it. The system and method described in this paper improve the efficiency of the repair process by making it easier to locate and track defects and generate repair parameters based on contextual information such as existing abrasive articles, defect details, etc.

[0020] As used herein, the term "vehicle" is intended to encompass a wide range of moving structures that receive at least one paint coating and / or varnish coating during manufacturing. While many examples herein relate to automobiles, it is explicitly envisioned that the methods and systems described herein are also applicable to trucks, trains, ships (with or without motors), airplanes, helicopters, motorcycles, and the like.

[0021] The term "paint" is used broadly herein to refer to any of the various layers of paint, filler, primer, varnish, clear coat, etc., applied during the finishing process of a vehicle. Additionally, the term "paint repair" refers to locating and repairing any visual artifacts (defects) on or within any of the paint layers. In some embodiments, the systems and methods described herein use a clear coat as the target paint repair layer. However, the proposed systems and methods are applicable to any particular paint layer (electrophoretic paint, filler, primer, varnish, clear coat, etc.) with minimal modification.

[0022] As used herein, the term "defect" refers to an area on a work surface that contains an imperfection that needs to be removed or repaired. A defect can include anything that detracts from visual aesthetics. For example, many vehicles have specular or reflective surfaces that may appear glossy or metallic after painting. A "defect" can include debris trapped within one or more of the various paint layers on a work surface. Defects can also include paint stains, excess paint including spots or drips, and dents or scratches.

[0023] As used in this article, the term "real-time" refers to data being processed within milliseconds, making it virtually immediately available for feedback. While some delay due to processing is unavoidable, "real-time" is intended to encompass systems and methods that can collect or input data and allow users to interact with that data without significant delay. For example, a user can input data into the system, and that data is subsequently available for viewing or editing virtually immediately.

[0024] Figure 1This is a schematic diagram of a vehicle repair environment useful in one embodiment of the present invention. Environment 100 includes a lighting system 110, which can be controlled by a motion controller 112 that can receive instructions from one or more application controllers 150. The application controllers can receive input or provide output to a user interface 160. In some embodiments, the lighting system 110 also includes an inspection system for imaging the surface of the vehicle 130.

[0025] The first of the two main challenges, namely the inspection of vehicle 130, is of interest due to the nature of the potential problem domain. Generally, the surface of interest is very large compared to the defects themselves, with differences ranging from several orders of magnitude. Furthermore, each paint layer in the finishing process (electrophoretic paint, primer, varnish, clear coat, etc.) differs in its visual appearance, with specular reflectivity being particularly noteworthy. Highly specular surfaces (i.e., high-gloss or high-reflectivity surfaces) pose unique imaging challenges. These issues together make the inspection and localization of small defects on the surface difficult.

[0026] Figure 2 The embodiments described herein illustrate a process that can be useful for repairing vehicles. Process 200 can be performed by repair technicians, for example, as part of the vehicle's entry into the workshop, or by repair technicians during repair operations.

[0027] In box 210, defects requiring repair are detected. Due to the specular reflective nature of many vehicle surfaces, specific lighting setups are often necessary. Fluorescent lamps, for example, are advantageous because they produce sharp lines in the reflection. A smooth surface will ensure that the reflected line has no sharp local deviations along its length. As the line passes through the defect (e.g., by moving the light source or the operator's relative position to the light source), the reflection will deviate according to the high curvature in the defect, thus aiding in detection. This is the principle behind structured light and deflection measurements.

[0028] Repair technicians or inspection systems can inspect the entire surface of a vehicle for defects that need to be repaired and can create a list of defects for later repair.

[0029] In box 212, for each detected defect, a repair option is selected. For example, depending on the location and size of the detected bump, a specific grinding operation can be recommended—for example, a specific abrasive grit size for the sanding step, a specific polishing for the polishing step, and a contact time for each step.

[0030] In box 214, a specific defect is located on the surface of the vehicle. For example, the list may include the coordinates of the defect that can be located using a 3D model of the vehicle surface. The list may include a description—for example, a hood defect near the driver's side door. The list may also use other suitable location identification information, such as alphanumeric identifiers.

[0031] Once the defect has been located, the selected repair can be performed. For example, for a dent, it may be necessary to add material 222 (e.g., fill material), remove material (e.g., sand the surface), or perform another operation 226 (e.g., smooth the dent from the surface).

[0032] After the defect has been repaired, a quality inspection can be performed on the surface surrounding the defect, as indicated in box 216. This can be a visual inspection by a technician or based on images taken by a technician.

[0033] Then, in box 218, the technician can continue to repair additional defects by repeating the operations in boxes 212 through 216. If no additional defects are found, a final quality inspection can be performed on the vehicle, as indicated in box 230.

[0034] like Figure 2 As illustrated, a repair technician can move between initial inspection and quality checks through a series of defect repairs. Because locating defects on the surface can be difficult, an iterative approach is preferred for repairing defects: locate the first defect 214, select a repair 212 for the defect, perform the repair—adding material 222, removing material 224 or other steps 226, and then move along the surface to locate the second defect 214. This prevents time loss due to first locating the defect (as illustrated in box 210) and then having to relocate each defect (as illustrated in box 214) for each repair.

[0035] Additionally, finding defects after repair can also be problematic. Once the surface is planarized with, for example, 1200 grit, the polished area is matte, which self-identifies that point for future operations (refining, polishing). However, this can be inefficient because it is not always clear which specific areas the user has investigated, and given a panel with many matte sanded areas, it is impossible to indicate that all the space between those points has been checked and found to be defect-free.

[0036] In some implementations, improved lighting setups are used for post-repair inspections. For many repair shops, a positive rating from an insurance company is crucial to their reputation and future business. Ensuring the vehicle is clean, free of repair materials, free of repair-induced defects, or free of incompletely repaired defects makes a positive rating more likely. Some post-repair items for repair shops include: dust on exterior panels, polishing compound in door frames or gaskets / molding parts, interior burns caused by welding spatter, dust on interior surfaces (dashboard, seats, etc.), masking tape on the vehicle (including wheel wells and under the vehicle), overspray (indirectly deposited paint with a dusty appearance) on adjacent panels and / or glass surfaces, debris cleaned from the vehicle (shops will handle things like fast food bags), any old parts removed from the back seat, floor, trunk (shops sometimes store removed parts in the vehicle after removal), or grease or dirt on the interior trim. Many of these tasks are difficult to identify using a computer vision system due to the different colors and shapes of vehicles, things around the vehicle, mirrored surfaces, etc.

[0037] In some implementations, the vehicle is moved from the repair area to the quality inspection area. In some implementations, the quality inspection area may be a space with consistent lighting. The quality inspection area may include structured patterns or repeating patterns for feature detection. The quality inspection area may include a reference image or background.

[0038] In some implementations, the quality inspection area includes a tent, such as one with grid lines (or other repeating patterns) and consistent lighting, for feature inspection or curvature. The lighting may be adjustable, or positioned to amplify the appearance of dust, or may cycle between different colors to make polishing compounds or masking tape stand out from the surrounding environment.

[0039] In some implementations, the quality inspection area for vehicles is separated from the repair station, allowing repair technicians to move the vehicles to the quality control station once repairs are complete. While imaging and analysis are being performed, technicians may be working on a second vehicle that has already been moved to the repair station. Once imaging is complete, technicians may receive instructions on areas requiring re-repair.

[0040] A separate lighting station can also be used to capture defect information before repair. Pre- or post-repair imaging can also serve as a repair record in case of customer disputes.

[0041] Another potential solution is to provide technicians with a system for real-time defect tracking. Digital options for marking defects would provide a map of defects that need fixing, eliminating the need for any removable markings on the vehicle. A digital defect tracking system could also help track which areas of the vehicle have been inspected for defects, which defects have been fixed, and which still need fixing.

[0042] In some embodiments described herein, the digital defect tracking system is implemented using an image capture system. For example, the lighting system 110 may be part of an external imaging system. A technician may interact with a display (e.g., user interface 160) coupled to the imaging system to mark defects on the captured images. The imaging system or controller (e.g., controller 150) may be able to compare the captured and marked images with a 3D model of the vehicle's surface to track which surfaces of the vehicle have been inspected for defects and which have not yet been inspected.

[0043] In some embodiments described herein, the digital defect tracking system is implemented using augmented reality, for example, enabling technicians to mark defects they see in real time. Technicians can use image capture devices, for example, built into glasses or a mask, to view the vehicle surface and indicate defects on the surface. Such a system allows technicians to indicate the location of defects in real time within a digital record. This may allow for the indication of repair operations, repair time, and repair personnel for the indicated defects.

[0044] In some implementations, a digital defect tracking system allows technicians to digitally indicate defects during a full inspection of the vehicle before any defect repairs are performed. In other implementations, technicians can use such a system to repair detected defects while tracking which areas have been inspected and which have not yet been inspected.

[0045] As already noted, lighting systems are another potential solution. Automotive OEMs can use deflection measurement equipment to provide sufficient light to image surfaces and automatically detect defects from the images. However, deflection measurement equipment is an expensive solution, while humans are typically fast and accurate at identifying defects on surfaces. Digital defect tracking systems, such as those discussed herein, can be used to instruct automated defect repair units to repair defects at locations indicated by technicians. Digital defect tracking systems can use images of the indicated defects—captured using augmented reality imaging devices or stand-alone surface imaging systems—to generate 3D coordinates for use by robotic systems. In some implementations, the 3D coordinates are generated using machine learning techniques. In some implementations, the 3D coordinates are generated by triangulation of the location using reference markers and multiple cameras. In some implementations, a 3D model of the vehicle is provided (e.g., from the manufacturer or other source), and the digital defect tracking system indicates the defect by comparing the defect indication to the 3D model. In some scenarios, the vehicle is damaged only on one side (e.g., only the driver's side or passenger side). The undamaged side can also be used to assess the repair needs of the damaged side.

[0046] Figures 3A to 3D An example is illustrated of the real-time marking and tracking of defects during vehicle repair according to the implementation scheme described herein. This system can improve efficiency by allowing repair technicians to track which areas have been inspected and which have not, the location and type of defects requiring repair, and which defects have already been repaired.

[0047] Figure 3A A schematic diagram of a vehicle 300 with several defects is illustrated. Users of the digital defect tracking system can record the location of defects in real time as they are noticed. First, a repair technician may notice defect 302, which could be, for example, a bump. Then, the technician may move toward the driver's side door and notice defect 304, which could be, for example, a paint stain. Then, the technician may notice a third defect 306.

[0048] Figure 3BAn interactive device 320 is illustrated, displaying a user interface through which repair technicians can interact with a defect detection and tracking system. The defect detection and tracking system may include information 322 about vehicle 300. Repair technicians can use interface 320 to indicate defects upon discovery of defect 310 and to enter defect details, as indicated by the identified defect 330. While a defect ID and type are indicated for each detected defect, it is explicitly envisioned that additional information may be displayed or otherwise accessible, such as the repair plan selected for each defect, or, if the defect has been repaired, details about the repair, such as the personnel, time, and materials used. The defect ID may also indicate which defects are pre-existing defects compared to accident-related damage. Technicians may also be able to add new defects, as indicated in box 324, or mark the completion of repairs for specific defects, as indicated in box 326. Figure 3B Some functionalities are illustrated herein, but it is clearly envisioned that digital defect tracking systems may include other functionalities as described in the embodiments herein.

[0049] Figure 3C and Figure 3D This example illustrates a user interface that users can interact with in a digital defect tracking system. Figure 3C Examples of defects 340 that have been detected and entered into the digital defect tracking system by technicians are illustrated. Uninspected areas 350 are also indicated. In addition to detected and repaired defects, the defect detection and tracking system according to the embodiments herein may be able to track inspected and uninspected areas of a vehicle. Figure 3D This example illustrates another user interface that can be presented later in the vehicle repair section. Figure 3D An example is shown of a vehicle 300 with several repaired defects 360 and one unrepaired defect 370. (Source: [Original Source Name]) Figure 3C Area 350 is no longer indicated as uninspected. Now, inspected area 352 is indicated as defect-free.

[0050] Figures 3B to 3DAn example of a user interface that a repair technician can interact with on a mobile device is illustrated. However, users can interact with the digital defect tracking system using other suitable interfaces, such as augmented reality interfaces, or through another image capture system that allows the user to indicate the location of a defect on an image or 3D model. The location of a defect indicated on an image can be correlated with its physical location on a vehicle. For example, the defect detection and tracking system may have access to a database of 3D models of the vehicle, provided, for example, by the vehicle manufacturer or another source. Using augmented reality, the systems and methods of this paper can correlate the user's movement with coordinates on the 3D model. In other embodiments, an image of the vehicle indicating the defect is used. The location of the defect can be determined from the image using machine learning techniques or other suitable techniques.

[0051] Repair technicians can use any suitable method to indicate the location of a defect. For example, in some embodiments, a user can point to the location on a vehicle and make audio or haptic cues that can be picked up by an augmented reality system. In other embodiments, a technician can use a suitable image capture device to capture an image and, for example, use a touchscreen to indicate the defect location on the image. In still other embodiments, a technician can place a marker on the vehicle, which is then captured by the image capture device. An image analyzer can then correlate the indicated defect location with 3D coordinates relative to the vehicle.

[0052] Figures 4A to 4D An example is illustrated of a system for real-time marking and tracking of defects during vehicle repair, according to an embodiment of this document. Figures 3A to 3D Examples of user interfaces generated by a digital defect tracking system are shown. Figures 4A to 4B Examples of devices that can reside in or be accessed through an implementation of a digital defect tracking system are shown. Figure 4A A schematic diagram 400 illustrates a restoration technician 410 wearing augmented reality-enabled glasses 420. Glasses 420 may include, or interact with, a camera or other image-capturing device.

[0053] The glasses 420 include one or more processors that interact with a registration system that generates icons corresponding to real-world points (such as points on a vehicle being repaired). For example, the point where a first curve intersects a second curve on the vehicle can be used by the registration system as a guide to the location of an indicated defect in three-dimensional space. The registration system can use the identified points to generate a map of the space. The registration system can also correlate the detected defect locations with the generated map, allowing these defect locations to be overlaid on the user's field of view 422.

[0054] Augmented reality systems (such as glasses 420) include or have access to an augmented reality generator implemented using one or more processors, which integrates images captured by an image capture device with a map generated by a registration system, allowing information to be overlaid on the images from the image capture device. The augmented reality generator provides a combined image to the user, presented on a display (e.g., projected onto glasses 420 or provided via another display).

[0055] Through glasses 420, the user has a field of view 422. When the vehicle 430 that needs repair enters the field of view 422, the previously marked defects 440 can be presented to the user 410 using glasses 420.

[0056] Although Figure 4A An embodiment in which a user wears augmented reality glasses 420 is illustrated, but it is clearly envisioned that the digital defect tracking system could be integrated into other devices with glass, plastic, or other suitable surfaces for displaying maps generated by the registration system. For example, many repair technicians could wear personal protective equipment including a transparent shield to protect the user's eyes. Figure 4B The example shown is a face mask 450. Figure 4C An example is the electric air-purifying respirator 460. Figure 4D Safety goggles 470 are shown as an example. Other equipment such as welding helmets may also be suitable.

[0057] Furthermore, while augmented reality technology has been discussed in this paper, it is also envisioned that, in some implementations, a virtual reality system with an external imaging system could be used. Other suitable technologies are also envisioned.

[0058] It should be noted that some augmented reality devices are designed to operate wirelessly, enabling remote storage and / or processing of data, which reduces power consumption requirements and allows for longer operation between charges. However, it is explicitly envisioned that some systems described herein may include at least some local storage or analytics components. For example, a helmet (such as a face mask 450) may include additional space for a power supply sufficient to manage local data and / or for analytics.

[0059] Use digital defect tracking systems (such as Figures 3A to 3D and / or Figures 4A to 4D The digital defect tracking system described herein reduces the difficulty of relocating defects for repair. Similarly, because defect information (location, type, etc.) is stored digitally, this information will not be erased, sanded away, or otherwise obscured by other repairs in the same area. Digital defect tracking systems (such as those described herein) can also help confirm that the entire vehicle surface has been inspected for defects and that all located defects have been repaired.

[0060] While defects are exemplified herein as small imperfections on a surface, it is also explicitly envisioned that the system described herein could also be used to store relevant information about larger defects. For example, for a larger dent, relevant information could include the amount of body filler required for repair, which is related to the amount paid for the defect repair. For instance, volume deviation from standards could be measured. This would help provide consistent quantification. Volume deviation can be measured in any suitable manner using appropriate dent measurement tools (such as those from Collision Edge). ™ The Dent Viewer MD from Innovative Repair Systems, or a Dentstick (available from Dentstick), can be used for measurement. For example, it can measure the size and / or curvature and / or depth of a dent.

[0061] Figures 5A to 5D This paper illustrates a system for accurately applying body filler to a vehicle according to an embodiment of the invention. Some defects require the application of filler or patch material to recreate the original curvature. Applying filler, especially where curvature needs to be matched (e.g., to maintain the symmetry of the vehicle), is a fusion of art and science. The filler is applied in a wet form and then sanded after drying. Before applying filler, the surface is sanded to expose the metal (e.g., to remove all paint). The filler is a mixture of two parts of material in the desired proportions and is typically applied by hand using a tool—a scraper, applicator, or other tool. When dry, the body filler is very hard. The goal of the repair technician is to apply enough filler to achieve the desired profile, but not too much, as over-application will require more time and material for sanding.

[0062] Figure 5A An example process for applying the main filler is illustrated. Figure 500 illustrates the first step, in which a surface 504, for example, dented in a collision, needs to be restored to the target profile 502. Figure 510 illustrates an intermediate point in the repair process, where material has been applied to form profile 506. Ideally, profile 506 is applied so that no additional material is needed; for example, profile 506 should not be smaller than the target profile 502. After the material removal step, the target profile 508 is achieved, as indicated in Figure 520.

[0063] To further complicate matters, the application and sanding of the main filler leaves behind a range of surface textures and heat capacities, ranging from the smooth, undamaged sections of the panel with a clear coat, to the feathered, worn clear coat sections exposing the metal, and the central areas of filler at various heights. Currently, the curve is verified visually and by physical touch to detect deviations, flatness, etc. However, "touch" can be misleading due to variations in surface texture and heat capacity. Furthermore, visual verification of the curve is difficult because the filler material is matte, as the eye perceives curvature differently on a matte surface than on a specular surface. Therefore, in some cases, the curve cannot be verified until a uniform gloss coating is applied and examined.

[0064] The process of repairing indentations and verifying contours and curvatures on filled surfaces can be significantly improved by using vision systems, such as those described in this paper.

[0065] Figure 5B An example is illustrated of a vehicle 530 in which a defective area 532 has been indicated, for example, using a digital defect tracking system. In some embodiments, the digital defect tracking system may, for example, retrieve a target contour from a database containing a 3D model of the vehicle 530. In other embodiments, assuming the passenger side of the vehicle 530 is relatively undamaged, a contour of a similar area on the opposite side of the vehicle may be imaged. Based on this image, a mirror image of the contour may be generated, which can be used as the target contour. Based on the generated target contour, feedback may also be provided to technicians using AR-enabled devices or another imaging technique regarding whether sufficient filler has been applied or whether the area has been over-sanded.

[0066] The system and method described in this paper can also periodically scan the processing region, for example, Figure 5C The image illustrates how to track filler application and abrasion volume. Figure 5C An image 540 that can be captured during the repair process is illustrated. Technician 542 uses applicator 544 to apply filter 546. The filler material is not perfectly smooth along the surface during application. Using audio, visual, or tactile feedback, the system described herein can provide real-time feedback and / or guidance to technicians based on periodic images and appropriate image analysis discussed below. This can be particularly useful when applying the bulk filler to reduce the amount of time required for sand-hardened filler material.

[0067] Image analysis techniques can also be used to provide information that is not easily detected by human technicians, such as the smoothness of a surface in different areas, transition areas between materials and between thermal capacities, and key geometry—for example, sharper, rounded body lines. According to the embodiments described herein, using projection systems or augmented reality systems, areas that indicate higher and lower points can also be provided in a color-coded or otherwise manner.

[0068] Although Figures 5A to 5C The discussion covers scenarios where a human operator is applying the main filler, but also explicitly points out that robotic repair units face similar challenges. Repair robots may have additional sensors for comparing curvature (e.g., cameras, lasers, analysis of deflection measurements, etc.), but robotic systems also have their own limitations. Real-time feedback on the position of the current profile relative to the target profile helps program the robotic repair unit to grind the applied main filler toward or near the target profile. Reference profile geometry, along with real-time information about the current profile, can be fed into the grinder, which is programmed to achieve a desired material removal rate, a desired amount of material removed, or a desired volume of material removed. The grinder can be periodically stopped for additional profile checks and reprogramming. For example, the robot may not know the specifications of the abrasive product (e.g., whether the correct product is installed and whether it is new or worn), and therefore lack accurate information about how much material is removed during the grinding operation.

[0069] A digital defect tracking system provides current and target contour information to an abrasive repair planner, which can select parameter settings for a robotic repair unit or another suitable intelligent tool to remove most of the material. In at least some implementations, technicians may need to perform final processing on transition areas through smoothing and feathering.

[0070] While this article has discussed augmented reality technology, it should be noted that in some cases, augmented reality devices may be too expensive. Figure 5D An example schematic diagram 550 illustrates an image of contour 567 captured by an image capture device 552. While an image capture device 552 is illustrated, other sensors, such as laser measurement systems (LIDAR or other suitable systems), may be suitable. The contour capture device 552 may have a field of view 554 or a measurement range 554.

[0071] As previously noted, specular surfaces can be difficult to image due to reflection. The presence of light sources in the room, along with the image capture device 552, can cause problems in accurately detecting the contour. Obtaining a mirrored contour from the undamaged side proves difficult. Some techniques that can be used may include a specific light source 560. For example, the light source 560 may be configured to flash a known light pattern, allowing deflection measurement techniques to capture the surface.

[0072] It should be noted that a single topography capturing device 552 and a single light source 560 are illustrated in schematic diagram 550. However, it is clearly contemplated that in other embodiments, any one or both of such components may be suitable. For example, a pair of stereo cameras may be able to detect speckle patterns applied to a surface. Additionally, the image capturing device 552 may have a moving mechanism (not shown) such that the image capturing device can be moved relative to a vehicle having a profile 556. In some embodiments, photogrammetry is used to obtain the topography of the vehicle.

[0073] While examples of applying body filler have been discussed in this paper, it is clearly envisioned that other applications could benefit from the systems and methods described herein—such as investigating weld joints or performing paint-free dent removal. For example, welding parameters may be specified by the manufacturer as the total number of welds along the joint or the spacing between welds. The weld can be imaged to determine whether a technician has met the required number and spacing.

[0074] Figure 6 A method for determining repair operations according to an embodiment of this document is illustrated. Method 600 can be implemented using a digital defect tracking system (such as the embodiment described herein) or using another suitable system.

[0075] At box 610, the target surface of the vehicle is retrieved. For smaller defects, the target surface may be a surface with small dents removed, bumps removed, or free of blemishes. For larger defects, such as large dents or areas requiring body filler or welding, the target surface may be more difficult to estimate. In some embodiments, retrieving the target surface includes scanning the opposite side of the vehicle to obtain a mirror profile, as indicated by box 602. In some embodiments, retrieving the target surface includes retrieving a CAD model provided by the manufacturer or otherwise generated, as indicated by box 604. In another embodiment, retrieving the target surface may include accessing a database of topographic and curvature samples, as indicated by box 606. Other suitable methods for retrieving the target surface are also envisioned, as indicated by box 608.

[0076] At box 620, the initial surface of the vehicle is captured; for example, the topography of the current surface can be obtained using an image from an image capture device 612. For example, photogrammetry techniques can be used to stitch the images together, or the images can be mapped onto a CAD model. Additionally, depth can be determined using the captured images and a reference. Other techniques for capturing the initial surface can also be used, as indicated in box 614. For example, a laser sensor can be used to obtain depth information. Based on the differences between the target surface and the initial surface, a repair plan can be generated. For example, it may be necessary to add material to improve the current surface, or it may be necessary to remove material to reduce the current surface size.

[0077] At box 630, the surface morphology in the current processing is obtained. The processing morphology can be obtained using any suitable technique, including photogrammetry, benchmarking, laser sensors, or another suitable option.

[0078] At box 640, the surface morphology currently being processed is compared with the target surface to provide a measure of the current repair progress.

[0079] At box 650, a repair status is generated based on a comparison between the current surface and the target surface. In some implementations, the generated status includes a percentage of completion, an estimated remaining sanding time, or the required bulk filler, or another indication.

[0080] At box 660, determine the next step for the repair. For additive processes, such as adding body filler, welding material, or another material, the next step can be determined based on the repair status—for example, if completed, move to sanding, or if not completed, add more material. For subtractive processes, such as sanding according to a profile, the next step could be, for example, moving to a different grit size or continuing at the current grit size.

[0081] The system and method described in this paper can generate instructions for the next step based on available context information. Of particular concern is the need to prevent burn-through caused by heat generated from excessive dwell time, worn abrasive articles, etc. Understanding the conditions associated with the abrasive article is especially important.

[0082] At box 680, retrieve the current or available abrasive specifications. The condition of the abrasive article or article wear can be obtained 682 by scanning or imaging the selected abrasive article. The system described herein can also receive information about the available abrasive article product series, grit size, polishing compound, and other materials.

[0083] As indicated in box 684, information about the selected or available tools used in the sanding operation can also be retrieved, such as tool brand, model, settings (e.g., air pressure and rotational speed), interface material (e.g., rubber or foam hardness), and user conditions (e.g., downforce, angle to the surface, etc.). Additional information related to the amount of force and / or friction that the tool can apply, or other information that may affect the material removal rate, can also be retrieved.

[0084] As information about the grinding operation is gathered, a better understanding of the relationship between abrasive and tool conditions and effective material removal rates can be achieved. This can provide accurate insights into tool conditions across the surface area contact range and quantify the residence time per area during the process. This can be used to alert technicians to the risk of coating burn-through, especially when profiling is frequently obtained during the process and / or a camera captures the entire process and analyzes the image feed in real time or frequently. Real-time feedback can also be provided, for example, from pressure sensors in the support pad, force or angle sensors in the tool, and capacitance / resistance sensors in the support pad, and foam / rubber deformation can be estimated in relation to force / angle to estimate surface area contact.

[0085] As indicated in box 686, information about abrasives and tools can be obtained from sensors; for example, an image of the work area can identify the tools used by a technician. Image analysis systems can identify abrasive articles or tools from an image by shape or color or by detecting barcodes or other identifiers. Similarly, barcode scanners or RFID sensors can detect tools or abrasive articles based on RFID tags or barcodes. In some embodiments, information about abrasive articles and / or tools can be manually entered, as indicated in box 688. For example, a user interface that a technician can use during a repair process is illustrated herein. For the current repair, the abrasive articles and / or tools used or planned to be used can be indicated. Other sources and types of abrasive specifications are also envisioned, as indicated in box 689.

[0086] At box 690, a set of grinding parameters for the grinding operation is generated. In some embodiments, the set of grinding parameters is provided via a user interface as a recommendation for the next step. Having recommended tools, artifacts, angles, and / or residence times can be particularly helpful for novice repair technicians. The set of grinding parameters may include the selected abrasive artifact type and grit size, the grinding tool, and / or the angle of attack or force to be applied. Additionally, the residence time may be generated based on heat capacity, calculated based on the known bulk filler composition and thickness. As indicated in box 692, the generated grinding parameters may be stored in a local data repository or in a data repository accessible wirelessly or via a cloud-based network.

[0087] At box 670, a set of parameters is provided (e.g., via a user interface) to the repair technician for use in the next step. If the next step is a sanding or polishing step, the parameters may include grinding specifications 632. The parameters may include information about the material to be applied (e.g., body filler, welding material, etc.), as indicated in box 634. Other information may also be provided, as indicated in box 636.

[0088] In some embodiments, the parameters may be provided as an overlay to the augmented reality screen, as indicated in box 674. In some embodiments, the parameters are provided to a user interface for display on a monitor or projector, as indicated in box 672. In some embodiments, the parameters are provided as an audible alert, as indicated in box 676. However, it is clearly contemplated that other communication methods are possible, as indicated in box 678.

[0089] Figure 7 An example user interface for a repair technician according to an embodiment of this document is illustrated. While the user interface 700 is illustrated as being presented on a device with a display (such as a tablet or mobile phone), it is explicitly contemplated that such information could be presented on a monitor in the workplace, projected onto a surface, or otherwise presented to one or more users. Figure 7 As illustrated, a digital defect tracking system can provide functionality throughout the entire vehicle repair process.

[0090] Vehicle information 720, along with other information related to the detected defects, can be presented for vehicle 710. For example, the vehicle can be presented with prominent defects—such as remaining defects 750 that still require repair. Users can use feature 702 to select defects and view or edit repair details of completed repairs 740 or remaining repairs 750. Users can also use feature 706 to mark a repair as completed.

[0091] The system can present technician information 760. A digital defect tracking system can track which technicians completed which remediation steps, for example, by requiring login, using biometrics to identify users, or by manually entering information. The system and method described in this paper can track additional details about the remediation, such as important timing information 730. For example, based on an insurance estimate, the target remediation time for defects 1 through 3 could have been 3 hours and 12 minutes. So far, only 2 hours and 45 minutes have been used for those defects, and the current remediation is 27 minutes behind the target. Besides whether the remediation is satisfactory, tracking the time spent on remediation can be useful for both quality control and training purposes.

[0092] Augmented reality interfaces may not be readily available in all repair shops or work environments. Instead, using a single display that can store information from multiple sources is more economical. Additionally, since many technicians own mobile phones, they can use mobile apps to interact with digital defect tracking systems. Mobile phones or tablets can also be used as augmented reality-enabled devices.

[0093] As illustrated by technician information 760 and repair time 730, a great deal of statistical information related to technicians can be tracked. In some implementations, information such as the time spent on previous repairs and touch time statistics can be included.

[0094] This article discusses many systems and methods for technician-user applications. However, it is explicitly envisioned that information collected using a digital defect tracking system can also be presented to supervisors or store managers in a useful format. For example, instead of a single vehicle 710, the user interface 700 could allow the supervisor to select from six different vehicles currently undergoing repairs. Instead of a list of completed and remaining repairs, the supervisor could see the completion percentage, ongoing repairs, and / or time statistics 730 for multiple vehicles at once. This information can be used to quickly identify the position of different vehicles in the repair workflow and how the technician is currently performing their repair work.

[0095] While visual displays can be beneficial for notification and alerting purposes, in some situations it may not be feasible for technicians (or all technicians) to have their own personal displays. In some workplaces, it is desirable to bring displays to technicians and have multiple staff members share a centralized projection system display. Projection systems can also provide the opportunity to project information onto surfaces that would otherwise be difficult to visually annotate.

[0096] Figure 8 An example of a defect tracking system according to an embodiment of this document is illustrated. Embodiments of this document relate to: a system for delivering visual warnings and information to a technician via a projection system; and a computer vision system for detecting and identifying the technician's identity, and / or identifying the technician's location in the environment and the possible location of a suitable projection surface within the technician's field of vision.

[0097] In some embodiments, the defect tracking system includes a projector, such as a DLP, LCoS, LCD, LED, laser, etc. In some embodiments, the defect tracking system is configured to detect, identify, and locate objects (e.g., products, tools, vehicles, people, etc.). In some embodiments, a computer vision system, such as a camera and edge computing device, can be used for detection. In some embodiments, detection includes an image capture device with a communication component that transmits images to a cloud-based computer vision system for computation. The system described herein may include one or more conventional cameras, web cameras, RGB+ depth cameras, or other suitable image capture devices. In the embodiments described herein, one or more image capture devices are aligned and calibrated with one or more projectors.

[0098] Figure 8An example embodiment of a digital defect tracking system implemented using a projection system according to the embodiments described herein is illustrated. System 800 exemplifies a projector 802 and a camera 806 having a field of view 804. However, it is explicitly contemplated that the embodiments described herein may utilize multiple projectors 802 and / or cameras 806. Using camera 806, real-time images can be provided to defect tracking system 810, which can perform image analysis and then provide information for projection by projector 802. Projector 802 is illustrated as projecting information onto vehicle 804; however, the information may be presented on any suitable surface. Figure 8 The illustration shows two defect locations 832 covering the vehicle 840, and describes 834. Although in Figure 8 The text only illustrates defect signs and types, but it clearly envisions that a projector with sufficient resolution could display more information, such as repair plans, status, equipment and materials used.

[0099] Camera 806 can also detect movement or audio from technicians or other people in the workplace. Voice or gesture recognition can be used to modify projected information, and computer vision can be used to provide context-aware information based on the identification of the current task before it is performed. Similarly, in some embodiments herein, alerts or other notifications can be provided audibly.

[0100] Given the current task to be performed, a worker may require a specific part or tool to complete their work. As discussed in this paper, vision systems can use registration systems to directly classify objects in a scene, or classify them by identifying and reading optical labels / barcodes (visible light or IR retroreflective). Classifying objects can include using classic feature-based detectors or registration systems that employ appropriate machine learning techniques.

[0101] The implementation described herein allows the defect tracking system to locate the required object in a scene. For example, camera 806 can locate wrench 803, and the controller can actuate light source 824 to highlight the area, for example, by providing an effect similar to floodlighting around the object. This will reduce unnecessary search time or frustration for workers caused by misplacing or tracking a large number of specific objects needed to complete a task.

[0102] The defect tracking system 810 can also control the operation of light sources 822 and 824, or communicate with a controller for the light sources. Light sources 822 and 824 may be capable of attenuating illumination. For example, light source 824 may provide a gradual change in floodlight effect based on the order of operations or based on a specific task. For example, all tasks required in the first step can be highlighted similarly. Additionally, for missing items, projector 802 can provide an indication of the required item, such as text, image, or location of the required item (e.g., from an inventory or asset management system).

[0103] The defect tracking system 810 can be triggered to assess whether something is missing from the scene, to display the location of the defect and / or any other information based on multiple modalities (e.g., direct voice request or gesture). For example, “Where is my wrench?” triggers camera 806 to search for wrench 803, or a raised hand causes projector 802 to add, change, or remove projected information 832, 834.

[0104] A defect tracking system, including a projection system, can provide real-time feedback and alerts to staff based on requested data or information about their assigned tasks. This system will reduce the installation and maintenance burden of pseudo-augmented reality in stores, where classic AR systems may be too bulky for staff to wear during operation, or where there are hazards / risks in environments where these systems might be damaged.

[0105] If the technician is not near the vehicle / is not actively working on the vehicle (e.g., as detected by a camera system), information can be projected onto the vehicle body or a surface near the vehicle to highlight details about the current vehicle—such as VIN, brand, model, year, etc. Additional projectable details include the current location during the repair process, a timeline of the schedule for upcoming stages, current touch time statistics, user statistics for different tasks performed (how long each task took to perform, etc.), or other similar information.

[0106] To issue invoices for products, technicians need to know the Repair Order Code (RO) of the vehicle being repaired, ensuring that the product / quantity / cost is correctly associated with the repair order. Repair orders can be managed through main store management software (such as that from 3M Company in St. Paul, Minnesota, USA). ® 3M RepairStack ™ ) to store or access. Using RO, technicians can more easily add detailed items to invoices.

[0107] The benefit of using the system described herein can be the ability to generate a planned repair process and provide it to technicians, allowing them to notice what may be needed. In some implementations, based on the repair plan, the main store management system can prompt technicians based on repair steps (e.g., the amount of main filler used for the "fill dent" step, or the estimated time required for the "joint seal weld line," or the windshield replacement step, etc.). Additional context surrounding the repair is helpful, particularly in determining the amount of material such as main filler, weld filler, etc.

[0108] In some implementations, system 810 projects details into a physical domain and can highlight areas of interest on the vehicle that need to be addressed as part of a repair task. For example, at the start of a current task, instruction blocks showing the step-by-step process of the task can be projected directly onto a surface on or near the vehicle (e.g., a floor, table, etc.).

[0109] In some implementations, camera 806 can detect a technician interacting with information projected by a projector and, in response, change the displayed content. For example, by touching a portion of a surface containing items on a menu, or an interface such as those presented herein, the defect tracking system can facilitate a technician manipulating the interface to navigate, for example, as a series of menus. Alternatively, the camera can locate an individual's hands in physical space and correlate those hands with the location of the displayed image to identify the desired action when the overlay is displayed. Similarly, a technician can navigate menus via voice commands. Furthermore, as sensor calibration systems become more sophisticated, the projection system can highlight the precise location relative to the vehicle to place necessary markers, light sources, or sensors for calibration. This reduces the burden on workers to accurately map these areas, as locations may vary between car manufacturers and vehicle models / years.

[0110] Figure 9A digital defect tracking system according to an embodiment of this document is illustrated. The defect tracking system 900 can be implemented according to the embodiments described herein. For example, the digital defect tracking system 900 can be used to capture, store, and present information about detected defects on a vehicle. These defects can be automatically detected by a computer vision system or manually detected and entered into the digital defect tracking system 900 by human technicians. The digital defect tracking system 900 can be accessed using a user interface on the screen of a device such as a mobile phone, tablet, or other computing device, or through an augmented reality-enabled device (allowing the user to view an AR overlay over the physical world), or through a projection system (allowing information to be presented on a work surface). In any of these embodiments, the presented information can be static or interactive, allowing the user to customize or select what they want to view.

[0111] The digital defect tracking system 900 includes or communicates with an imaging system 910. The imaging system may include devices for both capturing and displaying information. For example, the imaging system may include one or more image capture devices 902 (e.g., cameras, video cameras, etc.) for capturing images of the work area, including defect information, user information, and / or environmental information. The imaging system 910 may also include one or more projection devices 904. As used herein, the projection device 904 includes a conventional projector that projects images onto a surface, and a display that provides information via a graphical user interface. For example, in some embodiments, the projection device 904 is a projector capable of projecting onto a work surface—projecting onto a vehicle requiring repair, projecting onto a surface near the vehicle, etc. The projector 904 may project information interactively, for example, causing the image capture device 902 to capture information about the user's movement or voice and communicate this information to a surface analysis system 920, which then provides the new information to a graphical user interface generator 972 for presentation using the projector 904. However, it is also explicitly envisioned that the projection 904 could be presented on an augmented reality-enabled device. For example, information could be projected onto a transparent screen, such as a pair of glasses or a mask.

[0112] The imaging system 910 may include one or more light sources 906, each of which may have different light settings, such as intensity or color. The imaging system 910 may also include one or more movement mechanisms 908 responsible for moving any of the image capture device 902, projector 904, and / or light source 906. The movement mechanism 908 may include movement in any of the XYZ coordinate directions. For example, any of the image capture device 902, projector 904, or light source 906 may be able to rotate or swivel about a mounting, and / or move in three-dimensional space along a track system, or move more freely in three-dimensional space by being attached to a mobile robotic unit.

[0113] The controller 912 may, for example, control the operation of imaging system components 902 to 908, such as the capture rate of the image capture device 902, the light intensity or color of the light source 906, or the projection resolution 904. The controller 912 may also control one or more moving mechanisms 908 to correctly position and orient components 902 to 906.

[0114] The digital defect tracking system 900 is illustrated as encompassing an imaging system 910, a surface analysis system 920, a graphical user interface generator 972, and a database 980. However, it is explicitly contemplated that in some embodiments, at least some of these components may be remote from each other. For example, the database 980 may be accessed using wireless or cloud-based network protocols. Similarly, the digital defect tracking system 900 may include one or more processors remote from the imaging system 910, which perform the functions of the surface analysis system 920. Likewise, the GUI generator 972 may utilize processing circuitry or processing capabilities housed together with or separately from the surface analysis system 920 or the imaging system 910. In some embodiments herein, a communication component 914 is responsible for facilitating communication between the components of the digital defect tracking system 900, regardless of their location.

[0115] The digital defect tracking system 980 can access data of various types and sources, as illustrated by database 980. While database 980 is illustrated as a single data repository, it is explicitly contemplated that, in some embodiments, data in database 980 may be stored in multiple locations. Furthermore, while data in database 980 is illustrated as being split among data repositories 990, 982, and 985, it is explicitly contemplated that these divisions are purely for organizational understanding and are not intended to limit how data may be stored or organized in the embodiments described herein. Additionally, while database 980 is illustrated as including multiple different data useful to the embodiments described herein, the database is not intended to be an exhaustive list. In some embodiments, additional data 901 that may be useful in the embodiments described herein may also be accessible.

[0116] The detected defect data repository 990 can receive and store information about defects detected on one or more vehicles, including defect type 991, defect location 992 (which can be relative or absolute), repair plan 993 for the detected defect, repair status 994, and any other useful data 995. The defect information in the data repository 990 may also include defect information from historical repairs, such as defects resolved by a specific technician, the time and / or touches taken, the abrasive materials used, the tools used, etc. The defect information in the data repository 990 can be particularly useful for improving future repair operations and determining which technicians will benefit from additional mentoring.

[0117] The vehicle database 982 may include relevant information about vehicles that can be repaired or have already been repaired. For example, 3D models 983 of different vehicles can be particularly useful for locating defects on vehicles in real space. The 3D models 983 may be provided directly from the manufacturer or generated in other ways—for example, using computer vision systems, photogrammetry, or another suitable technique. Other vehicle information 984, such as previous repair history of similar vehicles or specific vehicles, may also be accessible.

[0118] The repair database 985 may include information about abrasive articles 986, including which are available and which have been used historically for similar repairs. Additionally, the abrasive article information may include the types and grit sizes of available abrasive articles 987. In some embodiments, wear levels or rates 988 are tracked for individual abrasive articles. The repair database 985 may also include information about available tools 989, including available support pads, operating angles, rotational speeds, etc. Optional repair plans or historical repair plans 999 may also be accessible.

[0119] The digital defect tracking system 900 may also include a surface analysis system 920 that provides analysis of the vehicle surface and / or ongoing repair processes based on information received from an imaging system 910 and a database 980. A defect marker receiver 960 can receive indications detected on the vehicle surface. Defect markers can be received via an input-output component (I / O component) 974. For example, a user can indicate a defect using a touchscreen of a mobile computing device, e.g., by pointing to the defect on an image or video feed. In other embodiments, a user can indicate a defect using an augmented reality-enabled device by physically touching the vehicle, pointing at the vehicle, or otherwise indicating the location of the defect. For systems using projection, an image capture device can capture gestures indicating a defect (e.g., touch or pointing). The defect marker receiver 960 can also receive defect markers by manual input, capturing auditory indications, or recognizing gestures.

[0120] The defect marker receiver 960 can receive indications of defect type 962 (e.g., bumps, paint stains, dents, etc.). It can also receive location 964. Location 964 can be a relative location (e.g., a point identified in a captured image) or an absolute location (e.g., a set of coordinates corresponding to a point on a vehicle). It can also retrieve defect status 966, such as requiring repair, repair in progress, or repair completed. Other defect markers 968 can also be retrieved.

[0121] The surface analysis system 920 may include a registration system 952. The registration system 952 analyzes received images or received image feeds and detects icons or otherwise identifies features in space. For example, identifiable curvature on a vehicle (such as tires, windshields, or headlights) may be easily identified by the registration system 952. Additionally, objects that may exist in real space (e.g., wrenches, sanders, transition zones between applied body filler and sanded metal, etc.) may also be identified using classic feature-based detectors or by using trained machine learning models. The registration system 952 may be able to generate a map of the space using a map generator 956 and / or generate the shape of the vehicle using a shape generator 954 based on the detected icons. In an embodiment where the vehicle specification retrieval unit 950 can retrieve a 3D model 983 of a given vehicle, the registration system 952 may identify sufficient features on the vehicle such that the map generator 956 can generate a map of the real space based solely on the 3D model. The registration system 952 may have other functionalities 958.

[0122] The vehicle surface mapper 970 can generate a defect map using images from the image capture device 902 and a map generated by the registration system 952, which associates detected defects with relative or absolute locations on the vehicle. In embodiments described herein, the surface map generated by the surface mapper 970 can be retrieved when the user begins, continues, or updates the defect status 966.

[0123] In some implementations, the surface analysis system 920 may also include a sanding evaluator 940. The sanding evaluator 940 can evaluate the current sanding operation, plan the next sanding operation, and / or provide real-time feedback to the user during the sanding operation.

[0124] The target contour retrieval unit 922 can retrieve target contours for repair. For example, the target contour retrieval unit 922 can retrieve data from the vehicle specification retrieval unit 950, or it can determine the correct curvature of the defective area based on an image from the image capture device 902. For example, the curvature of a portion of the vehicle opposite to the dented portion (e.g., the curvature of the driver's side door) may be a mirror image of the target curvature of the dented portion (e.g., the passenger side door).

[0125] The current profile retriever 924 can retrieve information about defective areas (e.g., the size of bumps or paint spots or information about the depth of dents) or information about the current surface during the repair process (e.g., after the first body filler is applied).

[0126] The sanding parameter generator 930 compares the current profile with the target profile and generates parameters for the next step. The parameters can be generated based on a retrieved repair plan obtained by a repair plan retrieval tool, which may include a target repair time and information about the required steps (e.g., body filler application, dent removal, and / or sanding).

[0127] The MRR characterizer 942 can calculate the material removal rate (MRR) required for the next sanding step. For example, based on time constraints, the next step may need to be completed within 30 minutes. Alternatively or additionally, the wear characterizer 942 can determine the actual material removal rate based on known parameters from previous steps. For example, a first material removal rate is predicted for the first step based on the selected abrasive article, tool speed, and applied force, and the sanding time is provided to the user. However, after this first step, it can be determined that significantly less material is removed than expected. The actual material removal rate of the first step can be determined based on the sanding time and the volume of material removed, and this actual material removal rate can then be used by the sanding parameter generator when generating the sanding parameters for the second sanding step. The sanding evaluator can also use the known heat capacity of the material (e.g., metal, body filler, etc.) to generate a heat map 944 of the work performed or in progress. The heat map can be used to generate and provide the user with warnings (audiovisual or tactile feedback) of potential burn-through. The sanding evaluator 940 may also have other functionalities 946.

[0128] The digital defect tracking system 900 also includes a GUI generator 972 that generates an interface for users to interact with the system 900. I / O components can receive or output feedback to the user. A defect overlay generator 978 can generate overlays for display on an image of space (as in an AR-enabled implementation), for display alongside an image of space (as in a user interface of a computing device), or for projection onto space using a projector. A GUI communicator 978 can communicate the generated user interface to the device for projection or display.

[0129] System 900 may also have other functionalities 916.

[0130] It should be clearly noted that throughout this specification, examples of surface modification of vehicle surfaces to remove defects related to paint are presented as a potential use case. However, other surface modifications are explicitly envisioned, such as other abrasive operations (sanding, grinding), other additive processes (e.g., additive manufacturing, binder deposition, etc.) or subtractive processes (material removal, cutting, etc.).

[0131] This paper describes a surface inspection system comprising image capture devices (such as cameras), one or more light sources, long-range sensors, etc. The systems and methods described herein are used to manage and perform the capture of said images and to process said images to obtain defect characterization information and surface characterization information. Systems and methods described herein are used to store and retrieve captured images, image metadata, defect detection and characterization results, and to manipulate said information to generate or improve repair strategies. Systems and methods described herein include devices for presenting information about surfaces, including mobile computing devices with displays, augmented reality-enabled devices, and projection systems. The systems described herein are explicitly envisioned to be interoperable with input / output components. Based on received input, the systems described herein are configured to change the presented information in real time.

[0132] The system and method described in this paper enable the coordination of surface repair operations in digital systems—recording defects, tracking the defect repair process, and providing feedback or instructions during the repair process.

[0133] However, it is explicitly envisioned that the systems and methods described herein can be applied to other industries. For example, while the vehicles and use cases described herein are envisioned to be repaired after a collision, OEM use cases are also envisioned to be relevant. Additionally, repeated or continuous evaluations can be performed on internal or external processes such as parts repair, evaluation of metal and / or paint finishes of other product groups, or even the use of mobile robots to perform high spatial resolution mapping of the environment.

[0134] Furthermore, it should be envisioned that the surface imaging system described herein can be used, for example, for other specular reflective surfaces, such as imaging the surface before and after the application of adhesive.

[0135] Figure 10 This is a block diagram of a repair policy generation architecture. The remote server architecture 1000 illustrates one implementation of a specific implementation of the repair policy generator 1010. As an example, the remote server architecture 1000 can provide computing, software, data access, and storage services without requiring the end user to know the physical location or configuration of the system delivering these services. In various implementations, the remote server can deliver these services over a wide area network (WAN), such as the Internet, using appropriate protocols. For example, the remote server can deliver applications over a WAN, and these applications can be accessed via a web browser or any other computing component. Figures 1 to 10The software or components shown or described, along with the corresponding data, may be stored on a server at a remote location. Computing resources in a remote server environment may be consolidated at a remote data center location, or they may be distributed. Remote server infrastructure can deliver services through a shared data center, even if they appear as a single access point for a user. Therefore, the components and functions described herein can be provided from a remote server at a remote location using a remote server architecture. Alternatively, they may be provided by a conventional server directly installed on the client device, or otherwise. Users can interact with system 810 using user interface 1022.

[0136] exist Figure 10 In the examples shown, some items are similar to those shown in the previous figures. Figure 10 Specifically, the digital defect tracking system can be located at a remote server location 1002. Therefore, the computing device 1010 accesses those systems via the remote server location 1002. The sensing system 1050 can also access the user interface 1022 using the computing device 1020.

[0137] Figure 10 It also depicts another example of a remote server architecture. Figure 10 It should also be understood that some components of the system described herein are located at a remote server location 1002, while other components are not located at that remote server location. By way of example, storage devices 1030, 1040, or 1060, or sensing system 1050, may be located at a location separate from location 1002 and accessed via a remote server at location 1002. Regardless of their location, they can be accessed directly by computing device 1020 via a network (WAN or LAN), hosted at a remote site, provided as a service, or accessed by a connectivity service residing at a remote location. Furthermore, data can be stored substantially anywhere and can be accessed intermittently by interested parties or forwarded to interested parties. For example, physical carriers may be used instead of electromagnetic carriers, or physical carriers may be used in addition to electromagnetic carriers.

[0138] It will also be noted that the components or portions thereof of the systems described herein can be mounted on a wide variety of different devices. Some of these devices include servers, desktop computers, laptop computers, embedded computers, industrial controllers, tablet computers, or other mobile devices such as handheld computers, cellular phones, smartphones, multimedia players, personal digital assistants, etc.

[0139] Figures 11 to 12 An example of a mobile device that can be used in the implementation shown in the previous figures is illustrated.

[0140] Figure 11 It is a handheld device 1121 that can be used as a user or client (e.g., Figure 13 A simplified block diagram of a handheld or mobile computing device (computing device 1320) in which the system (or a portion thereof) may be deployed. For example, a mobile device may be deployed in the operator compartment of computing device 1120 for generating, processing, or displaying data. Figure 12 This is another example of a handheld or mobile device.

[0141] Figure 11 A general block diagram of the components of a client device 1116, capable of running some of the components shown and described herein, is provided. The client device 1116 interacts with these components, or runs some of the components and interacts with others. In the device 1116, a communication link 1113 is provided that allows the handheld device to communicate with other computing devices and, in some embodiments, provides a channel for automatically receiving information, such as through scanning. Examples of the communication link 1113 include those allowing communication via one or more communication protocols, such as wireless services for providing cellular access to a network and protocols for providing local wireless connectivity to a network.

[0142] In other examples, the application may receive on a removable secure digital (SD) card connected to interface 1115. Interface 1115 and communication link 1113 communicate with processor 1117 (which may also embody a processor) along bus 1119, which is also connected to memory 1121 and input / output (I / O) components 1123, as well as clock 1125 and position system 1127.

[0143] In one embodiment, I / O component 1123 is provided to facilitate input and output operations, and device 1116 may include input components such as buttons, touch sensors, optical sensors, microphones, touchscreens, proximity sensors, accelerometers, orientation sensors, and output components such as display devices, speakers, and / or printer ports. Other I / O components 1123 may also be used.

[0144] Clock 1125 illustratively includes a real-time clock component that outputs the time and date. This clock also provides timing functionality for processor 1117.

[0145] Examplely, location system 1127 includes a component that outputs the current geographic location of device 1116. The location system may include, for example, a Global Positioning System (GPS) receiver, a LoRAN system, a dead reckoning system, a cellular triangulation system, or other positioning systems. The location system may also include, for example, mapping or navigation software that generates desired maps, navigation routes, and other geographic functions.

[0146] Memory 1121 stores operating system 1129, network settings 1131, applications 1133, application configuration settings 1135, data storage 1137, communication drivers 1139, and communication configuration settings 1141. Memory 1121 may include all types of tangible volatile and non-volatile computer-readable storage devices. The memory may also include computer storage media (described below). Memory 1121 stores computer-readable instructions that, when executed by processor 1117, cause the processor to perform computer-implemented steps or functions. Processor 1117 may also be activated by other components to facilitate the functionality of those components.

[0147] Figure 12 The device shown can be a smartphone 1271. The smartphone 1271 has a touch-sensitive display 1273 that displays icons or tiles or other user input mechanisms 1275. Mechanism 1275 can be used by the user to run applications, make phone calls, perform data transfer operations, etc. Generally, the smartphone 1271 is built on a mobile operating system and provides more advanced computing power and connectivity than non-smartphones.

[0148] It should be noted that other forms of device 1216 are possible.

[0149] Figure 13 This is a block diagram of a computing environment that can be used in the embodiments shown in the previous figures.

[0150] Figure 13 This is an example of a computing environment in which elements or parts of the systems and methods described herein can be deployed. References Figure 13 An example system for implementing some embodiments includes a general-purpose computing device in the form of a computer 1310. Components of computer 1310 may include, but are not limited to, a processing unit 1320 (which may include a processor), system memory 1330, and a system bus 1321 that couples various system components, including the system memory, to the processing unit 1320. System bus 1321 may be any of several types of bus architectures, including memory buses or memory controllers using any of a variety of bus architectures, peripheral buses, and local buses. The memory and programs described relative to the systems and methods described herein may be deployed on… Figure 13 In the corresponding part.

[0151] Computer 1310 typically includes a variety of computer-readable media. Computer-readable media can be any available medium accessible to computer 1310, and includes both volatile / non-volatile and removable / non-removable media. By way of example and without limitation, computer-readable media can include computer storage media and communication media. Computer storage media are distinct from and do not include modulated data signals or carriers. Computer storage media include hardware storage media, which includes volatile / non-volatile and removable / non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to computer 1310. Communication media can embody computer-readable instructions, data structures, program modules, or other data in a delivery mechanism and includes any information delivery medium. The term "modulated data signal" means a signal whose properties are set or altered in a way that encodes information in the signal.

[0152] System memory 1330 includes computer storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 1331 and random access memory (RAM) 1332. A basic input / output system 1333 (BIOS), containing basic routines (such as during startup) that facilitate the transfer of information between components within computer 1310, is typically stored in ROM 1331. RAM 1332 typically contains data and / or program modules that can be immediately accessed and / or are currently in operation by processing unit 1320. By way of example and without limitation, Figure 13 Examples include the operating system 1334, application program 1335, other program modules 1336, and program data 1337.

[0153] Computer 1310 may also include other removable / non-removable and volatile / non-volatile computer storage media. This is by way of example only. Figure 13 An example is a hard disk drive 1341 that reads from or writes to non-removable, non-volatile magnetic media, a non-volatile disk 1352, an optical disk drive 1355, and a non-volatile optical disk 1356. The hard disk drive 1341 is typically connected to the system bus 1321 via a non-removable memory interface (such as interface 1340), and the optical disk drive 1355 is typically connected to the system bus 1321 via a removable memory interface (such as interface 1350).

[0154] Alternatively or additionally, the functionality described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (e.g., ASICs), application-specific standard products (e.g., ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and the like.

[0155] The above discussion and Figure 13 The illustrated drives and associated computer storage media provide storage means for computer-readable instructions, data structures, program modules, and other data for computer 1310. Figure 13 For example, hard disk drive 1341 is exemplified as storing operating system 1344, application program 1345, other program modules 1346, and program data 1347. It should be noted that these components may be the same as or different from operating system 1334, application program 1335, other program modules 1336, and program data 1337.

[0156] Users can input commands and information into computer 1310 using input devices such as keyboard 1362, microphone 1363, and pointing devices 1361 (such as mouse, trackball, or touchpad). Other input devices (not shown) may include joysticks, gamepads, satellite receivers, or scanners. These and other input devices are typically connected to processing unit 1320 via user input interface 1360 coupled to the system bus, but may be connected via other interfaces and bus structures. Visual display 1391 or other types of display devices are also connected to system bus 1321 via an interface such as video interface 1390. In addition to a monitor, the computer may include other peripheral output devices, such as speakers 1397 and printer 1396, which may be connected via output peripheral interface 1395.

[0157] Computer 1310 operates in a networked environment using a logical connection (such as a local area network (LAN) or a wide area network (WAN)) with one or more remote computers (such as remote computer 1380).

[0158] When used in a LAN networking environment, computer 1310 connects to LAN 1371 via a network interface or adapter 1370. When used in a WAN networking environment, computer 1310 typically includes a modem 1372 or other components for establishing communication over a WAN 1373 (such as the Internet). In a networking environment, program modules may be stored in a remote memory storage device. Figure 13 For example, a remote application 1385 may reside on a remote computer 1380.

[0159] A defect tracking system includes: an image capture device configured to capture an image of a surface; a defect marker receiver configured to receive a defect indication, wherein the defect indication includes a defect on the surface; a defect mapper configured to map the defect onto the surface; and a user interface generator configured to generate a defect map for displaying the detected defect.

[0160] A display component is configured to display the defect map.

[0161] The system can be implemented such that the captured image includes the defect indication.

[0162] The system can be implemented such that an image analyzer is used to detect the defect indication, the image analyzer being configured to identify the defect indication.

[0163] The system can be implemented such that the defect flag receiver receives a user instruction indicating the defect.

[0164] The system can be implemented to receive the user's instruction via a touchscreen.

[0165] The system can be implemented such that the captured image includes an image captured by an augmented reality-enabled device, and wherein the defect indication is received through the augmented reality-enabled device.

[0166] The system can be implemented such that the defect indication includes the user touching the surface at or near the defect.

[0167] The system can be implemented such that the display component includes a projection system that projects an updated defect map onto the surface.

[0168] The system can be implemented such that the defect indication is received by a user-instructed receiver.

[0169] The system can be implemented such that the user instructs the receiver to include the image capture device.

[0170] The system can be implemented such that the image capturing device is a first image capturing device, and wherein the user indicates that the receiver is a second image capturing device.

[0171] The system can be implemented such that the user instructs the receiver to detect a defect detected by the user's instruction.

[0172] The system can be implemented such that the user indicates the receiver is a microphone, and wherein the defect indication is an audible defect indication.

[0173] The system can be implemented such that the defect indication includes the defect type, defect location, defect state, or surface profile at the defect.

[0174] The system can be implemented such that the defect indication includes its position relative to the surface.

[0175] The system can be implemented such that the defect indicates a location on the surface.

[0176] The system can be implemented such that the defect location includes the defect region on the surface.

[0177] The system can be implemented such that it includes a surface map retriever, wherein the surface map includes a three-dimensional (3D) model of the surface, and wherein the location includes a set of coordinates corresponding to points on the 3D model.

[0178] The system can be implemented such that it includes a registration system configured to identify icons in an environment based on an environmental image and to generate a registration map based on those icons.

[0179] The system can be implemented such that it includes a surface mapper configured to generate a surface map including the icon and the detected defects based on the defect map and the registration map.

[0180] The system can be implemented such that it includes a repair plan generator that generates a repair plan for removing the detected defect based on the defect indication.

[0181] The system can be implemented such that it includes a light source.

[0182] The system can be implemented such that it includes a light source controller configured to control the position and orientation of the light source.

[0183] The system can be implemented such that it includes a moving mechanism.

[0184] The system can be implemented such that it includes a moving mechanism for the image capture device.

[0185] The system can be implemented such that the defect map is a second defect map, wherein the second defect map is generated by updating the first defect map to include the mapped defect locations.

[0186] A system for mapping surface defects onto a vehicle is proposed. The system includes: a device configured to present a user interface; an environmental image capturing device configured to capture an image of the environment; a user interface generator configured to generate the user interface, wherein the user interface includes input / output components configured to receive a defect indication for a defect; and a defect mapper configured to associate the defect with a location in the image of the environment based on the defect indication. The user interface generator is configured to update the user interface in response to the associated defect location.

[0187] The system can be implemented such that it includes a registration system configured to detect features in the environmental image.

[0188] The system can be implemented such that the detected feature includes the curvature of the surface, which includes the defect.

[0189] The system can be implemented such that the feature is a first feature, wherein the registration system is configured to detect a second feature in the environmental image, and a registration map generator generates a registration map based on the detected first feature and the detected second feature.

[0190] The system can be implemented such that it includes a surface specification retriever configured to retrieve a 3D model of the surface, and wherein the defect mapper generates a defect map based on the 3D model and the registration map.

[0191] The system can be implemented such that the defect map is overlaid on the environmental image.

[0192] The system can be implemented such that the registration system uses a feature-based detection algorithm to detect the feature.

[0193] The system can be implemented such that the registration system uses a trained machine learning algorithm to detect the feature.

[0194] The system can be implemented such that the feature and the defect are different.

[0195] The system can be implemented such that the defect indication includes the defect type, defect location, and defect repair status.

[0196] The system can be implemented such that the defect indication is a defect repair status change, wherein the defect repair status change is a repaired indication indicating that the defect has been repaired, and wherein based on the repaired indication, the user interface generator updates the user interface to change the representation of the defect.

[0197] The system can be implemented such that the changed representation includes removing the defect from the user interface.

[0198] The system can be implemented such that the change in representation includes a change in the color of the defect on the user interface.

[0199] The system can be implemented such that the device is a mobile computing device.

[0200] The system can be implemented such that the device includes an image capture device.

[0201] The system can be implemented such that the image capturing device is configured to capture a surface image of the surface.

[0202] The system can be implemented such that the defect indication is detected from the surface image.

[0203] The system can be implemented such that the user interface is a graphical user interface, and wherein the input / output component is configured to receive the defect indication through the graphical user interface.

[0204] The system can be implemented such that the device is an augmented reality-enabled device.

[0205] The system can be implemented such that the device is a projection system.

[0206] The system can be implemented such that the projection system includes a camera that captures images of the environment.

[0207] The system can be implemented such that the projection system includes a light source with a moving mechanism configured to change the position or orientation of the light source.

[0208] A sanding parameter generation system is proposed, comprising: a target contour retrieval device configured to retrieve a target contour of a vehicle surface; a current contour retrieval device configured to retrieve a current contour of the vehicle surface; an image capture device configured to capture an image of the vehicle surface; a sanding parameter generator configured to generate parameters for a sanding operation on the vehicle surface based on the difference between the target contour and the current contour and the captured image; and a communication component configured to communicate the generated parameters.

[0209] The system can be implemented such that it includes: an image analyzer configured to determine the contact time between a tool and the vehicle surface based on a captured image of the vehicle surface; a work map generator configured to generate a work map of the surface; a burn-through indicator generator configured to generate a burn-through indicator based on the work map; and a graphical user interface generator configured to generate a user interface including the work map.

[0210] The system can be implemented such that the captured images are a series of captured images, wherein each captured image in the series is associated with a timestamp.

[0211] The system can be implemented to communicate the generated work to the user interface generator.

[0212] The system can be implemented such that the burn-through indication is communicated to the user interface generator.

[0213] The system can be implemented such that the working map includes the burn-through indication.

[0214] The system can be implemented such that the burn-through indicator indicates that more than 50% of the coating on the surface has been removed.

[0215] The system can be implemented such that the burn-through indicator indicates that more than 90% of the coating on the surface has been removed.

[0216] The system can be implemented to display the generated user interface on a mobile computing device.

[0217] The system can be implemented such that the generated user interface is displayed on a transparent surface via an augmented reality-enabled device.

[0218] The system can be implemented such that the generated user interface is provided to a projection system, which projects the user interface onto a flat surface.

[0219] The system can be implemented such that the generated user interface is presented to a projection system, which projects the user interface onto the surface of the vehicle.

[0220] The system can be implemented such that the target profile is retrieved from a 3D model of the vehicle, including the surface of the vehicle.

[0221] The system can be implemented such that the target profile is retrieved from an image of a vehicle, including the surface of the vehicle.

[0222] The system can be implemented such that the image includes the opposite side of the vehicle.

[0223] The system can be implemented such that it includes a material removal rate characterizer configured to calculate the material removal rate for the milling operation.

[0224] The system can be implemented such that the material removal rate is calculated based on the current profile and a previously recorded profile, and wherein the material removal rate is based on the detected volume of material removed from the previously recorded profile.

[0225] The system can be implemented such that the material removal rate is calculated based on the current profile and the target profile.

[0226] The system can be implemented such that the generated milling parameters are based on the calculated material removal rate.

[0227] The system can be implemented such that it includes a tool specification retriever configured to retrieve tool instructions for the sanding operation.

[0228] The system can be implemented such that the tool indicates a rotational speed.

[0229] The system can be implemented such that the tool indicates the orbital rotation speed.

[0230] The system can be implemented such that the tool indicates the rotational speed of a random orbit.

[0231] The system can be implemented such that the tool indicates the applied force.

[0232] The system can be implemented such that it includes a light source.

[0233] The system can be implemented such that it includes a light source moving mechanism configured to change the position or orientation of the light source.

[0234] The system can be implemented such that it includes a light source controller configured to change the color or intensity of the light source.

[0235] A method for repairing dents in a vehicle is proposed. The method includes: generating a target contour for a region of the vehicle including the dent; capturing an image of the region of the vehicle using an image capture device; generating a current contour for the region based on the image; and generating contour differences for the region based on the image. The contour differences include locations of excess filler material in the region. The method further includes: generating sanding parameters for a sanding operation on the region using a sanding parameter generator, wherein the sanding parameters are based on the contour differences; and generating and displaying a user interface including the sanding parameters using a user interface generator.

[0236] The method can be implemented such that it includes using a projection system to display the user interface.

[0237] The method can be implemented such that the projection system includes a projector and the image capture device.

[0238] This method can be implemented such that the user interface is projected onto the surface of the vehicle.

[0239] This method can be implemented such that the surface includes the region.

[0240] The method can be implemented such that the projection system includes a projector moving mechanism configured to change the positioning or orientation of the projector.

[0241] The method can be implemented such that the projection system includes an image capture device moving mechanism configured to change the positioning or orientation of the image capture device.

[0242] The method can be implemented such that it includes displaying the user interface on a device with a screen.

[0243] This method can be implemented to make the device a device that supports augmented reality.

[0244] The method can be implemented such that it includes generating a material removal rate for the sand milling operation.

[0245] The method can be implemented such that the material removal rate is generated based on the difference between the current profile and the previously captured profile.

[0246] This method can be implemented such that the material removal rate is generated based on the difference between the current contour and the target contour.

[0247] The method can be implemented such that it includes retrieving tool parameters associated with the tool used in the sanding operation, and wherein the material removal rate is generated based on the tool parameters.

[0248] The method can be implemented such that it includes retrieving abrasive article parameters associated with the abrasive article used in the sand milling operation, and wherein the material removal rate is generated based on the abrasive article parameters.

[0249] A method for repairing surface defects on a vehicle is proposed, comprising: receiving an indication of the surface defect using a defect indication retrieval device; receiving an image of the vehicle from a camera, wherein the image includes the surface defect; generating the location of the surface defect on the vehicle based on the received image; generating a defect map using a defect mapr, the defect map including the surface defect; and generating a user interface for display, the user interface including the defect map.

[0250] The method can be implemented such that it includes receiving an indication that the state of the surface defect has changed, and updating the defect map to indicate the new state of the surface defect.

[0251] The method can be implemented such that the user interface is a graphical user interface, and the method further includes displaying the graphical user interface on the device's display.

[0252] The method can be implemented such that it includes displaying the user interface on an augmented reality-enabled device.

[0253] The method can be implemented such that it includes using a projector to display the user interface.

[0254] The method can be implemented such that the defect indicator retriever includes a camera, wherein the method further includes capturing an image of the surface defect, and wherein the surface defect indicator is detected within the captured image.

[0255] The method can be implemented such that the indication of the surface defect includes: defect type, defect location, or defect state.

[0256] The method can be implemented such that receiving the instruction includes the camera capturing an image of the defect pointed to by the user.

[0257] The method can be implemented such that receiving the instruction includes detecting markings on the surface of the vehicle.

[0258] The method can be implemented such that receiving the instruction includes receiving user input through the user interface.

[0259] This method can be implemented such that the position is a relative position.

[0260] The method can be implemented such that it includes retrieving a 3D model of the vehicle.

[0261] This method can be implemented such that the location includes a set of coordinates corresponding to points on the 3D model.

[0262] A projection system for repairing a vehicle is proposed, comprising: a projector configured to project a user interface onto a surface; an image capture system configured to capture a first image containing an area of ​​the vehicle and a second image of the vehicle; a light source; and a defect tracking system. The defect tracking system includes: a defect indication receiver configured to detect defects on the vehicle based on the first image or the second image; a defect map generator configured to generate a defect map based on the second image, including the location of the detected defects relative to the vehicle; a user interface generator configured to generate the user interface including the defect map; and wherein the defect tracking system is configured to communicate the generated user interface to the projector.

[0263] The system can be implemented such that the image capture system is configured to capture a series of first images.

[0264] The system can be implemented such that it includes a feature detector configured to detect features in the region based on the first image and to generate a map of the region based on the detected features.

[0265] The system can be implemented such that each of the first images in the series of first images is analyzed by the feature detector.

[0266] The system can be implemented such that it includes a vehicle specification retriever that retrieves a 3D model of the vehicle; and wherein the generated map includes detected defects mapped to the 3D model.

[0267] The system can be implemented such that the user interface includes a menu of options for the user, and wherein the image capture system includes a camera configured to capture a user image of the user in the area, and wherein, based on analysis of the user image, the user interface generator modifies the user interface such that the projector projects the new user interface.

[0268] The system can be implemented such that the image capture system includes a camera configured to capture an image of a second region, wherein changes in defect state are detected based on analysis of the image of that region.

[0269] The system can be implemented such that the defect mapper generates a new defect map that includes the detected change in defect state.

[0270] The system can be implemented to perform real-time image analysis of the area, thereby updating the defect map in real time.

[0271] The system can be implemented such that the defect is a first defect, wherein the defect map indicates a second defect, and wherein the defect mapper generates the new defect map such that the second defect is indicated as unchanged.

[0272] The system can be implemented such that a first camera captures the first image, and a second camera captures the second image.

[0273] The system can be implemented such that detecting defects includes detecting the location pointed to by the user in the first image, and analyzing the second image to detect the defect.

[0274] The system can be implemented such that it includes assigning a first location to a location in the first image and assigning a second location to a detected defect.

[0275] The system can be implemented such that it includes a registration system configured to detect features in the first image and generate a region map based on the detected features; and a surface mapper configured to generate the coordinate locations of detected defects based on the region map and the second image.

[0276] The system can be implemented such that it includes a vehicle specification retriever configured to retrieve a 3D model of the vehicle, wherein the coordinate position corresponds to a point on the 3D model.

[0277] The system can be implemented such that it includes a light source and a light source controller, wherein the light source controller is configured to change the parameters of the light source based on the first image or the second image.

[0278] The system can be implemented such that the parameter is positioning, orientation, color, or intensity.

[0279] The system can be implemented such that it includes a microphone configured to receive audio signals, and wherein a new user interface is generated based on the audio signals.

Claims

1. A defect tracking system, the defect tracking system comprising: an image capture device configured to capture an image of a surface; a defect flag receiver configured to receive a defect indication, wherein the defect indication comprises a defect on the surface; a defect mapper configured to map the defect to the surface; a user interface generator configured to generate a defect map showing detected defects; and a display component configured to display the defect map.

2. The system of claim 1, wherein the captured image comprises the defect indication.

3. The system of claim 2, wherein the defect indication is detected using an image analyzer configured to identify the defect indication.

4. The system of claim 2, wherein the defect flag receiver receives a user indication of the defect indication.

5. The system of claim 1, wherein the display component comprises a projection system that projects an updated defect map on the surface.

6. The system of claim 5, wherein the defect indication is received by a user indication receiver.

7. The system of claim 6, wherein the user indication receiver comprises the image capture device.

8. The system of claim 6, wherein the image capture device is a first image capture device, and wherein the user indication receiver is a second image capture device.

9. The system of claim 1, wherein the defect indication comprises a defect type, a defect location, a defect state, or a surface profile at the defect.

10. The system of claim 9, wherein the defect indication comprises a location relative to the surface.

11. The system of claim 9, wherein the defect indication comprises a location on the surface.

12. The system of claim 9, wherein the defect location comprises a defect area on the surface.

13. The system of claim 1, and further comprising a registration system configured to identify a landmark in the environment based on an environmental image, and generate a registration map based on the landmark.

14. The system of claim 1, and further comprising a repair plan generator that generates a repair plan for removing the detected defect based on the defect indication.

15. The system of claim 1, wherein the defect map is a second defect map, and wherein the second defect map is generated by updating a first defect map to include the mapped defect location.

16. A system for mapping surface defects on a vehicle, the system comprising: a device configured to present a user interface; an environmental image capture device configured to capture an image of an environment; a defect flag receiver configured to receive a defect indication, wherein the defect indication comprises a defect on the surface; a defect mapper configured to map the defect to the surface; a user interface generator configured to generate a defect map showing detected defects; and a display component configured to display the defect map. a user interface generator configured to generate the user interface, wherein the user interface includes an input / output component configured to receive a defect indication for a defect; a defect mapper configured to associate the defect with a location in the image of the environment based on the defect indication; and wherein the user interface generator is configured to update the user interface in response to the associated defect location.

17. The system of claim 16, and further comprising a registration system configured to detect a feature in the environment image.

18. The system of claim 17, wherein the feature is a first feature, wherein the registration system is configured to detect a second feature in the environment image, and based on the detected first feature and the detected second feature, a registration map generator generates a registration map.

19. The system of claim 17, wherein the registration system uses a feature-based detection algorithm to detect the feature.

20. The system of claim 17, wherein the feature and the defect are different.

21. The system of claim 16, wherein the defect indication includes a defect type, a defect location, a defect repair status.

22. The system of claim 16, wherein the defect indication is a defect repair status change, and wherein the defect repair status change is a repaired indication indicating that the defect has been repaired, and wherein based on the repaired indication, the user interface generator updates the user interface to change a representation of the defect.

23. The system of claim 22, wherein the changed representation includes removing the defect from the user interface.

24. A sanding parameter generation system, the sanding parameter generation system comprising: a target profile retriever configured to retrieve a target profile of a vehicle surface; a current profile retriever configured to retrieve a current profile of the vehicle surface; an image capture device configured to capture an image of the vehicle surface; a sanding parameter generator configured to generate parameters for a sanding operation on the vehicle surface based on a difference between the target profile and the current profile and the captured image; and a communication component configured to communicate the generated parameters.

25. The system of claim 24, and further comprising: an image analyzer configured to determine a contact time between a tool and the vehicle surface based on the captured image of the vehicle surface; a work map generator configured to generate a work map of the surface; a burn through indication generator configured to generate a burn through indication based on the work map; and a graphical user interface generator configured to generate a user interface including the work map.

26. The system of claim 25, wherein the captured image is a series of captured images, wherein each of the captured images of the series of captured images is associated with a time stamp.

27. The system of claim 25, wherein the burn-through indication is communicated to a user interface generator.

28. The system of claim 27, wherein the work map includes the burn-through indication.

29. The system of claim 24, wherein the target profile is retrieved from an image of a vehicle including the vehicle surface.

30. The system of claim 29, wherein the image includes opposite sides of the vehicle.

31. The system of claim 24, and further comprising a material removal rate characterizer configured to calculate a material removal rate for the sanding operation.

32. The system of claim 31, wherein the material removal rate is calculated based on the current profile and a previously recorded profile, and wherein the material removal rate is based on a detected volume of material removed from the previously recorded profile.

33. The system of claim 32, wherein the material removal rate is calculated based on the current profile and a target profile.

34. The system of claim 32, wherein generated sanding parameters are based on the calculated material removal rate.

35. The system of claim 32, and further comprising a tool specification retriever configured to retrieve a tool indication for the sanding operation.

36. The system of claim 32, wherein the tool indication includes a rotational speed, an orbital rotational speed, or a random orbital rotational speed.

37. The system of claim 32, wherein the tool indication includes an applied force.

38. The system of claim 24, and further comprising a light source.

39. The system of claim 38, and further comprising a light source movement mechanism configured to change a positioning or orientation of the light source.

40. The system of claim 38, and further comprising a light source controller configured to change a color or intensity of the light source.

Images