Defect mapping and repairing system and method

Abstract

A system (100) for identifying and repairing one or more defects on a surface of an object (102). The system may have a robotic paint repair apparatus (110) having a robotic arm (114) and a tool (120, 122) mounted to the robotic arm. The tool may remove one or more defects. The system may have a camera (124) positioned within the repair area adjacent the robotic paint repair device. The camera is configured to scan a portion of an object surface having at least one of the one or more defects and collect scan data. The system may have a controller configured to control the camera to scan an area of the surface based on first data collected at a different location than the repair area that represents a location of one or more defects on the surface of the object.

Description

[0001] Priority declaration

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 363,272, filed on April 20, 2022, the contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an object inspection system and a method for inspecting an object, and more particularly, to an object inspection and repair system and method for identifying defects at locations where a robotically implemented repair is performed using one or more surface modification tools. Background Art

[0004] The automotive industry often needs to prepare the surface of a vehicle part or replacement part (e.g., a bumper) for various purposes (e.g., painting), or to repair the surface of an automotive part or replacement part due to defects caused during the painting or coating process. Typical surface preparation and repair processes include, for example, physical surface modification of the vehicle surface, such as sanding and polishing. Surface preparation and repair of surface defects can be performed using different tools, materials, and fluids.

[0005] In the automotive industry (e.g., automotive original equipment manufacturing (OEM) and aftermarket sectors), clear coat repair of mirror surfaces has not been automated. Technology is needed to automate this process as well as other paint applications (e.g., primer sanding, clear coat defect removal, clear coat polishing, etc.) suitable for inspection and repair using abrasives and / or robots.

[0006] In terms of object inspection, objects manufactured in the past are usually visually inspected by personnel to detect flaws, defects or other unwanted features on the corresponding surfaces of these objects. Visual inspection by personnel is costly (e.g., personnel need to be paid to visually inspect the objects produced), may cause worker fatigue due to repeated manual inspections and repairs, and is less reliable because the detection rate depends on the various visual abilities of the inspectors. To address these problems, electronic systems have been implemented that use cameras, light sources and computers to capture images of objects and use the images for analysis to detect defects including unwanted surface features. However, these systems perform inspections before and at locations different from the repair location. From the perspective of identifying defects on vehicles through these systems, errors are introduced in determining the location of these defects due to the movement of the vehicle on the bracket, track or assembly line structure and the vibration and interaction with the repair tool. This error can cause the area of ​​the vehicle surface to be repaired to be larger than expected. These larger-than-expected areas can lead to higher costs, slower repair times and other disadvantages as further discussed herein. Summary of the invention

[0007] Various examples will now be described to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0008] The present disclosure describes systems, devices, methods, and techniques related to various problems in identifying position errors and automated defect-specific repair (such as for paint or other applications). The current process of manually inspecting and / or repairing workpieces is very time-consuming. As described above, the disadvantage of electronically implemented defect recognition systems is that position errors are introduced from the time the defect location is determined until the defect is repaired. The inventors of the present invention have invented systems, devices, methods, and techniques that allow automated defect recognition of object surfaces and more accurate automated repair of defects on object surfaces. In addition, the inventors of the present invention recognize that scanning at the repair point can collect data that can be used for other purposes, such as improving the accuracy of the automated defect recognition system (e.g., improving the understanding of the defect location and / or improving the understanding of one or more features of the defect, such as defect type, shape, or even defect absence (false confirmation)). In addition, the inventors of the present invention recognize that scanning at the repair point can collect data about a defect (such as location), which can be extrapolated as a whole to other defects on the object. This can reduce the need for repair point scanning for each individual defect on the object. Instead, the coordinates of the various defects collected during the initial scan of the object can be re-determined at the repair point, and this more accurate data can be used to perform the repair. In addition, these various systems, devices, methods, and techniques can reduce repair time, reduce tool wear and other waste, improve aesthetics, and increase production line yields. Thus, a variety of recognized and unrecognized benefits can be obtained.

[0009] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram showing an assembly line with a painted vehicle, a defect scanning system, and a robotic repair device according to one example of the present application.

[0011] Figure 1A is an enlarged view of an end effector portion of a robotic repair device according to one example of the present application, showing one or more tools and a camera.

[0012] Figure 2is a schematic diagram of a system according to one example of the present application, the system including a controller, a robotic repair system with visual inspection capability, and an initial inspection system.

[0013] Figure 3 is with Figure 1 A perspective view of a conveyor assembly of an assembly line incorporating a tracked object and further illustrating a world coordinate system used according to one example of the present application.

[0014] Figure 4 is a schematic diagram illustrating an example system for robotic paint repair using a paint repair robot manipulating one or more surface modification tools and a second robot having a camera according to one example of the present application.

[0015] FIG. 5A to FIG. 5B is a schematic diagram of a method in which a position error is introduced in locating a defect due to movement of an object in which the defect is located from a first position to a second position, and a surface modification is performed on the repair area to resolve the defect, Figure 5C Further illustrated is the use of systems, devices, methods, and techniques according to one example of the present application to reduce the repair area.

[0016] Figure 5D It is shown that according to one example of the present application, the localization of a defect can be used for global realignment of other defects.

[0017] FIG. 6A to FIG. 6C A schematic diagram of a method in which another position error is introduced due to vibration of the object when locating a second defect, and the second defect and vibration dynamics are analyzed using a system, device, method and technique according to an example of the present application to better address defect repair.

[0018] Figure 7 is a schematic diagram of an example robotic repair system according to the present application, the robotic repair system including a portion of defect repair robots, at least one defect repair robot having a camera, wherein at least one defect repair robot scans a portion of a vehicle chassis while other defect repair robots manipulate sanding tools and polishing tools.

[0019] Figure 8 The present invention is a flowchart of a method for performing automated defect identification and automated defect repair on an object according to an example of the present application.

[0020] In the accompanying drawings, the same reference numerals represent the same elements. Although the above-mentioned drawings, which may not be drawn to scale, set forth various embodiments of the present disclosure, other embodiments may also be envisioned as described in the detailed description. In all cases, the present disclosure describes the present disclosure by representation of exemplary embodiments rather than explicit limitations. It should be understood that those skilled in the art may devise many other variations and embodiments that fall within the scope and spirit of the present disclosure. DETAILED DESCRIPTION

[0021] The present disclosure provides an automated system and method that uses defect inspection for robotically implemented repairs that employ an end-of-arm system equipped with tools for surface modification of an object surface. The automated system and method also allow for end-of-arm defect identification using one or more cameras. As an example, the one or more cameras may be mounted approximately vertically, offset, or arranged in another manner relative to the one or more tools performing the surface modification. The use of one or more cameras for defect inspection at the repair location provides the various benefits discussed previously, including reduced processing time because the size of the area covered by the repair can be reduced. The present application also recognizes other benefits, such as improved aesthetics, reduced waste, reduced tool wear, and data collection that can be used for data analysis improvements, etc. In addition, in some cases, damage to the vehicle or end-of-arm system can be avoided by using information collected by the system to avoid repairs that the system is not suitable for making (e.g., locations where the tool may inadvertently affect nearby vehicle surfaces, or locations where there is not actually a defect).

[0022] One or more cameras may be mounted on an end effector at the end of a movable robotic arm (with or without an associated tool assembly) such that the one or more cameras can be moved relative to an object (e.g., approaching a portion to zoom and increase resolution). The surface modification tool may include functional components configured to contact and prepare the surface of an object, and one or more sensors and / or actuators configured to detect operating state information of the end effector tool and / or change the operating state of the end effector tool. The various sensors and / or actuators may include one or more force sensors, one or more force-torque sensors, one or more force control units, etc.

[0023] It should be understood that although an illustrative implementation of one or more embodiments is provided below, any number of techniques (whether currently known or not yet in existence) may be used to implement the invention. Figures 1 to 8The disclosed systems and / or methods described herein should in no way be limited to the exemplary implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the full scope of the appended claims and their equivalents.

[0024] The term "vehicle" as used herein is not limited to automobiles, but includes cars, trucks, boats, airplanes, helicopters, buses, or other means of transportation. The term "defect" as used herein refers to an unwanted feature, such as a blemish or an unwanted surface condition. The term "defect" may include, but is not limited to, hair or other fibers, dust or other particles in / on a painted surface; areas on a painted surface where the coating is missing, such as scratches, dents, or other types of depressions; and protruding, raised, or convex portions of a painted surface; and / or undesirable color areas, such as different colors where the paint was not properly applied or properly cured. Examples of defects include: microbubbles—tiny sub-millimeter areas where solvent bursts through the varnish coating; orange peel—uneven layers of paint; mars—areas where the vehicle is touched; light paint, etc. The terms "board," "processor," "processing component," and "server" (and other descriptive names used) may be used interchangeably, and these terms generally refer to some processor-based entity without limiting the entity referred to to any particular hardware or software configuration. The term "fluid" as used herein refers to any one or combination of a pure fluid, a fluid combined with particles (such as a slurry, fragments from surface modification, and the like). The term "surface modification" and the like include surface repair, grinding, scraping (scuffing), sanding, polishing, buffing, and the like. The term "substantially (or approximately)" means a difference of up to but not exceeding 15% from the amount or value provided, including 15% of the present value (for example, in terms of perfect parallelism, between the exact value amount and the value of 15 degrees (including this value)). The term "surface" is not limited to mirror surfaces, but can include matte, metal or other types of finishes. The term "surface" does not necessarily mean a painted surface in all cases.

[0025] In one embodiment, the functions or algorithms described herein can be implemented with software. Software may include computer executable instructions stored on a computer-readable medium or a computer-readable storage device, such as one or more non-transitory memories or other types of hardware-based storage devices, which may be local or networked. In addition, these functions correspond to modules, which may be software, hardware, firmware, or any combination thereof. Multiple functions may be performed in one or more modules as needed, and the described embodiments are merely exemplary. Software may be executed on a digital signal processor, an ASIC, a microprocessor, or other types of processors running on a computer system (such as a personal computer, a server, or other computer system), thereby transforming such a computer system into a specially programmed machine.

[0026] Figure 1 1 is a schematic diagram showing an assembly line system 100 having a paint vehicle 102, a carriage 104, a track system 106, a defect identification system 108, and a robotic repair device 110. The robotic repair device 110 may have a base 112, an arm 114, an end effector 116, and an assembly 118. Figure 1A As shown, assembly 118 may include a first tool 120 , a second tool 122 , one or more cameras 124 , and a light source 126 .

[0027] Figure 1 A painted vehicle 102 is shown mounted to a carriage 104 in a known manner. The carriage 104 may be coupled to a track system 106 to transport the painted vehicle along the assembly line system 100 as indicated by arrow A. Although Figure 1 Although a vehicle body or chassis is shown in FIG, it should be appreciated that the systems, processes, techniques, and apparatuses according to the present disclosure may be used for any object (e.g., bumper, hub cap), and are not limited to the automotive field. Figure 1 The assembly line system 100 is depicted as a continuous process, but it should be understood that the system, process, technique and equipment may also be used for a non-continuous process, such as performing a portion of the assembly at one location, then pausing manufacturing, then moving the vehicle 102 to another location (such as another facility), and performing the other portion of the assembly at another location. The track system 106 is purely exemplary and may include various types of transport mechanisms. The track system need not be continuously movable, but may be a docking station, a diversion station, or other types of configurations known in the art. In some embodiments, the carriage 104 may not be coupled to the track system 106.

[0028] After baking to cure the paint, the painted vehicle 102 (sometimes referred to as a chassis, a body, or simply an object) mounted on the bracket 104 can enter the defect identification system 108. According to some examples, the defect identification system 108 can perform a global scan of all visible outward surfaces of the painted vehicle 102 that may have defects. However, according to further examples, partial scans can be performed on only some surfaces. Figure 2 and Figure 3 , discusses various aspects of the defect identification system 108 in further detail. The defect identification system 108 may be constructed and operated in one or more of the manners described in U.S. Patent Application No. 15 / 932,865 filed on May 9, 2018 and U.S. Patent Application No. 16 / 866,110 filed on May 4, 2020, the entire disclosure of each of which is incorporated herein by reference.

[0029] The paint vehicle 102 can travel from the defect identification system 108 along the track system 106 to a defect repair location 128 (also sometimes referred to herein as a defect repair area or a second location). Robotic repairs can be performed at the defect repair location 128 on one or more defects on the paint vehicle 102 by a robotic repair device 110 located within the defect repair location 128.

[0030] The base 112 may be coupled to an arm 114 of the robotic repair device 110. Figure 1 As shown, the robotic arm 114 is movable in any of six dimensions relative to the base 112, with the ability to perform translation or rotation about the x-axis, y-axis, and / or z-axis. The robotic repair device 110 may have a force control unit (discussed later) and an end effector 116 with an assembly 118 mounted thereon.

[0031] Now refer to Figure 1A, the first tool 120 and the second tool 122 can be configured to selectively interact with a surface (such as the surface of the painted vehicle 102). In one embodiment, the first tool 120 can be a support pad configured to hold an abrasive for sanding, grinding, etc., or another suitable abrasive tool. During the grinding operation, the first tool 120 can grind the surface of the painted vehicle 102 with the aid of a grinding disc 130 or other suitable abrasive article to remove material. The first tool 120 can be attached to the assembly 118 using an adhesive, a hook and loop, a clip system, a vacuum, or other suitable attachment system. Similarly, the second tool 122 can be a second grinding tool, such as an abrasive pad 132, which is configured to polish or buff the surface of the painted vehicle 102. However, according to further examples, the second tool 122 can have other configurations known in the art, such as wiping media, finer grain sanding tools, fluid removal tools (vacuum or air knife), etc. The second tool 122 can be attached to the assembly 118 using an adhesive, a hook and loop, a clip system, a vacuum, or other suitable attachment system.

[0032] The assembly 118 can be configured such that the first tool 120 and the second tool 122 share substantially parallel (and in fact substantially aligned) actuation axes A1 and B1, respectively. In other words, the first tool 120 can have a first axis A1 about which the first tool 120 is configured to rotate to perform surface modification of a workpiece. The second tool 122 can have a second axis B1 about which the second tool 122 is configured to rotate. The first axis A1 and the second axis B1 can be in Figure 1 However, the first tool 120 may be located on the opposite side of the assembly 118 from the second tool 122. The first tool 120 and the second tool 122 may be any one or a combination of linear, rotary, orbital or random orbital devices.

[0033] In a paint or varnish coating repair process, a fluid may be dispensed onto a workpiece before, during, or after using the first tool 120 or the second tool 122. Such process fluid may be combined with particulate matter from the process to produce a slurry. The particulate matter that constitutes such a slurry is typically caused by a sanding process, which typically occurs prior to a polishing or buffing step (e.g., using the second tool 122). One or more wiping tools or utensils (not shown) may be used as needed to remove the slurry and / or excess liquid.

[0034] One or more cameras 124 may be mounted on the assembly 118 adjacent to the first tool 120 and the second tool 122. The one or more cameras 124 may have an axis C1 (through the center of the one or more lenses) that may be arranged substantially perpendicular to the first axis A1 and the second axis B1. However, one or more mirrors may be employed so that the one or more cameras 124 may be arranged in any desired manner relative to the first tool 120 and the second tool 122. It is desirable to mount the one or more cameras 124 on the assembly 118 near the center of gravity of the assembly 118 because this may reduce the possibility of obstruction and / or undesirable vibration of the one or more cameras 124. The assembly 118 may be pivoted as needed so that the first tool 120, the second tool 122, or one of the one or more cameras 124 is in an interfacing relationship with the surface of the painted vehicle 102. With respect to the one or more cameras 124, the one or more cameras 124 may be positioned as needed by the robotic repair apparatus 110 to scan the surface of the painted vehicle 102 in a desired area, as discussed further.

[0035] For example, one or more cameras 124 may be a high-resolution (e.g., greater than 12MP) digital camera. According to some examples, one or more cameras 124 may have a zoom lens. However, a zoom lens is not required in all examples because, according to some examples, the robotic repair device 110 can move the position of one or more cameras 124 as needed. Thus, the robotic repair device 110 can move one or more cameras 124 toward or away from the surface of the painted vehicle 102 as needed. According to one example, one or more cameras 124 can be configured to focus on an area of ​​approximately 150mm×150mm at a distance of approximately 1m. However, other area sizes and distances are also contemplated, and the areas and distances provided above are for exemplary purposes. Defects on the painted vehicle 102 are typically less than 1mm 2 Various criteria, such as resolution, zoom capability, distance, area, etc., may be manipulated as needed to achieve the desired result of identifying the presence of one or more defects on the surface of the painted vehicle 102 using the one or more cameras 124 .

[0036] The light source 126 can be mounted to the assembly 118 adjacent to the one or more cameras 124. According to some examples, the light source 126 can be a plain white light. However, the characteristics of the light source 126, such as size, color, position relative to the one or more cameras 124, etc., can be modified as desired as discussed in U.S. Patent Application Nos. 15 / 932,865 and 16 / 866,110.

[0037] One or more cameras 124 and light sources 126 are arranged in Figure 1A120 and / or the second tool 122. Other arrangements and positioning relative to the first tool 120 and / or the second tool 122 are also contemplated. For example, it is also contemplated that one or more cameras 124 and / or light sources 126 can be mounted to a gantry system or other features coupled to the robotic repair device 110. Thus, in some arrangements, one or more cameras 124 and / or light sources 126 can be carried, bypassing the assembly 118 and / or the end effector 116 and not requiring coupling to the assembly 118 and / or the end effector 116. However, in such an arrangement, the gantry system or other features can still be manipulated to move with the movement of the arm 114.

[0038] By mounting to the assembly 118, the end effector 116, the force control unit (discussed later), and the arm 114, the first tool 120, the second tool 122, and the one or more cameras 124 can be positioned within the degrees of freedom provided by the robotic repair device 110 (in most cases, 6 degrees of freedom), and can be positioned within any other degrees of freedom (e.g., the compliant force control unit) within its reference frame. This arrangement allows the first tool 120, the second tool 122, and the one or more cameras 124 to be positioned as needed to perform repairs and imaging. The one or more cameras 124 and the light source 126 can also be manipulated to sweep across the surface of the painted vehicle 102 to collect images from multiple locations, as is known in defect identification systems such as U.S. Patent Application Nos. 15 / 932,865 and 16 / 866,110.

[0039] Figure 2 A schematic diagram of a system 200 including a controller 202, a defect identification system 108, and a robotic repair apparatus 110 is shown according to one example. The controller 202 may be in electronic communication with the defect identification system 108 and the robotic repair apparatus 110.

[0040] As previously described, the defect recognition system 108 may be configured to detect defects such as the painted vehicle 102 ( Figure 1 ) on the surface of an object. Figure 2 As shown, the defect identification system 108 may include a first plurality of light sources 204 arranged along Figure 1 The carrier (and Figure 1The first plurality of light sources 204 are placed along a path or direction along which objects to be produced are conveyed by a conveyor (e.g., a conveyor). Each of the first plurality of light sources 204 can be individually configured to be selectively activated or energized, respectively, and thereafter selectively and controllably emit light energy. In a non-limiting example, the first plurality of light sources 204 each include a light emitting diode type of light source, but other types of light sources can also be utilized. The first plurality of light sources 204 can be distributed around the conveyor or moving component to effectively generate a substantially uniform amount of light around and on the object as the object moves along the path or direction. The first plurality of light sources 204 can be arranged to generate a substantially uniform amount of intensity on and around the object along the path or direction as the object moves.

[0041] The defect identification system 108 may also include a plurality of inspection cameras 206, which are also placed along the path or direction of the carrier conveying the object 14. The plurality of inspection cameras 206 can cooperatively receive reflected light energy reflected from the surface of the object. Each camera in the plurality of inspection cameras 206 can be selectively powered on and selectively activated once powered on.

[0042] The reflected light energy collected by the plurality of inspection cameras 206 includes first data or image data (image information) about surface features (e.g., visual features). The first data can be used to detect defects on the surface of the object and for other purposes discussed herein, including guiding one or more cameras 206 ( Figure 1A ) to scan a specific portion of the surface of the object. The defect identification system 108 having multiple inspection cameras 206 and light sources 204 can be arranged to capture first data about substantially the entire visible surface of the object (eg, the exterior surface of a vehicle).

[0043] The defect identification system 108 may have a dedicated processing component 208 , which may include several different computer processors operating under stored program control, or a single computer processor component. According to a further example, the processing component 208 may be a component of the controller 202 .

[0044] The processing component 208 may have many features or components not specifically shown. These may include, for example, an image tracking server or processor, a post-processing server or processor, a "NAS" or archive server or processor, a trigger board, and an encoder. The processing component may also include a simulator 210.

[0045] The encoder can be communicatively coupled to the tracking server. In one non-limiting embodiment, the encoder can include a commercially available friction wheel encoder manufactured and sold by Edon Controls, Inc. of Troy, Michigan. Other types of position encoders can be used. The encoder can be movably coupled to the conveyor or moving assembly and frictionally engaged to the carrier and rotate (e.g., rotate) as the carrier moves along the conveyor or moving assembly. This rotation can provide continuous information about the position of the carrier to the processing component 208, thereby providing continuous information about the position of the object along the path or direction. The simulator 210 can include a Simulink processor. Tools are commercially available Simulator 210 can be a processing system that is separate and distinct from processing component 208. In some examples, simulator 210 can be communicatively coupled to a computer system and monitor that is remote from processing component 208. Thus, in some examples, simulator 210 can be communicatively coupled to controller 202 in a direct manner.

[0046] Tracking can also be achieved using a camera-based visual tracking system, and / or tracking can be achieved using a 3D camera instead of an encoder. For a camera-based vision system, there is a visual tracking server that sends position data to the trigger board.

[0047] The processing component 208 can be electronically coupled to an output monitor and / or display component 212. The output monitor and / or display component 212 can include, or be a portion of, a display computer portion operating under stored program control. The output monitor and / or display component 212 can include multiple display computer portions. The processing component 208 can be electronically coupled to each of the first plurality of cameras 206, the one or more tracking cameras 214, and the one or more high-speed cameras 216.

[0048] The image processing server or processor may be communicatively coupled to the image capture server or processor. The post-processing server or processor may be communicatively coupled to the image processing server or processor. The post-processing server or processor may be communicatively coupled to a "NAS" or archive server or processor. The trigger board, image server, image processing server, post-processing server, NAS, display computer portion, and tracking server may each be connected to a communication network (such as, but not limited to, Ethernet) via a switch and thereby selectively communicate with each other via the network. The first plurality of light sources 204 may also be connected to the network. Each of the plurality of cameras 206 and the plurality of light sources 204 may be individually and selectively "powered on" or "activated" upon receiving a command from the trigger board or server.

[0049] Services such as windows services (e.g., standalone programs) are also contemplated. Example services are: PLC service - communicates with the PLC to obtain factory data about the vehicle; Tracking service - communicates with the trigger board and other services to coordinate the scanning process; Image capture service - acquires images from the camera, Image processing or GPU service - performs image processing steps to find defective areas of interest in the captured frames; Classification service - a neural network classifies the found areas; Clustering service - locates the found areas on the 3D surface and clusters multiple images of the same defect on the surface, determines the defect size, and ultimately the defect type, and then creates images and data about the found defects. These services also include reporting services and projection display services (for displaying images on output monitors). These services are distributed on the servers in various configurations.

[0050] The trigger board can be loaded with a table to plot the vehicle position that triggers the camera and light source during a scan. The trigger board can input the vehicle position from an encoder (or visual tracking system) and then trigger the camera and light source at the specified position. The trigger board can also have inputs from or to the photoeye that are used to resynchronize to a predetermined tracking synchronization position when the vehicle interrupts the photoeye. The trigger board can use the position information from the encoder to determine the identity and order of the light sources from the first plurality of light sources 204 to illuminate and the identity and order of the first plurality of cameras 206 to activate. Essentially, the raw image is captured along the surface of the object as the object moves along the direction or path. The raw captured image data can be transmitted to a processing component 208 (such as to an image capture server). The raw captured image data can then be transmitted to other components, such as an image processing server, controller 202, etc., for analysis.

[0051] It is also contemplated that the controller 202 and / or the processing component 208 (such as through an image processing server and / or a post-processing server) can execute a selected sequence of image processing algorithms that work together to effectively produce first data (sometimes referred to herein as first scan data) that includes information such as a processed image of each original captured image data (from the received original image). The processed image (first data or first scan data) can contain one or more areas of interest on the surface, or can include the entire visible surface. The first data can additionally include information related to one or more defects. Analysis can be performed to determine the identity (one or more characteristics) and location of a corresponding defect in the one or more defects on the surface. The first data can then be transmitted to the controller 202, a storage medium (e.g., a NAS server) and utilized as further discussed herein. The display of the first data (and indeed the scan data discussed subsequently) can occur in "real time", near "real time" (within a delay of less than 4 seconds), or can be retrieved from the storage medium for review.

[0052] The defect identification system 108 may also include one or more tracking cameras 214, which may be coupled to a tracking server (component of the processing component 208). These one or more tracking cameras 214 (and / or one or more high-speed cameras 216) may cooperatively provide position information to the tracking server, which is related to the location or position of the object to be inspected that is moved due to the movement of the conveyor or moving component. The one or more tracking cameras 214 and / or high-speed cameras 216 may supplement or replace the encoder. The one or more tracking cameras 214 and / or high-speed cameras 216 may be coupled to the trigger board and the processing component 208 (such as the tracking server). The one or more tracking cameras 214 may collect object position information along a path or direction for the trigger board, and such information may be used alone or in combination with position information from the encoder. The one or more high-speed cameras 216 may collect "stereo information" such as vibration information using at least two cameras. The one or more high-speed cameras 216 may be used to determine the orientation of the object within the carrier and the orientation information as needed.

[0053] Although the defect identification system 108 can be very effective in identifying defects or potential defects, the defect identification system 108 is also a complex system with many components. The defect identification system 108 is generally not suitable for use near the robotic repair device 110 because of possible obstructions, vibrations, or other interferences, and also because the robotic repair device 110 may require a large amount of space. As previously mentioned, Figure 1As discussed in , it is common practice to transport the vehicle (object) to a location separate from the defect identification system 108 for processing by the robotic repair device 110. In addition, in some configurations, a single defect identification system 108 can be used to provide defect information for use in more than one downstream robotic repair device 110.

[0054] According to examples herein, the robotic repair apparatus 110 as described above can be used to sand and polish one or more defects on a surface. The robotic repair apparatus 110 can have one or more cameras 124 (as previously discussed), which can be used to locate the paint / varnish coating / pad or other defects to be repaired. The robotic repair apparatus 110 includes a moving mechanism 302, which can be used to move the end arm assembly 118 ( Figure 1 ) is moved to the vicinity of the defect repair area. The robotic repair device 110 may include one or more sensors 304, such as force sensors or other sensors (one or more) and / or actuators (one or more) described herein. The robotic repair device 110 may include a dedicated controller that controls the movement and sensing of the arm 114 and related components. However, it is clearly contemplated that in some embodiments, the arm 114 and / or components mounted thereon may have their own controllers or may be controlled by the controller 202. The dedicated controller may receive and execute movement and sensing commands such as from the controller 202.

[0055] The end arm assembly 118 has been discussed previously and may include various tools, such as the Figure 1A However, Figure 2 As shown, it is clearly contemplated that in other embodiments, some components may be located elsewhere (not on assembly 118 ), but rather connected to arm 114 .

[0056] The first tool 120 can be mounted on the arm 114. In some embodiments, the first tool 120 is coupled to the first end effector 306. In some examples, the second tool 122 can be mounted to the arm 114. The second tool 122 (if used) can be connected to the second end effector 308. The fluid removal tool 310 can be mounted to the arm 114. However, it is clearly contemplated that in some examples, some of these components may not be mounted to the arm 114 or may be on a different arm than the arm 114. For example, the arm 114 can support a first tool 120, such as a sanding tool, and the second arm can support one or more cameras 124, the second tool 122, and / or other components.

[0057] In one example, the arm 114 is moved into position by an arm movement mechanism 312. In one embodiment, the first tool 120 and the second tool 122, the one or more cameras 124, and the fluid removal tool 310 may also be moved into position by the arm movement mechanism 312, or may each have their own movement mechanism that moves them to a suitable position on or near the surface. The robotic repair apparatus 110 may have a force control unit 314. The force control unit 314 may also be located on the arm 114 to control the interaction between the arm 114, the end effector system, and the workpiece surface.

[0058] In some examples, air line 316 and fluid distributor 318 may be provided from arm 114 to assembly 118 ( Figure 1A ) is supplied to provide the necessary air and fluid supply that may be needed to operate the first tool 120 and / or the second tool 122. The fluid removal tool 310 may also be connected to a force control unit 314. The fluid removal tool 310 may be, for example, a fabric-based wiping medium, an air knife, a vacuum system, or other suitable tool. However, it is also conceivable that in some embodiments, the fluid removal tool 310 may be connected to an independent force control unit rather than a force control unit for the first tool 120 and the second tool 122. It is also conceivable that in other embodiments, the fluid removal tool 310 may be a passive tool without an associated force control unit. In some examples, the fluid removal tool 310 may be moved in space using a mechanism that controls variables, such as pitch, tilt, and deflection of the active wiping motion of the fluid removal tool 310.

[0059] The force control unit 314 can maintain an appropriate force or pressure between the first tool 120, the second tool 122, and / or the fluid removal tool 310 and the surface of the object. In some examples, the fluid removal tool 310 can be used in conjunction with a fluid removal mechanism 322. The fluid removal mechanism 322 can be a pad, vacuum, brush, or scraping tool for removing particulate matter, debris, liquid, or slurry from the wiping medium of the fluid removal tool 310. The fluid removal mechanism 322 can help provide a suitable absorbent and effective wiping medium to clean the workpiece surface more than once.

[0060] The controller 202 may be in electronic communication with the camera (one or more cameras 124) and the robotic paint repair device 110. The controller 202 may be configured to control the one or more cameras 124 to scan an area of ​​the surface (a portion of the entire exposed surface) based on the first data (determined and provided by the defect recognition system 108 discussed above). The scanning of the area by the one or more cameras 124 may collect and provide scanning data representing a location and / or other information related to one or more defects on the surface of the object at the repair location. In contrast, the first data represents information collected by the defect recognition system 108 at a location other than the repair location, which is a location operated by the robotic repair device 110 and where the one or more cameras 124 are located. The controller 202 is configured to manipulate the robotic arm (arm 114) to position a tool (the first tool 120 or the second tool 122) based at least on the scanning data collected by the one or more cameras 124 at the repair location.

[0061] The controller 202 may be a digital controller having one or more processors, and may be implemented in software or may be implemented by a combination of software and hardware. The controller 202 may have various functions and capabilities, such as the functions and capabilities of the processing component 208 described above. Various other functions may also be envisioned, including serving as an interface between the defect identification system 108 and the robotic repair device 110. The controller 202 may have various functions. These functions may be implemented in hardware, software, firmware, or any combination thereof, locally or remotely. If implemented in software, these functions may be stored on or transmitted through a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media or communication media, computer-readable storage media corresponding to tangible media such as data storage media, and communication media including, for example, any media that facilitates the transfer of computer programs from one place to another according to a communication protocol. In this manner, computer-readable media may generally correspond to: (1) a tangible, non-transitory computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes and / or data structures for implementing the techniques described in this disclosure.A computer program product may include a computer-readable medium.

[0062] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. Likewise, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, optical cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, optical cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium.

[0063] However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other transient media, but refer to non-transient tangible storage media. Disks and optical disks used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks and Blu-ray discs, where disks usually reproduce data magnetically, while optical discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0064] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits, and any combination of these components. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functions described herein may be provided in dedicated hardware and / or software modules. Similarly, these techniques may be implemented entirely in one or more circuits or logic elements.

[0065] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless communication devices or wireless handsets, microprocessors, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in hardware units in conjunction with appropriate software and / or firmware, or provided by a collection of interoperative hardware units (including one or more processors as described above).

[0066] In one example, the functions, techniques, or algorithms described herein can be implemented in software. Software can be composed of computer executable instructions stored on a computer-readable medium or a computer-readable storage device, such as one or more non-transitory memories or other types of hardware-based storage devices, which can be local or networked. In addition, these functions correspond to modules, which can be software, hardware, firmware, or any combination thereof. Multiple functions can be performed in one or more modules as needed, and the described embodiments are only examples. Software can be executed on a digital signal processor, an ASIC, a microprocessor, or other types of processors running on a computer system (such as a personal computer, a server, or other computer system), thereby transforming such a computer system into a specially programmed machine.

[0067] Figure 3 4 shows a global coordinate system 400 of an object 101 to be inspected (here a painted vehicle 102 having a surface 103) as the object 101 is transported along a repeatable object travel path or direction. The object travel path or direction may be defined in a three-dimensional world coordinate system having a world origin and world coordinate axes. As previously described, the painted vehicle 102 may be mounted to a carriage 104, which may be coupled to a track system 106 ( Figure 3 The travel path of the painting vehicle 102 may be a path followed by the painting vehicle 102 along the track system 106 . Figure 3 The world or global coordinate system shown can be defined by the point on the floor where the geometric center of the track system 106 (on which the carriage 104 and the paint vehicle 102 are located) is located, with the "z" axis pointing upward from the floor, the "y" axis pointing to the side of the conveyor, and the "x" axis pointing in the opposite direction of the forward motion of the conveyor, as shown. Figure 3 shown. Figure 3 It is merely an example of a coordinate system with an origin that can be utilized. Figure 3 As shown, surface 103 may have one or more defects 105 . Figure 3 The coordinate system can be obtained by Figure 2 The controller 202 and / or processing component 208 are implemented to understand the location of one or more defects 105 relative to the global coordinate system 400. This location information can be captured as referenced above. Figure 2 First data for discussion.

[0068] Figure 4 is a schematic diagram of a robotic paint inspection and repair system 500 according to an example. Figure 1 and Figure 2Many aspects of the robotic paint inspection and repair system 500 are discussed. The robotic paint inspection and repair system 500 may include a robotic repair unit 502 including a robotic arm 504 and a robotic inspection unit 506 (a second robot), wherein the robotic repair unit 502 includes a robotic arm 504 and the robotic inspection unit 506 includes a robotic arm 508. These systems may be controlled by a motion controller that may receive instructions from one or more application controllers 510 (e.g., the controller 202). In addition to the controller 202 ( Figure 2 ) In addition to or in lieu of controller 202, application controller 510 can receive input or provide output to user interface 512.

[0069] The robotic repair unit 502 includes a force control unit 514 and an end effector 516 that can be aligned with the component 118 as previously described. The robotic inspection unit 506 can include various components, including one or more cameras 518 mounted on the robotic arm 508. The one or more cameras can have a structure similar to the one or more cameras 124 described previously. Figure 4 As shown, the force control unit 514 can be coupled to the assembly 118 via the end effector 516. As previously described, the assembly 118 can carry the first tool 120 and the optional second tool 122. The first tool 120 and the second tool 122 can be constructed in the manner discussed previously. One or more cameras 518 of the robotic inspection unit can perform a visual inspection on the surface 103 of the painted vehicle 102. If one or more defects are detected, then these defects can be repaired by the robotic repair unit 502.

[0070] The robotic paint inspection and repair system 500 differs from previous systems or devices in that one or more cameras 518 may be implemented on a separate robotic component from the robotic repair device. However, as previously discussed Figure 1 As in the embodiment of the present invention, inspection is performed at the defect repair location 128. In other words, although the one or more cameras 518 are not directly mounted to the robotic repair unit 502, the one or more cameras 518 are very close to the robotic repair unit 502 when the robotic repair unit 502 performs polishing, sanding, etc.

[0071] The robotic repair unit 502 and the robotic inspection unit 506 may have a base fixed to a track system that is configured to travel with the vehicle being repaired. However, as previously described, the arms 504 and 508 and other components may be movable. Depending on the location of the defect, the robotic repair unit 502 and the robotic inspection unit 506 may need to move closer to or farther from the vehicle, or may need to move higher or lower relative to the vehicle. Figure 4 A Cartesian coordinate system is shown with reference to the x-axis, y-axis and z-axis. Figure 3 The global coordinate system 400 of FIG. 4 is used. It should be appreciated that according to some examples, the robotic inspection unit 506 may not be offset relative to the robotic repair unit across the vehicle in the y-axis direction. Instead, the robotic inspection unit 506 may be placed in another location, such as on the same side of the vehicle as the robotic repair unit 502, and offset in the x-axis direction, for example. The positions of the first tool 120 and the second tool 122 may be changed by manipulation as previously described.

[0072] Figure 5A and Figure 5B A schematic diagram of a method is shown in which position errors in locating a defect are introduced due to movement of an object where the defect is located from a first position to a second position and surface modification of a repair area to address the defect. Figure 5C A method is shown in which one or more cameras 124 ( Figure 1 ) or having one or more cameras 518 ( Figure 3 )'s robotic inspection unit 506 is capable of reducing the size of an area undergoing surface modification to remove one or more defects.

[0073] Specifically, Figure 5A Initially, the use of a global coordinate system 400 by the defect identification system 108 ( Figures 1 to 3 ) is located in a Cartesian coordinate system. When the defect 600 is located in the defect detection area by the defect recognition system 108, the defect 600 will have the coordinates shown. Defect repair using a robot requires highly accurate position information about the location of the identified defect on the surface of the object. This position information can be captured as part of the first data mentioned above. However, since manufacturing generally requires assembly lines and the movement of objects from one manufacturing step to another, the object may be shifted in position due to the movement. As an example, when an object (vehicle) is moving along Figure 1 When the path shown by arrow A in FIG. 1 is traveled from the defect scanning position to the defect repairing position, the object (vehicle) may have been displaced in one or more of the x-direction, y-direction, and / or z-direction on the carriage (and / or rotated around one or more of the x-, y-, or z-axis). Alternatively, when the rail system or other transport mechanism moves from the defect scanning position to the defect repairing position, the rail system or other transport mechanism may be displaced in one or more of the z-direction, x-direction, or y-direction (and / or rotated around one or more of the z-, x-, or y-axis) (see FIG. 1 ). Figure 1 ). Therefore, although the location of defect 600 may have been correctly determined as Figure 5AHowever, due to the error introduced during the movement from the defect scanning position to the defect repairing position, the position of the defect 600 may have shifted to Figure 5B In other words, defect 600 is moved from Figure 5A Location to Figure 5B Movement of the position may introduce position errors.

[0074] It should also be noted that in some systems, the carriage (and therefore the object) continues to move even while the repair process is occurring. In other words, while the surface repair is occurring, the object is moving at the repair location. This means that even after the object reaches the repair location, additional position errors may continue to be introduced. Since new data is collected directly at the repair point, the systems, devices, and methods according to the present disclosure are able to account for these ongoing position errors.

[0075] Typically (in previous systems), given that Figure 5B 602 or a portion of the surface will be subjected to surface modification to eliminate any positional offset (position error). The relatively large area 602 will ensure that the defect 600 is processed, even if the location of the defect is different from the location recorded in the first data. Compared with the surface modification of area 604, the relatively large area 602 of surface modification will result in increased tool wear, material usage, processing time and reduced aesthetics. In addition, if one or more defects appear in challenging areas, the larger area 602 will hinder effective repair. The area 604 used for surface modification is relatively smaller than area 602. In the case where the defect (one or more) appears near challenging areas, such as edges, feature lines, etc., the smaller repair area (area 604) can allow defect repair closer to those areas.

[0076] This can be accomplished by using one or more cameras 124 ( Figure 1 ) or a robotic inspection unit 506 ( Figure 3 ) captures and provides scan data (or second data) about an updated position of the defect 600 at the defect repair position to obtain area 604, the updated position being generated due to movement to the defect repair position.

[0077] A controller (e.g., controller 202) can be configured to use the first data and the scan data in various ways. For example, the controller can be configured to perform a comparison of the scan data with the first data. Based on the comparison of the scan data with the first data, the controller can use the location of one or more defects from the scan data (reflecting Figure 5B The first data is updated by using the update position of the first data.

[0078] Figure 5D A further determination by the controller is shown, including extrapolating or updating the position of at least another defect 606. Specifically, based on the update to the first data, the controller can be configured to re-determine the position of each defect (or a subset) of the one or more defects (captured in the first data) on the entire (or a portion) surface of the object, including re-determining the position of one or more defects on a second portion of the surface outside the scanning area of ​​the camera. For example, the controller can determine that the entire object has been offset and / or rotated in a particular manner, so that the offset and / or rotation can be relatively predicted or determined for the global set of defects initially identified. Based on the position of each of the one or more defects on the entire surface of the object re-determined by the controller, the controller can be configured to manipulate the robotic arm to position the tool to perform surface modification on the one or more defects on the second portion of the surface outside the camera scan. In other words, it is contemplated that one or more cameras do not have to collect scanning data about each defect on the object before performing surface modification.

[0079] In addition to collecting and re-determining only the location data, the controller (e.g., controller 202) can be configured to collect one or more features about the defect 600 in the scan data. Therefore, the scan data needs to represent not only the location information, but also features such as size (depth, length, width), shape (protrusion, depression, concave / convex), properties (color, hair, sand, dust), and / or confirmation of existence (i.e., determination that the defect 600 actually exists and is not a false confirmation from the first data). Therefore, based on the comparison of the scan data to the first data, the controller can update or enhance the first data to reflect one or more features of the one or more defects from the scan data. Because a set of detailed information related to the defect (including improved location data) can be collected directly at the repair point, it is anticipated that in some embodiments, lower upfront accuracy may be required when collecting the first data compared to a system where the first data is the only data available for defect location and characterization. In such an embodiment, a smaller, cheaper, and / or simpler system can be advantageously employed to collect the first data.

[0080] The controller may also be configured to perform various analyses on the scanned data by comparing it to the first data. Such analyses may indicate that relative position movement has increased, which may be due to a loose carriage on the track system. Such analyses may indicate inaccuracies in the first data, which may be due to one of the multiple cameras of the defect identification system 108 being mispositioned (perhaps due to being bumped, misaligned, or jostled). The controller may issue an alert to personnel indicating that the accuracy of the first data has decreased and / or simply indicate that personnel should check, for example, carriage mounting, camera positioning, or other standards of the system.

[0081] FIG. 6A to FIG. 6C shows that the position error may not be caused only by the object moving from the first position ( Figure 5A ) to the second position ( Figure 5B ). In addition, position errors may be introduced by vibrations of the object caused by its movement in the repair position. Figure 6C It is envisioned that vibrations can be measured locally at the defect repair location. Specifically, the scan data is based on multiple images taken at intervals over a period of time. Fig. 6A As shown, the first image can be captured at the first time. Figure 6B As shown, a second image can be taken at a second time. Due to the vibration of the object at the defect repair location, Fig. 6A and Figure 6B Defect 700 in has moved to another location, such as Figure 6C As shown. A high-speed camera (or cameras) that capture multiple frames per second can be used to capture the vibration dynamics of the object. The controller can be configured to determine the positional offset of the defect 700 caused by the vibration of the object (by Figure 6C denoted by arrow V in ). This information can be used to improve the positioning of one or more tools of the robotic repair equipment, and can also be used to optimize other criteria, such as the engagement force of the tool, the modification of the force to accommodate movement, the change of the repair area (knowing the direction and amplitude of the vibration can allow the size of the repair to be scaled), the change of the speed of the repair tool, and the decision to skip the repair at this time if the movement will interfere with the repair when performing surface modification to remove the defect 700. Information about the defect type (e.g., dust / pit / micropore / orange peel wrinkles, etc.) can be better estimated from the second scan because the robot may capture a higher resolution image of the defect than obtained in the first scan. Better information about the true size of the defect can also be obtained because the original scan estimate of the defect size, height and depth may be affected by shadows. The moving robot arm may better control the shadows. This information will help the robot perform better repairs and avoid practices such as attempting to repair dents that may be misclassified as dirt by existing systems. Obtaining better quality defect data from the second scan performed by the robot also improves the quality of data sent to the factory data analysis system, which is used to drive continuous improvement of the painting process. As an example, the first defect detection system may have a false confirmation classification, such as reporting a 10% probability of a defect existing at a certain location. The second scan helps avoid attempting repairs in areas where confidence in the first detection is low.

[0082] Figure 7 shows the same Figure 1 and Figure 1A The component 118 discussed above is identical to the multiple robot mounted and manipulated components 118A, 118B and 118C. Figure 71 and 12. Some other aspects of the present disclosure are shown that are worth noting. First, the one or more cameras 124 of the assembly 118A are shown performing a scan 800 of a portion 802 of a surface 804 of the vehicle 102 within the defect repair location 128. The portion 802 (corresponding to the scan area) may include at least a first defect 806 and a second defect 808. The relative positions of the first defect 806 and the second defect 808 may be used to identify the defect previously identified. Figure 5D 1, so that in some cases, other defects such as defect 810 can be processed without scanning by one or more cameras 124 of the second assembly 118B. Assembly 118B shows the first tool 120 performing surface modification, such as sanding to remove defect 810. Assembly 118C shows surface modification, such as polishing, in progress.

[0083] Figure 7 Scan data 812 is shown being collected and transmitted as the vehicle 102 moves along the assembly line as indicated by arrow A2. Components 118A, 118B, 118C may be manipulated in concert by a plurality of robots to address defects 806, 808, 810, etc. in various ways. Component 118A is shown being brought closer by manipulating the robotic arm as indicated by arrow S to move one or more cameras 124 toward surface 804 to collect different data regarding a first defect 806 and a second defect 808.

[0084] Figure 8 A method 900 for identifying and repairing one or more defects on a surface of an object is shown. The method 900 may include collecting first data 902 representing the location of one or more defects on the surface of the object at a first location. This may be in the defect identification system 108 ( Figure 1 and Figure 2 ). The method 900 may include transferring the object from the first location to the second location, and performing repair of one or more defects at the second location by a robotic paint repair device having a robotic arm and a tool mounted to the robotic arm 904. The method 900 may include scanning a portion of a surface of the object at the second location based on the first data 906. The method 900 may include contacting the surface with the tool to perform surface modification of the object to remove the one or more defects, wherein the location of contacting the surface is determined at least in part by scanning a portion of the surface of the object at the second location 908.

[0085] The method 900 may optionally include other steps, including at least determining the location of one or more defects on substantially the entire surface of the object based on the first data. The method 900 may optionally include comparing the scan data with the first data, updating the first data with the location of the one or more defects from the scan data, and re-determining the location of each of the one or more defects on the entire surface of the object. Contacting the surface to perform surface modification occurs at a second portion of the surface of the object outside the scanning range of the portion of the surface of the object. Updating the first data may include updating the first data to reflect one or more features of the one or more defects from the scan data. Scanning at the second position may include moving the camera toward the object using a robotic arm to change the size of the scanning area and the scan data collected. Scanning the portion of the surface at the second position may include taking multiple images at intervals over a period of time. The method 900 may optionally include determining a shift in the location of the one or more defects caused by vibration of the object. In addition to including determining a shift in the location of the one or more defects caused by vibration of the object, the method 900 may optionally include determining a shift in the location of the one or more defects caused by the transfer of the object from the first position to the second position.

[0086] The disclosure herein includes, but is not limited to, the following illustrative examples:

[0087] Example 1 is a system for identifying and repairing one or more defects on the surface of an object, which system may include any one or combination of a robotic paint repair device, a camera, and a controller. The robotic paint repair device may have a robotic arm and a tool mounted to the robotic arm. The tool may be configured to contact the surface to perform surface modification of the object, thereby removing one or more defects. The camera may be positioned adjacent to the robotic paint repair device in the repair area, and the camera is configured to scan a portion of the object surface having at least one of the one or more defects, and collect scan data. The controller may communicate with the camera and the robotic paint repair device. The controller may be configured to control the camera to scan the area of ​​the object surface based on first data representing the position of one or more defects on the object surface collected at a position different from the repair area. The controller may be configured to manipulate the robotic arm to position the tool based at least on the scan data.

[0088] Example 2 is a system according to Example 1, wherein the camera can be mounted to the second robot or to the robotic paint repair device.

[0089] Example 3 is a system according to any one or combination of Examples 1-2, wherein the controller can be configured to control movement of one of the second robot and the robotic paint repair device to adjust the position of the camera relative to the object.

[0090] Example 4 is a system according to any one or combination of Examples 1 to 3, wherein the controller can be configured to perform a comparison of the scan data with the first data.

[0091] Example 5 is a system according to any one or a combination of Examples 1 to 4, wherein based on a comparison of the scan data with the first data, the controller can update the first data with the location of one or more defects from the scan data.

[0092] Example 6 is a system according to any one or a combination of Examples 1 to 5, wherein, based on an update to the first data, the controller can be configured to re-determine the location of each of one or more defects on a portion of the surface of the object, including the location of one or more defects on a second portion of the surface of the object outside of the camera scan, the portion of the surface of the object including less than the entire surface of the object.

[0093] Example 7 is a system according to any one or a combination of Examples 1 to 6, wherein, based on an update to the first data, the controller can be configured to redetermine the position of each of the one or more defects on the entire surface of the object, including the position of one or more defects on a second portion of the object surface outside of the camera's scan.

[0094] Example 8 is a system according to any one or a combination of Examples 1 to 7, wherein, based on the location of each of the one or more defects on the entire surface of the object re-determined by the controller, the controller is configured to manipulate the robotic arm to position the tool to perform surface modification on the one or more defects on a second portion of the object surface outside of the camera scan.

[0095] Example 9 is a system according to any one or combination of Examples 1 to 7, wherein based on a comparison of the scan data with the first data, the controller updates the first data to reflect one or more characteristics of the one or more defects from the scan data.

[0096] Example 10 is a system according to any one or combination of Examples 1 to 9, wherein the controller issues an alarm based on the comparison of the scan data with the first data.

[0097] Example 11 is a system according to any one or a combination of Examples 1 to 10, wherein the scanning data is based on multiple images taken at intervals over a period of time, and wherein based on the scanning data, the controller is configured to determine an offset in the location of one or more defects caused by vibration of the object.

[0098] Example 12 is a system according to any one or combination of Examples 1 to 11, wherein the object includes a vehicle and the surface of the object includes a mirror surface, and wherein the first data is collected at a location of the vehicle before it moves along an assembly line to a repair area.

[0099] Example 13 is a system according to any one or combination of Examples 1 to 12, wherein the vehicle moves along an assembly line during repair, and the first data and the scanning data are collected while the vehicle moves along the assembly line.

[0100] Example 14 is a method for identifying and repairing one or more defects on a surface of an object. The method may include any one or a combination of the following: collecting first data representing the location of one or more defects on the surface of the object at a first location; transferring the object from the first location to a second location, performing the repair of the one or more defects at the second location by a robotic paint repair device having a robotic arm and a tool mounted to the robotic arm; scanning a portion of the surface of the object at the second location based on the first data; and contacting the surface of the object with the tool to perform surface modification of the object to remove the one or more defects, wherein the location of contacting the surface of the object is determined at least in part by scanning a portion of the surface of the object at the second location.

[0101] Example 15 is a method according to Example 14, further optionally including: comparing the scan data with the first data; updating the first data using the location of one or more defects from the scan data; and re-determining the location of each of the one or more defects on the entire surface of the object.

[0102] Embodiment 16 is a method according to any one or a combination of Examples 14 to 15, wherein contacting the object surface to perform surface modification occurs on a second portion of the object surface outside the scanning range of that portion of the object surface.

[0103] Example 17 is a method according to any one or a combination of Examples 14 to 16, further optionally comprising: comparing the scan data with the first data; updating the first data to reflect one or more characteristics of the one or more defects from the scan data.

[0104] Example 18 is a method according to any one or combination of Examples 14 to 17, wherein scanning a portion of the surface of the object at the second position includes using a robotic arm to move the camera toward or away from the object.

[0105] Example 19 is a method according to any one or a combination of Examples 14 to 18, wherein scanning a portion of the surface of the object at a second position includes taking multiple images at intervals over a period of time, and also includes determining a shift in the location of one or more defects caused by vibration of the object.

[0106] Example 20 is a method according to any one or a combination of Examples 14 to 19, which, in addition to determining the offset of the position of one or more defects caused by the vibration of the object, also includes determining the offset of the position of one or more defects caused by the transfer of the object from a first position to a second position.

[0107] Example 21 is a method according to any one or combination of Examples 14 to 20, further comprising moving a vehicle along an assembly line while contacting a surface of the object with a tool to perform surface modification of the object to remove one or more defects.

[0108] Example 22 is a method according to any one or a combination of Examples 14 to 21, further comprising collecting first data representing the location of one or more defects on a surface of an object at a first location and moving a vehicle along an assembly line while scanning a portion of the surface of the object at a second location based on the first data.

[0109] Example 23 is a method for identifying and repairing one or more defects on the surface of an object, the method optionally including any one or combination of the following: scanning at a first position to collect first scanning data; determining at least one position of one or more defects on substantially the entire surface of the object from the first scanning data; transferring the object from the first position to a second position, and performing repair of the one or more defects at the second position by a robotic paint repair device having a robotic arm and a tool mounted to the robotic arm; scanning only a portion of the surface of the object at the second position to collect second scanning data, wherein the portion of the surface selected for scanning is based on the first scanning data; comparing the first scanning data with the second scanning data; based on the comparison, at least updating the position of one or more defects to reflect the offset of the position of one or more defects; and contacting the surface of the object with a tool to perform surface modification of the object to remove one or more defects, wherein the position of contacting the surface of the object is determined based on the offset of the position of the defect.

[0110] Example 24 is a method according to Example 23, wherein updating the position of one or more defects to reflect the offset of the position of one or more defects includes re-determining the position of each of the one or more defects on the entire surface of the object.

[0111] Example 25 is a method according to any one or combination of Examples 23 to 24, wherein contacting the object surface to perform the surface modification occurs on a second portion of the object surface outside the scanning range of that portion of the object surface.

[0112] Example 26 is a method according to any one or combination of Examples 23 to 25, further comprising updating the first scan data to reflect one or more characteristics of the one or more defects from the second scan data.

[0113] Example 27 is a method according to any one or combination of Examples 23 to 26, wherein scanning only a portion of the surface of the object at the second position includes using a robotic arm to move the camera toward or away from the object.

[0114] Example 28 is a method according to any one or a combination of Examples 23 to 27, wherein scanning only a portion of the surface of the object at the second position includes taking multiple images at intervals over a period of time, and also includes determining a shift in the location of one or more defects caused by vibration of the object.

[0115] Example 29 is a method according to any one or combination of Examples 23 to 28, further comprising moving a vehicle along an assembly line while contacting a surface of the object with a tool to perform surface modification of the object to remove one or more defects.

[0116] Example 30 is a method according to any one or combination of Examples 23 to 29, further comprising moving the vehicle along the assembly line while scanning at a first location to collect first scan data and scanning only a portion of the surface of the object at a second location to collect second scan data.

[0117] The various examples 1 to 30 described above can be combined in any combination. The elements therein can be combined in any combination. Unless otherwise specified, the elements thereof are optional.

[0118] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A system for identifying and repairing one or more defects on a surface of an object, the system comprising: a robotic paint restoration apparatus having a robotic arm and a tool mounted to the robotic arm, wherein the tool is configured to contact the surface of the object to perform surface modification of the object to remove the one or more defects; a camera positioned adjacent to the robotic paint repair device within a repair area, the camera configured to scan a portion of the object surface having at least one of the one or more defects and collect scan data; as well as a controller in communication with the camera and the robotic paint repair apparatus, the controller being configured to control the camera to scan an area of ​​the surface of the object based on first data representing locations of the one or more defects on the surface of the object collected at a location different from the repair area, wherein the controller is configured to manipulate the robotic arm to position the tool based at least on the scanning data.

2. The system according to claim 1, wherein: The camera is mounted to a second robot or to the robotic paint repair device.

3. The system according to claim 2, wherein: The controller is configured to control movement of one of the second robot and the robotic paint repair apparatus to adjust a position of the camera relative to the object.

4. The system according to claim 1, wherein: The controller is configured to perform a comparison of the scan data with the first data.

5. The system according to claim 4, wherein: Based on a comparison of the scan data with the first data, the controller updates the first data with the locations of the one or more defects from the scan data.

6. The system according to claim 5, wherein: Based on an update to the first data, the controller is configured to re-determine the position of each of the one or more defects on a portion of the surface of the object, including the position of one or more defects on a second portion of the surface of the object outside of the scan of the camera, wherein the portion of the surface of the object includes less than the entire surface of the object.

7. The system according to claim 5, wherein: Based on the update to the first data, the controller is configured to re-determine the position of each of the one or more defects on the entire surface of the object, including the position of one or more defects on a second portion of the object surface outside the scan of the camera.

8. The system according to claim 7, wherein: Based on the position of each of the one or more defects on the entire surface of the object re-determined by the controller, the controller is configured to manipulate the robotic arm to position the tool to perform surface modification on the one or more defects on a second portion of the surface of the object outside of the camera scan.

9. The system according to claim 4, wherein: Based on a comparison of the scan data with the first data, the controller updates the first data to reflect one or more characteristics of one or more defects from the scan data.

10. The system according to claim 4, wherein: Based on the comparison of the scan data with the first data, the controller issues an alarm.

11. The system according to claim 1, wherein: The scan data is based on a plurality of images taken at intervals over a period of time, and wherein based on the scan data, the controller is configured to determine a shift in the location of the one or more defects caused by vibrations of the object.

12. The system of claim 1, wherein: The object comprises a vehicle, and the surface of the object comprises a mirrored surface, and wherein the first data is collected at a location of the vehicle before it moves along an assembly line to the repair area.

13. The system according to claim 11, wherein: The vehicle moves along the assembly line during the repairing, and the first data and the scan data are collected as the vehicle moves along the assembly line.

14. A method of identifying and repairing one or more defects on a surface of an object, the method comprising: collecting, at a first location, first data representing locations of the one or more defects on the surface of the object; transferring the object from the first location to a second location where repair of the one or more defects is performed by a robotic paint repair apparatus having a robotic arm and a tool mounted to the robotic arm; scanning a portion of the surface of the object at the second location based on the first data; as well as The tool is contacted with the surface of the object to perform surface modification of the object to remove the one or more defects, wherein the location of contacting the surface of the object is determined at least in part by scanning a portion of the surface of the object at the second location.

15. The method according to claim 14, further comprising: comparing the scan data with the first data; updating the first data using the locations of the one or more defects from the scan data; and Each of the one or more defects is relocated across the surface of the object.

16. The method according to claim 15, wherein: Contacting the object surface to perform the surface modification occurs at a second portion of the object surface outside of the scan range of the portion of the object surface.

17. The method according to claim 14, further comprising: comparing the scan data with the first data; The first data is updated to reflect one or more characteristics of the one or more defects from the scan data.

18. The method according to claim 14, wherein: Scanning a portion of the surface of the object at the second position includes moving a camera toward or away from the object using the robotic arm.

19. The method according to claim 14, wherein: Scanning a portion of the surface of the object at the second location includes capturing a plurality of images spaced apart over a period of time, and further includes determining a shift in the location of the one or more defects caused by vibration of the object.

20. The method of claim 19, further comprising determining a shift in the position of the one or more defects caused by a vibration of the object, and further comprising determining a shift in the position of the one or more defects caused by a transfer of the object from the first position to the second position.

21. The method of claim 14, further comprising moving the vehicle along an assembly line while contacting the surface of the object with the tool to perform surface modification of the object to remove the one or more defects.

22. The method of claim 21 further comprising collecting the first data representing the location of the one or more defects on the surface of the object at the first location and moving the vehicle along the assembly line while scanning a portion of the surface of the object at the second location based on the first data.

23. A method of identifying and repairing one or more defects on a surface of an object, the method comprising: Scanning at a first location to collect first scan data; determining from the first scan data at least one location of the one or more defects across substantially the entire surface of the object; transferring the object from the first location to a second location where repair of the one or more defects is performed by a robotic paint repair apparatus having a robotic arm and a tool mounted to the robotic arm; scanning only a portion of the surface of the object at the second location to collect second scan data, wherein the portion of the surface selected for scanning is based on the first scan data; comparing the first scan data with the second scan data; Based on the comparing, updating at least the location of the one or more defects to reflect a shift in the location of the one or more defects; as well as The tool is contacted with the surface of the object to perform surface modification of the object to remove the one or more defects, wherein the location of contacting the surface of the object is determined based on a shift in the location of the defect.

24. The method according to claim 23, wherein: Updating the location of the one or more defects to reflect the shift in the location of the one or more defects includes relocating each of the one or more defects across the surface of the object.

25. The method according to claim 24, wherein: Contacting the object surface to perform the surface modification occurs at a second portion of the object surface outside of the scan range of the portion of the object surface.

26. The method of claim 23, further comprising updating the first scan data to reflect one or more characteristics of the one or more defects from the second scan data.

27. The method according to claim 23, wherein: Scanning only a portion of the surface of the object at the second position includes moving a camera toward or away from the object using the robotic arm.

28. The method of claim 23, wherein: Scanning only a portion of the surface of the object at the second position includes capturing a plurality of images spaced apart over a period of time, and further includes determining a shift in the position of the one or more defects caused by vibration of the object.

29. The method of claim 23, further comprising moving the vehicle along an assembly line while contacting the surface of the object with the tool to perform surface modification of the object to remove the one or more defects.

30. The method of claim 29, further comprising moving the vehicle along the assembly line while scanning at the first location to collect the first scan data and scanning only a portion of the surface of the object at the second location to collect the second scan data.

Images