Robot repair control system and method

Abstract

An imaging and repair system (100) is presented, including a first imaging system (110) configured to image and detect defects on a workpiece surface. The first imaging system comprises a first camera configured to acquire a plurality of first images of the workpiece surface. The plurality of first images are stored in a data source. The system also includes a second imaging system (110) configured to image and characterize the orange peel of the workpiece surface in an area proximate to the defect. Characterizing the workpiece surface includes identifying an orange peel delta value. The system also includes a defect repair processor configured to select a repair plan based on the defect type. The system also includes a defect correction unit configured to modify the selected repair plan based on the orange peel characterization of the workpiece surface. The system also includes a defect repair tool (120) configured to automatically execute the modified repair plan.

Description

Background Art

[0001] Clear coat repair is one of the final operations that should be automated in the original equipment manufacturing (OEM) department of automobiles. Not only this process, but also other painting applications (such as primer sanding, clear coat defect removal, clear coat polishing, etc.) suitable for the use of abrasives and / or robotic inspection and robotic repair are desired to be automated.

[0002] As a conventional effort to automate the detection and repair of painting defects, there is a system described in U.S. Patent Application Publication No. 2003 / 0139836, which discloses the use of electronic imaging to detect and repair painting defects on a vehicle body. This system creates three-dimensional painting defect coordinates for each painting defect by comparing and collating the imaging data of the vehicle with the CAD data of the vehicle. These painting defect data and painting defect coordinates are used to formulate a repair plan for automated repair using a plurality of automatic robots that perform various tasks including sanding and polishing of the painting defects.

Summary of the Invention

[0003] An imaging and repair system is presented, which includes a first imaging system configured to image and detect defects on a workpiece surface. The first imaging system comprises a first camera configured to acquire a plurality of first images of the workpiece surface. The plurality of first images are stored in a data source. The system also includes a second imaging system configured to image and characterize the orange peel texture of the workpiece surface in areas adjacent to the defect. Characterizing the workpiece surface includes identifying the delta value of the orange peel texture. The system also includes a defect repair processor configured to select a repair plan based on the defect type. The system also includes a defect repair unit configured to modify the selected repair plan based on the orange peel texture characterization of the workpiece surface. The system also includes a defect repair tool configured to automatically execute the modified repair plan. [Brief explanation of the drawing]

[0004] While the drawings are not necessarily drawn to exact scale, similar figures may represent similar components in different drawings. The drawings provide a general, albeit illustrative, representation of the various embodiments discussed in this document.

[0005] [Figure 1] This is a schematic diagram of a robotic paint repair system in which embodiments of the present invention are useful.

[0006] [Figure 2] This specification describes a method for repairing robotic defects according to embodiments of this specification.

[0007] [Figure 3A] An image of orange peel skin is shown. [Figure 3B] An image of orange peel skin is shown.

[0008] [Figure 4] An embodiment of the repair plan generator according to this specification is shown.

[0009] [Figure 5A]This embodiment of the specification shows an imaging system for detecting orange peel texture on a workpiece surface. [Figure 5B] This embodiment of the specification shows an imaging system for detecting orange peel texture on a workpiece surface.

[0010] [Figure 6] This specification describes a method for characterizing the workpiece surface area in an embodiment of this specification.

[0011] [Figure 7] This specification describes a method for modifying a repair plan according to embodiments of this specification.

[0012] [Figure 8] This is a repair plan generation system architecture.

[0013] [Figure 9] An embodiment of a mobile device that can be used in the embodiments shown in the preceding figures is shown. [Figure 10] An embodiment of a mobile device that can be used in the embodiments shown in the preceding figures is shown.

[0014] [Figure 11] This is a block diagram of a computing environment that can be used in the embodiments shown in the preceding figures. [Modes for carrying out the invention]

[0015] Recent advances in imaging technology and computing systems have made it possible to perform clear coat inspection processes at production speeds. Specifically, recent studies have shown that stereo deflection measurement can provide images and locations of paint and clear coat defects, along with spatial information (providing coordinate position information and defect classification), at appropriate resolution to enable subsequent automated spot repairs.

[0016] As used herein, the term "vehicle" is intended to encompass a wide range of moving structures that are at least once subjected to painting with paint or a clear coat during manufacture. Although many examples herein relate to automobiles, it is explicitly contemplated that the methods and systems described herein are also applicable to trucks, trains, boats (with or without motors), airplanes, helicopters, and the like.

[0017] As used herein, the term "work surface" is intended to encompass any surface on which defect repair is attempted. The work surface can include the area having the detected defect, the area around the detected defect that is affected during the polishing operation, and the area around the affected area that is characterized by the presence of orange peel and may include a textured surface.

[0018] As used herein, the term "painting" is used herein in a broad sense to refer to any of the various layers such as the e-coat, filler, primer, paint, clear coat, etc. of a vehicle applied in the finishing process. Further, the term "paint repair" includes identifying and repairing the location of any visual artifact (defect) on or within any paint layer. In some embodiments, the systems and methods described herein use a clear coat as the target paint repair layer. However, the presented systems and methods are applicable to any particular paint layer (e-coat, filler, primer, paint, clear coat, etc.) with little or no modification.

[0019] As used herein, the term "defect" refers to an area on the work surface that impedes visual aesthetics. For example, many vehicles have a shiny or metallic appearance after painting. A "defect" can include debris trapped within one or more of the various paint layers on the work surface. Defects can also include excessive painting, including dirt, stain or sag in the paint, and dents.

[0020] Painting repair is one of the last remaining steps in the vehicle manufacturing process that is still largely manual. Historically, this has been due to two main factors: the lack of sufficient automated inspection and the difficulty of automating the repair process itself. The repair criteria for paint and clear coat are related to the aesthetics judged by the human eye of the sales agency that accepts the vehicle and the ultimate customer who will inspect the vehicle before purchase. Robots have traditionally been designed to provide "perfect" or highly "regular" repairs with distinct and clear edges and uniform cut parts (see Figure 4B). Unfortunately, this makes the repair highly visible to the human eye. The systems and methods described herein address ways to add irregularities to the painting repair process so that the repaired defect better blends into the vehicle surface and is less likely to be detected by the customer.

[0021] Figure 1 is a schematic diagram of a robotic painting repair system in which embodiments of the present invention are useful. System 100 generally includes two units: a visual inspection system 110 and a defect repair system 120. Both systems can be controlled by motion controllers 112, 122, which can receive commands from one or more application controllers 150. The application controller can receive input from or provide output to a user interface 160. The repair unit 120 includes a force control unit 124 that can cooperate with an end effector 126. As shown in Figure 1, the end effector 126 includes two tools 128, as further described in co-pending U.S. Provisional Patent Application No. 62 / 940950, filed November 27, 2019. However, other configurations are also explicitly contemplated.

[0022] A more detailed description of the robotic repair track can be found in concurrently pending U.S. Provisional Patent Application No. 62 / 941286, filed November 27, 2019, which is incorporated herein by reference.

[0023] The first of two main challenges, the inspection of the vehicle 130 by the inspection unit 110, is of interest due to the nature of the underlying problem domain. Generally, the surface of the object is extremely large compared to the defect itself, with the difference being several orders of magnitude. This leads to a trade-off between field of view and resolution in the selection of sensors. Furthermore, each coating layer of the finishing process (e-coat, primer, paint, clear coat, etc.) has a different visual appearance, with specularity being particularly noteworthy. Highly specular surfaces (i.e., high-gloss or highly reflective surfaces) present unique imaging challenges. These problems combined make inspection difficult. Advances in leveraging increasing computing resources have been achieved in this field in recent years, making several commercially available solutions available. The existence of a sufficiently capable inspection system 110 is important for identifying defects for repair by the repair unit 120.

[0024] The current state of technology in vehicle paint repair involves manually sanding and polishing defects to completion, with or without the assistance of power tools, using fine-grit abrasives and / or polishing systems, while maintaining the desired finish (e.g., comparable to a mirror finish in a clear coat). Skilled technicians performing such repairs leverage extensive training and their own senses to monitor the progress of the repair and make adjustments as needed. Such advanced skills are difficult to achieve with robotic solutions, where sensory capabilities are limited.

[0025] In addition, while the removal of abrasive material is a pressure-driven process, many industrial manipulators generally operate natively in position tracking / control regimes and are optimized with positional accuracy and precision in mind. This results in extremely rigid systems with highly rigid error response curves (i.e., small displacements result in very large corrective forces) where force control (i.e., joint torque and / or orthogonal forces) is inherently inadequate. Closed-loop force control techniques have been used (with limited usefulness) to address the latter, along with more recent (and more successful) force-controlled flanges that provide more flexible (i.e., less rigid) displacement curves, which are far better suited to sensitive force / pressure-driven processing. However, the problem of robust process planning / control remains, and is the focus of this research.

[0026] Figure 2 shows a robotic defect repair method according to one embodiment of the present invention. Method 200 outlines how a robotic repair system repairs a defect according to at least some embodiments described herein.

[0027] In block 210, commands are received from a robot controller, such as the application controller 150 in Figure 1. The commands include movement commands for different components of the robot repair unit.

[0028] In block 220, the robot motion controller moves the abrasive article attached to the tool to a designated position, preparing it to engage with the defect. The location of the defect on the vehicle is known from the inspection system. Moving the abrasive article to a designated position includes moving the article over or near the defect. This position may be referred to as the nominal orientation of the backup pad.

[0029] In block 230, the polishing article engages with the defect. Engaging with the defect may include sanding the defect area, as shown in block 232, or polishing the defect area, as shown in block 234. Engaging with the defect may also include changing various repair parameters of the backup pad, such as the speed 232 of the backup pad, the force 334 applied to the backup pad, the orientation offset 236 of the backup pad, and the final shape 238 of the repair produced by the sanding and polishing operations, relative to the nominal orientation.

[0030] In block 240, the defective area is cleaned. Cleaning may include wiping away any fluids used in sanding or polishing, as well as wiping away any debris. After the cleaning step, as shown in block 242, the tool can be re-engaged with the defect. For example, a dual-mount tool system may have a sanding unit and a polishing unit available to achieve the next repair step after cleaning has been achieved.

[0031] In block 250, the defective area is inspected to determine whether the repair is sufficient. If additional repair is required, method 200 may receive a new instruction, as indicated by arrow 260, and the method may be repeated. Inspecting the defect repair may include taking an image after the repair 252, which may be presented to the repair operator or stored as needed. Inspection may also include verifying the repair, as shown in block 254, which may include comparing the image before and after the repair, detecting whether the defect is visible / conspicuous to the human eye, or other preferred verification techniques.

[0032] Human repair technicians introduce some randomness into the repair process for defects, often resulting in the repair blending into the surrounding surface. This is more difficult to replicate in robotic repair systems due to the texture of the workpiece surface. When paint is applied to a workpiece surface in a manufacturing environment, environmental interference (such as vehicle movement or vibration, circulating air movement, and the properties of the applied paint) can produce textured surfaces, as shown in Figures 3A and 3B, which are commonly referred to as "orange peel" due to their appearance. Orange peel can also be intentionally added to clear coat surfaces, thereby texturizing various levels of orange peel and enhancing the aesthetics of various surfaces on a vehicle.

[0033] Generally, it is desirable to "completely" repair all defects present in the paintwork; however, the concept of complete repair is largely subjective and therefore difficult to define formally. Informally, a "complete" repair is interpreted as the final result being visually indistinguishable to the human eye from other defect-free areas of the workpiece surface, while the concept of optimal repair is interpreted as the best possible repair given some starting condition. For example, it is impossible to repair all defects to a complete state. Furthermore, since vehicle manufacturing is often an assembly line process, time is a crucial parameter regarding defect repair. In that case, efficient repair can be characterized as a repair that makes the defective area indistinguishable in the shortest possible time.

[0034] The human eye is inherently adept at noticing "perfect" or "regular" details, as well as abrupt transitions in boundaries or textures. This sensitivity to abrupt visual boundary transitions necessitates controlling the transition between repaired and unrepaired areas. One purpose of repair is to maximize the concealment of defect repairs on the finished product.

[0035] To better integrate repairs with the surrounding workpiece surface area, a reliable method for characterizing the surface area is required. Currently, orange peel is often characterized on a numerical scale from 1 to 10, for example, Figure 3A shows an example of "2" and Figure 3B shows an example of "5". However, different vehicle manufacturers may measure and characterize orange peel differently.

[0036] In addition, while handheld devices exist that can provide numerical characterization of orange peel texture, these devices cannot be calibrated to meet manufacturer standards. Furthermore, the handheld devices are not networked with the repair system and do not provide outputs that can be used to modify or adjust the repair plan. For example, the repair plan (described in more detail in concurrently pending U.S. Provisional Patent Application No. 62 / 941286, filed November 27, 2019) may need to be modified based on whether the surrounding orange peel texture is rated "1" or "8" so that the repair blends better with the surface.

[0037] In addition, while systems exist that can characterize orange peel texture in general, it is necessary to characterize the orange peel texture in the area immediately surrounding the defect, because it can vary, for example, from one area of ​​a car's hood to another. Some systems can rely on surface sampling to estimate the orange peel value in a given defect area, but this is only an estimate and may be inaccurate.

[0038] A system is needed that can automatically detect and characterize orange peel texture within the area surrounding a defect. This system must incorporate orange peel texture characterization into the repair plan generation sequence so that the repair plan is based on the orange peel texture on the surface immediately surrounding the defect. In some embodiments, the repair plan is automatically modified based on the automatic detection of orange peel texture, thereby making orange peel texture characterization an internal system parameter that does not require operator intervention. Alternatively or additionally, orange peel texture characterization may be presented to the operator in the context of the repair plan or as a separate component.

[0039] The system and method herein provides an integrated orange peel detection system within a robotic repair system for automatically detecting and characterizing orange peel in the area surrounding a defect on a vehicle surface. The system can also detect and characterize the contour of the vehicle surface. The orange peel system may include a camera and a structured light system, which may also include components used for detecting the defect itself. The system and method herein may also provide an orange peel detection system mounted on a robotic repair unit so as to be able to measure orange peel simultaneously with defect repair.

[0040] Figure 4 shows a repair plan generator according to an embodiment of this specification. The first imaging system 310 may have a light source 312, a camera (or other imaging device) 314, an analysis unit 316, and other components 318. The second imaging system 320 may also have a light source 322, a camera 324, an analysis unit 326, and other components 328. Although imaging systems 310 and 320 are shown as separate systems with separate components, in some embodiments, the systems may share components. For example, a single light source 312 may be used for both the defect detection system 310 and the orange peel texture evaluation system 320. In addition, a single analysis unit, together with a processing unit, may support both the defect detection system 310 and the orange peel texture evaluation system 320. Furthermore, although each of systems 310 and 320 is shown to have processors 316 and 326, it is explicitly conceivable that in some embodiments, systems 310 and 320 may simply capture images of the surface, and analysis and processing may be performed in a separate device that receives the images from systems 310 and 320.

[0041] The database 350 may be communicably linked to the repair plan generator 300, for example, via a network, or it may be incorporated into the memory of the repair plan generator 300. The database 350 may include, but is not limited to, information on polishing articles 352 that may be currently in use, or polishing articles 352 that may be available for use by the robotic repair unit 370. The database 350 may also include information on a number of defects 354 that may be detected on the surface, such as excessive paint, scratches, paint stains, particles trapped beneath the paint layer, or other defects. The database 350 may also store historical repair information 356, categorized, for example, by detected defects, repair trajectories used, or orange peel on the surface during repair, or other characteristics. The database 350 may also include information on available repair trajectories 358, and may include information on which trajectories may be suitable for a given defect, or information based on detected orange peel. As described in more detail in U.S. Provisional Patent Application No. 62 / 941286 filed November 27, 2019, the track may be based on a repair path designed to fuse the repair to the work surface of the vehicle. Database 350 may also include information about orange peel 362, such as calibration information for converting the calculated orange peel into a standard orange peel designation used by a given vehicle manufacturer. Other orange peel characteristic information may also be stored, such as deviations from the track required based on the depth or frequency of detected orange peel. Database 350 may also include other information 364.

[0042] The repair plan generator 300 is communicatively coupled to a robotic repair unit 370, which includes a robotic arm 372, and the robotic arm 372 is coupled to a force control unit 375 associated with an end effector 376. The robotic repair unit may also have other feature parts 378, such as a rail unit that allows movement along a production line. The end effector 376 may be coupled to a tool having an abrasive article 380 that contacts defects on the vehicle surface and performs an abrasive action. The robotic repair unit 370 is configured to move the abrasive article 380 along a path on the vehicle surface, applying a constant amount of force over a given time at a series of points on the vehicle surface. The force can also be applied at a given angle depending on the abrasive article. The repair plan generator 300 generates a trajectory that the robotic repair unit 370 will perform. The trajectory is generated based on the detected defect, known characteristics of the vehicle surface (e.g., color, number, and thickness of the coating) and known characteristics of the abrasive article (e.g., wear level), and orange peel characteristics at the location of the defect.

[0043] The defect detection unit 302 detects defects on the vehicle surface, for example, by receiving images from the first imaging system 310 and receiving analysis from the analysis unit 316 indicating the presence of defects. The defect analysis unit 304 can detect the type of defect and other characteristics of the detected defect. For example, a defect may be detected based on the refraction of light from the surface in an unexpected way. The defect analysis unit 304 can determine that the detected defect is a piece of debris trapped beneath one or more paint or clear coat layers. The trapped debris may affect the area on the vehicle surface detected by the defect analysis unit 304. The depth of the debris is also important for repair.

[0044] The orange peel analysis unit 306 receives images from the second image system 320 and can determine the intensity of the orange peel on the surface based on those images. The orange peel is characterized as a series of repeating ridges on the vehicle surface, similar to the external ridges of the orange peel. Referring again to Figures 3A and 3B, the orange peel refers to the undulating bumps within the glossy surface of the paint. The undulating bumps can be larger (Figure 3B) or smaller (Figure 3A) depending on the process conditions. Currently, many vehicle manufacturers characterize the orange peel using a single number, where 1 refers to a closely formed orange peel pattern and 10 refers to a very mild orange peel.

[0045] The orange peel characteristic evaluation unit 308 can characterize orange peel based on the analysis from the orange peel analysis unit 306. Many orange peel characterizations are currently performed using handheld devices or reference cards held up to the area, and may be performed periodically in paint shops to ensure that the expected level of orange peel is maintained in specific areas of the vehicle. This process is highly subjective. In addition to providing numerical data on orange peel on a vehicle, the orange peel characteristic evaluation unit 308 may also provide other characterization information, including the average height, frequency, and any other characteristics that may be useful in correcting the repair trajectory of the detected orange peel.

[0046] The track acquisition unit 332 acquires tracks for repair from, for example, the track data source 358. The acquired tracks may be based, for example, on the type or size of the detected defect and / or on the location of the defect, for example, whether the vehicle surface is relatively flat or curved. The track correction unit 334 may modify the acquired tracks based on the orange peel characteristics identified by the orange peel characteristic evaluation unit 308. The acquired tracks may be acquired from a database, for example, as a pre-generated track template, or from a track generator.

[0047] For example, if the scanning area is flat and orange peel is minimal, the total processing time for the selected trajectory can be reduced by 25%, compensating for the fact that a process to smooth the orange peel of the entire defect is not required. However, if the scanning area has orange peel that is much more pronounced than expected, the total time for the repair trajectory can increase by 20%, allowing the process to compensate for the depth and frequency of the orange peel.

[0048] The output communication unit 342 can also provide information regarding detected defects, characterized orange peel, and / or repair tracks to another device. For example, an operator can receive such information on a mobile device such as a smartphone or computer. In addition, such information may be presented on a user interface associated with a given repair area.

[0049] The repair plan communication unit 340 communicates the corrected trajectory to the robot repair unit 370. In some embodiments, the process of acquiring and correcting the trajectory for the robot repair unit 370 is performed in situ, so that the imaging systems (310, 320) verify the location of the defect, detect and characterize the orange peel on the vehicle surface around the defect, and provide a corrected trajectory for the robot repair unit 370 to perform all while the vehicle is in position for repair. During vehicle manufacturing, the vehicle may be on a line moving without stopping at any given station. In such implementations, the repair plan generator 300 needs to quickly collect information on defects and surfaces and determine an appropriate repair trajectory within minutes or seconds so that repairs can be completed in a timely manner. Even in manufacturing situations where it is assumed that the vehicle will stop at a fixed position at a repair station, long repair times are not acceptable.

[0050] The repair plan generator 300 may also include other components, such as any of the components described in U.S. Provisional Patent Application No. 62 / 941286, filed November 27, 2019.

[0051] The imaging systems 310 and 320 are shown in Figure 4 as two separate systems. However, as shown in Figure 5, they may share at least some components in some embodiments. In addition, either or both of the imaging systems 310 and 320 may be mounted on the robotic repair unit 370. For example, in one embodiment, the orange peel imaging system 320 may be mounted on the end effector unit 376 or elsewhere on the robotic repair arm 370 so that the orange peel is imaged, analyzed, and characterized immediately before the polishing article 380 contacts the vehicle surface to perform the repair. In some embodiments, both the first system 310 and the second system 320 are mounted separately from the robotic repair unit 370.

[0052] Figures 5A and 5B illustrate a repair imaging system according to embodiments of this specification. Figure 5A shows an imaging system 400 imaging a surface 410. The surface 410 is, for example, a reflective surface having one or more clear coating layers applied to the surface. The system has a distance 408 between the camera 406 and the surface 410. A light bar 404 provides light to a focal area 412 on the surface 410. The system 400 has a processing unit 402 that provides power, control and network access to the camera 406 and the light bar 404. In some embodiments, separate control and power are provided for each of the camera 406 and the light bar 404, as shown in Figure 5A. However, it is intended that both the camera 406 and the light bar 404 may be powered and controlled by a single power supply 402. Although a light bar 404 is shown, other illumination sources can also be explicitly conceived.

[0053] In some embodiments, the light bar 404 is positioned at a constant angle to the focal region 412, as shown in Figure 5A, with a preferred angle being one that positions the reflected bar in the central 25% of the image, thereby reducing optical distortion. In some embodiments, the light from the light bar 404 is brighter than any other potentially interfering light source. In some embodiments, blue and / or yellow hues are isolated from interference. In some embodiments, light diffusion makes individual LED point light sources undetectable.

[0054] In some embodiments, the distance 408 between the camera 406 and the surface 410 is relatively small, for example, less than 1000 mm, or even less than 900 mm, or even less than 800 mm, or even less than 700 mm, or even less than 600 mm, or even less than 500 mm, or even less than 400 mm, or even less than 300 mm.

[0055] In one embodiment, the length of the light 408 is greater than the focal region 412. The length of the light 408 may be 10 times the size of the focal region 412. In one embodiment, the length of the light 408 is approximately 450 mm, while the focal region 412 is 10 mm.

[0056] In one embodiment, system 400 is part of a robotic repair unit, for example, attached to a robotic arm. System 400 may be included as part of an end-effector assembly or mounted upstream of a force control unit. System 400 may also be configured to move while taking images of a surface; for example, system 400 may move across a vehicle surface to collect images. In another embodiment, system 400 may be configured to remain stationary while acquiring images.

[0057] Figure 5B shows another diagram of the camera system 450. The light source 460 is positioned between the first camera 470 and the second camera 470. In some embodiments, the cameras 470 and the light source 460 can pivot relative to the mount.

[0058] Cameras 406 and 470 are, in some embodiments, general-purpose cameras configured to acquire images, which can be stored in a database such as database 350 as part of repair database 356 (for example, before or after repair images) and may be retrieved later for display to an operator or for subsequent analysis. Cameras 406 and 470 may also be used for other monitoring purposes, for example, to monitor the repair process to detect wear on polishing articles or other functional problems of robotic repair units or other manufacturing line equipment.

[0059] Cameras 406 and 470 are also networked to a manufacturing line in some embodiments, allowing them to receive information about the vehicle being imaged. This makes it possible to calibrate cameras 406 and 570 to parameters of a given vehicle, including paint color, paint and clear coat thickness, and surface curvature where defects are detected. For example, many surfaces on a car are not flat. Knowing where defects are located on the car and accessing a CAD model of the vehicle allows for curvature information about the car at the point of the defect, enabling better analysis of the images acquired by camera 406 or 470.

[0060] Figure 6 shows a method for characterizing a workpiece surface according to an embodiment of this specification. Method 500 provides a characterized workpiece surface which may be useful for correcting or generating repair trajectories for repairing defects in or near the characterized workpiece surface.

[0061] In block 510, an image of the workpiece surface is acquired. In some embodiments, acquiring the image includes setting up a light source, as shown in block 502, which may include positioning a light bar or other suitable light source at a certain angle to the surface being imaged. The light source can also be identified, for example, by ensuring that sufficient reflection is captured from the correct area on the surface being imaged, as shown in block 504.

[0062] Image acquisition may include the camera acquiring an image, as shown in block 512. The camera may be a high-resolution camera, for example, such that there are enough pixels in the image to detect surface features. However, due to the number of pixels, characterizing orange peel on a surface can be time-consuming. Therefore, in some embodiments, it is desirable to reduce the required calculation size. Due to the short timeframe for detecting and repairing defects, the calculation of orange peel and other surface features must be performed quickly so as not to slow down the production line in order to adapt to defect repair.

[0063] As shown in block 514, the image can be sampled, which includes sampling pixels in the image to be processed. A smear process may be used to generalize the image so that the calculation is of a manageable size, as shown in block 516. Other processing may be performed, as shown in block 518. For example, confirmation of the presence of hotspots may be saved or output to the operator.

[0064] In block 520, contours are detected. Contours can be detected by focusing on the difference in nearby pixels that show a curve not associated with the planned curvature of the work surface. For example, an automobile may have an intended curvature known from the CAD model of the surface. Contours that are much smaller than the intended curvature, at least an order of magnitude smaller, may be detected. Contours can be detected by detecting a change in the color of pixels that indicate curvature. For example, referring again to the orange peel images in Figures 3A and 3B, contours can be detected in the change in the color of adjacent pixels.

[0065] In block 530, the contour is characterized. Characterizing orange peel typically involves calculating delta values ​​indicating the orange peel level, for example, the numbers "2" and "5" referring to Figures 3A and 3B, respectively. In addition, the calculated delta values ​​can be calibrated against the manufacturer's standards. Since many manufacturers have addressed the orange peel problem by developing their own orange peel scales, one advantage of using a networked orange peel characterization system in the manufacturer's repair system is its ability to output orange peel characterizations that conform to the standards used by the manufacturer. Other characterizations may be performed, as shown in block 536.

[0066] In block 540, the characteristic evaluation is output. Outputting the characteristic evaluation may include outputting the calculated or calibrated delta value to a user interface visible to the operator. Outputting may also include storing the calculated or calibrated delta value in a database. Outputting may also include providing the calculated or calibrated delta value to the repair plan generator.

[0067] Method 500 proceeds automatically in some embodiments. Method 500 can be initiated when a defect is detected, for example, by analyzing the same image used to detect the defect. In another embodiment, Method 500 proceeds automatically only when the repair robot is within a given distance of the defect, as shown in Figure 5A, so that, for example, the orange peel is calculated for a given defect at substantially the time of repair. Method 500 may also proceed automatically at different times, for example, at substantially the time when the defect is detected.

[0068] Figure 7 shows a method for generating a repair plan for a defect according to an embodiment of this specification. Method 600 may be used by a repair plan generation system to provide a repair plan for a robotic repair unit.

[0069] In block 610, defects on the workpiece surface are identified. Characterizing the defect may include the defect type, the severity of the defect, or its location on the vehicle. For example, the defect type may include dirt or scratches. The severity of the defect may refer to the area on the workpiece surface affected by the defect, the length of the defect, the height or depth of the defect, or other characteristics. The location of the defect may include the coordinate position on the workpiece surface of the vehicle. Characterizing the defect may also include the height of the defect relative to the workpiece surface, or its depth relative to the paint layer, such as whether the defect is located within the paint layer or the clear coat layer. Characterizing the defect may also include obtaining an image of the workpiece surface before repair. Other characteristics related to the defect may also be obtained.

[0070] Identifying a defect may include identifying the type of defect. Defect types may include excessive paint, embedded debris, scratches, dents, air pockets, stains, or other defects in the painted surface. Identifying a defect may also include identifying the location of the defect on the vehicle's surface. Identifying the location of a defect may also include associating the defect with a point on the vehicle's surface using a CAD model or surface mesh of the vehicle. Identifying a defect may also include identifying the affected area on the vehicle's surface; for example, embedded debris may affect not only the area of ​​the debris but also the surface area immediately surrounding the debris. Orange peel texture can be characterized using, for example, method 500 described with respect to Figure 6. However, other methods may also be preferable, provided they are performed automatically and consistently for each defect.

[0071] In block 620, the orange peel texture of the surface at the defect point is characterized. Previous attempts in repairing defects have relied on surface orange peel sampling, which identifies the average orange peel texture of a portion of the vehicle (e.g., the "hood" of a car) and assumes that the average is locally retained near the defect. The advantages of the systems and methods herein include the ability to detect and characterize orange peel texture at the defect point in order to better tailor repair plans for defects.

[0072] In block 630, a repair path is selected. Although block 630 is shown following block 620, it is explicitly conceivable that the order may be reversed. The path may be selected based on the detected defect size, defect type, defect location, vehicle color, or other preferred parameter. The selected path may include a shape 632 or route that the abrasive article follows for repair. Route 632 may be a closed path that starts and ends at the same point, or an open path that starts and ends at different points. In one embodiment, the location corresponds to a regular shape, including a circle, ellipse, rose, outer trochoid, or inner trochoid. In another embodiment, the location 1442 corresponds to an irregular shape. The shape may include a curved line or straight line, a concave portion or a convex portion, or other feature. Generating a path may also include generating one or more orientations. For example, the backup pad can make even contact with the work surface, thereby applying uniform pressure across the surface of the backup pad and on the work surface. In another embodiment, the backup pad is inclined with respect to at least a portion of the generated path. This inclination can be inward or outward and, in some embodiments, can be changed during repair.

[0073] The generated path is time-parameterized as shown by block 634 in order to generate a repair trajectory. Time-parameterizing the path involves assigning velocity and acceleration along the generated path. Generating the time-parameterized path requires satisfying dynamic constraints, such as the maximum velocity and acceleration achievable by the end effector tool and the robot itself, as well as jerk. Time-parameterizing may also involve verifying the constraints after the trajectory is generated to ensure that the robot and end effector can achieve the trajectory. The abrasive may be in contact with the workpiece surface at various positions along the path for a given amount of time 634, or may experience different velocities at different points along the path 632. The trajectory may also include other parameters 636 at different points, such as applied force and grinding angle.

[0074] In block 640, the selected trajectory is modified, taking into account the characterized orange peel at the defect points. For example, mild orange peel may allow for a reduction in time 634 at one or more points along the path 632, while more severe orange peel may require additional time 636 at different points.

[0075] Method 600 can be performed such that the modified trajectory can be immediately implemented by the robotic repair unit. In some embodiments, the modification to the selected trajectory may not be presented to the human operator at all, thereby leaving the orange peel characteristic evaluation as an internal parameter calculated and used only by the repair plan generator. However, in other embodiments, any information about the detected orange peel is presented to the operator or stored for later retrieval by the operator.

[0076] Figure 8 is a block diagram of the repair plan generation architecture. The remote server architecture 800 shows one embodiment of the implementation of the repair plan generator 810. In one embodiment, the remote server architecture 800 can provide computing, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system delivering the services. In various embodiments, the remote server can deliver services over a wide area network such as the Internet using an appropriate protocol. For example, the remote server can deliver applications over a wide area network, and they can be accessed through a web browser or any other computing component. The software or components, and corresponding data, shown or described in Figures 1 to 7 can be stored on servers located in remote locations. Computing resources in a remote server environment can be aggregated in remote data center locations, or they can be distributed. The remote server infrastructure can deliver services through shared data centers, which appear as a single access point to the user. Therefore, the components and functions described herein can be provided from remote servers in remote locations using the remote server architecture. Alternatively, they may be provided by a conventional server, installed directly on the client device, or provided in other ways.

[0077] In the embodiment shown in Figure 8, some items are the same as those shown in the previous figure. Figure 8 specifically shows that the repair plan generation system can be located at the remote server location 802. Thus, the computing device 820 accesses these systems through the remote server location 802. The operator 850 can similarly access the user interface 822 using the computing device 820.

[0078] Figure 8 also shows another embodiment of the remote server architecture. Figure 8 shows that it is also conceivable that some elements of the system described herein may be located at the remote server location 802, while others may not be located there. For example, storage 830, 840, or 860, or the repair system 870, may be located in a location separate from location 802 and accessed via the remote server at location 802. Regardless of where they are located, they may be directly accessed by computing devices 820 via a network (either a wide area network or a local area network), hosted at a remote site by a service, provided as a service, or accessed by a connected service located at a remote location. Data may also be stored in substantially any location and accessed intermittently by or transferred to the parties concerned. For example, physical carriers may be used instead of, or in addition to, electromagnetic carriers.

[0079] It should also be noted that the elements of the systems described herein, or parts thereof, can be deployed on a wide variety of different devices. Some of these devices include servers, desktop computers, laptop computers, embedded computers, industrial controllers, tablet computers, or other mobile devices such as palmtop computers, mobile phones, smartphones, multimedia players, and personal digital assistants.

[0080] Figures 9 and 10 show embodiments of mobile devices that can be used in the embodiments shown in the preceding figures.

[0081] Figure 9 is a simplified block diagram of an exemplary embodiment of a handheld or mobile computing device that can be used as a user or client handheld device 916 (such as computing device 820 in Figure 8) from which the system (or a part thereof) can be deployed. For example, the mobile device can be deployed in the operator compartment of computing device 820 for use in generating, processing, or displaying data. Figure 10 is another embodiment of the handheld or mobile device.

[0082] Figure 9 provides a schematic block diagram of the components of a client device 916 capable of performing some of the components shown and described herein. The client device 916 interacts with them or performs some and interacts with some. The device 916 is provided with a communication link 913 that enables the handheld device to communicate with other computing devices and, under some embodiments, provides a channel for automatically receiving information, such as by scanning. Examples of communication links 913 include enabling communication via one or more communication protocols, such as a wireless service used to provide cellular access to a network, and a protocol that provides a local wireless connection to a network.

[0083] In other embodiments, applications can be received on a removable Secure Digital (SD) card connected to interface 915. Interface 915 and communication link 913 communicate with a processor 917 (which may also be an embodiment of a processor) along bus 919, which is also connected to memory 921 and input / output (I / O) components 923, as well as clock 925 and position information system 927.

[0084] In one embodiment, the I / O component 923 is provided to facilitate input and output operations, and the device 916 may include input components such as buttons, touch sensors, optical sensors, microphones, touchscreens, proximity sensors, accelerometers, and compass sensors, and output components such as display devices, speakers, and / or printer ports. Other I / O components 923 can be used in a similar manner.

[0085] Clock 925 includes, exemplarily, a real-time clock component that outputs the time and date. It can also provide timing functions to processor 917.

[0086] Exemplary, the location information system 927 includes components that output the current geographical location of device 916. These may include, for example, a global positioning system (GPS) receiver, a LORAN system, a dead reckoning system, a cellular triangulation system, or other positioning systems. It may also include, for example, mapping software or navigation software that generates a desired map, navigation route, and other geographical functions.

[0087] Memory 921 stores the operating system 929, network settings 931, applications 933, application configuration settings 935, data storage 937, communication drivers 939, and communication configuration settings 941. Memory 921 may include all types of tangible volatile computer-readable memory and non-volatile computer-readable memory devices. It may also include computer storage media (described below). Memory 921 stores computer-readable instructions, which, when executed by the processor 917, cause the processor to perform steps or functions implemented by the computer according to the instructions. The processor 917 can also be activated by other components to facilitate their functions.

[0088] Figure 10 shows that the device may be a smartphone 1071. The smartphone 1071 has a touch-sensitive display 1073 that displays icons or tiles or other user input mechanisms 1075. Mechanisms 1075 can be used by the user for purposes such as running applications, making phone calls, and performing data transfer operations. Generally, the smartphone 1071 is built on a mobile operating system and offers more advanced computing power and connectivity than a feature phone.

[0089] Please note that other forms of device 1016 are possible.

[0090] Figure 11 is a block diagram of a computing environment that can be used in the embodiments shown in the preceding figures.

[0091] Figure 11 is an embodiment of a computing environment from which elements, or (for example) parts thereof, of the systems and methods described herein can be deployed. Referring to Figure 11, an exemplary system for implementing some embodiments includes a general-purpose computing device in the form of a computer 1110. The components of the computer 1110 may include, but are not limited to, a processing unit 1120 (which may include a processor), a system memory 1130, and a system bus 1121 that connects various system components, including the system memory, to the processing unit 1120. The system bus 1121 can be one of several types of bus structures, including a memory bus or memory controller, peripheral bus, and local bus, using any of various bus architectures. The memory and programs described with respect to the systems and methods described herein can be deployed in the corresponding parts of Figure 11.

[0092] Computer 1110 typically includes various computer-readable media. Computer-readable media can be any available media accessible by computer 1110, and include both volatile / non-volatile media and removable / non-removable media. For example, but not limited to, computer-readable media may include computer storage media and communication media. Computer storage media are different from and do not include modulated data signals or carrier waves. Computer storage media include hardware storage media, including both volatile / non-volatile removable / non-removable media, implemented in any way or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disc storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices, or other magnetic storage devices, or any other media that can be used to store desired information and are accessible by computer 1110. A communication medium can embody computer-readable instructions, data structures, program modules, or other data in a transfer mechanism, and includes any information distribution medium. The term "modulated data signal" means a signal in which one or more of its characteristics are set or modified in a manner that encodes information within the signal.

[0093] System memory 1130 includes computer storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 1831 and random access memory (RAM) 1132. A basic input / output system (BIOS) 1133, which includes basic routines useful for transferring information between elements within the computer 1110 during startup, is typically stored in ROM 1131. RAM 1132 typically includes data modules and / or program modules that are immediately accessible by and / or currently running on the processing unit 1120. As an example, but not limited to, Figure 11 shows an operating system 1134, an application program 1135, other program modules 1136, and program data 1137.

[0094] Computer 1110 may also include other removable / non-removable volatile / non-volatile computer storage media. As a mere example, Figure 11 shows a hard disk drive 1141, a non-volatile magnetic disk 1152, an optical disk drive 1155, and a non-volatile optical disk 1156 that read from or write to non-removable non-volatile magnetic media. The hard disk drive 1141 is typically connected to the system bus 1121 via a non-removable memory interface such as interface 1140, and the optical disk drive 1155 is typically connected to the system bus 1121 via a removable memory interface such as interface 1150.

[0095] Alternatively, or furthermore, the functions described herein can be performed, at least in part, by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (e.g., ASICs), application-specific standard products (e.g., ASSPs), system-on-a-chip systems (SOCs), and complex programmable logic devices (CPLDs).

[0096] The drives discussed above and shown in Figure 11, and their associated computer storage media, provide storage for computer-readable instructions, data structures, program modules, and other data relating to the computer 1110. In Figure 11, for example, the hard disk drive 1141 is shown as storing the operating system 1144, application programs 1145, other program modules 1146, and program data 1147. Note that these components may be the same as or different from the operating system 1134, application programs 1135, other program modules 1136, and program data 1137.

[0097] The user can input commands and information to the computer 1810 via input devices such as a keyboard 1162, a microphone 1163, and a pointing device 1161 such as a mouse, trackball, or touchpad. Other input devices (not shown) include joysticks, gamepads, satellite receivers, and scanners. These input devices and other input devices are often connected to the processing unit 1120 via a user input interface 1160 coupled to the system bus, but they can also be connected via other interfaces and bus structures. A visual display 1191 or other types of display devices are also connected to the system bus 1121 via an interface such as a video interface 1190. In addition to the monitor, the computer may also include other peripheral output devices such as a speaker 1197 and a printer 1196, which can be connected via an output peripheral interface 1195.

[0098] Computer 1110 operates in a networked environment using logical connections such as a Local Area Network (LAN) or Wide Area Network (WAN) to one or more remote computers, such as remote computer 1180.

[0099] When used in a LAN network environment, computer 1110 is connected to LAN 1171 via a network interface or adapter 1170. When used in a WAN network environment, computer 1110 typically includes a modem 1172 or other means for establishing communication over WAN 1173, such as the Internet. In a networked environment, program modules can be stored in remote memory storage devices. Figure 11 shows, for example, that a remote application program 1185 may reside on a remote computer 1180.

[0100] An imaging and repair system is presented, which includes a first imaging system configured to image defects on a workpiece surface. The imaging and repair system also includes a second imaging system configured to image and characterize the workpiece surface at the defect. The system also includes a defect repair processor configured to select a repair plan based on the defect type. The system also includes a defect repair unit configured to modify the selected repair plan based on the characterization of the workpiece surface. The system also includes a defect repair tool configured to automatically execute the modified repair plan.

[0101] The system may be implemented such that the first imaging system comprises a first camera configured to acquire multiple first images of the workpiece surface. The multiple first images are stored in a data source.

[0102] The system may be implemented such that a second imaging system includes a second camera configured to acquire multiple second images of the workpiece surface. The multiple second images are stored in a database. The second camera is separate from the first camera.

[0103] This system can be implemented such that the second set of images includes images of the workpiece surface adjacent to the defect.

[0104] This system may also include a light source.

[0105] The system may also include a defect detection unit configured to detect defects in a first set of images.

[0106] This system may be implemented such that detecting defects includes detecting the defect type, defect size, defect severity, or defect location.

[0107] This system may also be implemented to include a defect analysis unit configured to correlate the location of defects with their position on a 3D representation of the workpiece surface.

[0108] This system may be implemented such that a second imaging system is instructed to image the workpiece surface based on the above location.

[0109] This system can be implemented such that characterizing the workpiece surface includes identifying the delta value of the orange peel texture.

[0110] This system may also be implemented to include an orange peel processor configured to sample pixels from multiple second images and calculate delta values ​​based on the sampled pixels.

[0111] This system may be implemented to include an orange peel processor configured to smear images in multiple second images and calculate delta values ​​based on the sampled pixels.

[0112] The system may be implemented such that the defect repair tool includes a drive robot arm, a force control unit coupled to the drive robot arm, an end effector coupled to the force control unit, and a polishing tool coupled to the end effector. The polishing tool is configured to engage with the workpiece surface at the defect.

[0113] This system may be implemented to include an imaging system mount configured to be coupled to a second imaging system.

[0114] This system can be implemented such that the second imaging system includes a light source.

[0115] This system can be implemented such that the light source is tilted relative to the workpiece surface.

[0116] This system can be implemented such that the second imaging system is within 1 meter of the surface when it takes a second set of images.

[0117] This system can be implemented such that the work surface is a vehicle. The characterization of the work surface includes a flat surface, and based on the characterized flat surface, the process time of the modified repair plan is shorter than that of the selected repair plan.

[0118] An orange peel imaging system is presented, which includes an orange peel light source configured to direct light onto a region on the surface of a workpiece. The system also includes an orange peel camera configured to acquire an orange peel image of the region. The system also includes a process camera configured to acquire an image of defects within the region. The system also includes a processor configured to analyze the orange peel image, detect contours within the region, calculate delta values ​​for the detected contours, and output the calculated delta values.

[0119] The orange peel imaging system may also include an orange peel communication unit configured to communicate the output delta value to a second system.

[0120] The orange peel imaging system may also include a mount configured to connect the orange peel camera to a robotic repair unit.

[0121] The orange peel texture imaging system can be implemented such that the robotic repair unit is a drive unit that can position the orange peel texture camera within 2 meters of the workpiece surface.

[0122] The orange peel texture imaging system can be implemented such that the robotic repair unit includes a force control unit coupled to an end effector. The end effector is coupled to an abrasive that contacts the workpiece surface at the detected defect.

[0123] The orange peel skin imaging system can be implemented such that analyzing the orange peel skin image includes sampling pixels within the image of the above-mentioned region.

[0124] The orange peel skin imaging system can be implemented such that analyzing the orange peel skin image includes smearing within the image of the aforementioned region.

[0125] The orange peel skin imaging system can be implemented so that images of the above-mentioned region are stored in a database.

[0126] The orange peel skin imaging system can be implemented so that images of defects are stored in a database.

[0127] The orange peel imaging system may also include a repair generation system configured to generate repair plans for defects. The repair plans are generated, at least in part, by the analysis of the orange peel images.

[0128] A method for characterizing orange peel texture on a vehicle surface is presented. This method includes acquiring an image of a region of the vehicle surface using a camera. This method also includes processing the acquired image using a processor to detect contours within the region. This method also includes determining the delta values ​​of the contours using a processor. This method also includes outputting the analysis results using a communication channel. The analysis results include a display of the detected contours and the detected delta values ​​of the contours.

[0129] This method can be implemented such that processing the acquired image includes color correction of the image.

[0130] This method can be implemented such that processing the acquired image includes applying a Gaussian blur to the image.

[0131] This method may also include determining the scale of the detected contours.

[0132] This method may also include directing a focused light beam from a light source onto a region of the vehicle surface.

[0133] This method can be implemented such that the camera is within the range described above. This range is within 1 meter.

[0134] This method can be configured such that the camera is positioned within the above range by a driven robot arm.

[0135] This method can be implemented such that the camera automatically moves within the above range in response to a detected defect within that area.

[0136] This method can be implemented such that the acquisition step, processing step, identification step, and output step are performed automatically in response to the detected defects within the above-mentioned region.

[0137] This method can be implemented such that the communication channel outputs the analysis results to a repair plan generator, which generates a repair plan for the detected defect based on the detected contour and detected delta value.

[0138] A method for generating a repair process for defects on a workpiece surface is presented. This method includes detecting defects within a region on the workpiece surface. It also includes characterizing the region on the workpiece surface. Based on the detected defects, the method includes selecting a repair plan. Furthermore, the method includes modifying the selected repair plan based on the characterized parameters to obtain the repair process. The characterization, selection, and modification steps are performed automatically based on the detected defects on the workpiece surface.

[0139] This method may be implemented so that detecting defects includes detecting the defect type, defect size, or defect severity.

[0140] This method may be implemented such that detection includes detecting whether the defect is a repairable defect.

[0141] This method may be implemented such that selecting a repair plan includes selecting a repair path along the workpiece surface for the abrasive article.

[0142] This method can be implemented such that the repair path includes a time-parameterized series of positions on the workpiece surface with respect to the abrasive, and the abrasive experiences pressure, velocity, contact angle, or duration at each position.

[0143] This method may be implemented such that modifying the selected repair plan based on the characterized area includes modifying the experienced pressure, velocity, contact angle, or duration.

[0144] This method can be implemented to further include characterizing a region of the work surface using an orange peel camera to acquire an image of the region of the work surface, analyzing the image of the region to detect contours, and calculating the delta value of the detected contours.

[0145] This method may also include calibrating the calculated delta value.

[0146] This method can be implemented such that the revised repair plan is provided to the robotic repair unit for automated execution.

[0147] This method may also include outputting an image of the detected defect.

[0148] This method may also include outputting the characteristics of the workpiece surface as numerical values.

[0149] This method can be implemented such that the above-mentioned region is characterized as being flatter than expected, and the selected repair plan is modified, which includes reducing the repair time to compensate for the flatter surface.

[0150] This method can be implemented such that the above-mentioned area is characterized as having a more pronounced orange peel texture than expected. Modifying the selected repair plan includes increasing the repair time to be compensated.

Claims

1. An imaging and repair system, A first imaging system configured to image and detect defects on a workpiece surface, wherein the first imaging system comprises a first camera configured to acquire a plurality of first images of the workpiece surface. A second imaging system configured to characterize the orange peel texture of the workpiece surface, including imaging the orange peel texture of the workpiece surface in a region adjacent to the defect and identifying the delta value of the orange peel texture, A defect repair processor configured to select a repair plan based on the defect type, A defect correction unit configured to modify the selected repair plan based on the evaluation of the orange peel texture characteristics of the workpiece surface, A defect repair tool configured to automatically execute the aforementioned revised repair plan, Equipped with, The second imaging system is an imaging and repair system that is instructed to image the workpiece surface based on the location of the detected defect received from the first imaging system.

2. The imaging and repair system according to claim 1, wherein the second imaging system comprises a second camera configured to acquire a plurality of second images of the work surface, the second camera being separate from the first camera.

3. It is a Yuzu Skin Processor, Pixels are sampled from the images in the plurality of second images, Based on the sampled pixels, the delta value is calculated. The imaging and repair system according to claim 2, further comprising an orange peel skin processor configured as such.

4. It is a Yuzu Skin Processor, The images in the aforementioned plurality of second images are blurred, Based on the blurred image, the delta value is calculated. The imaging and repair system according to claim 2 or 3, further comprising an orange peel skin processor configured as such.

Images