Automatic vehicle repair system

The defect detection and ranking system automates defect identification and prioritization, using robotic repair to optimize the vehicle painting process, addressing human bias and time inefficiencies in manual correction.

JP7954264B2Active Publication Date: 2026-11-193M INNOVATIVE PROPERTIES CO
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2025169101
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-10-28
Filing Date
2025-10-07
Publication Date
2026-11-19
Estimated Expiration
2040-10-20

AI Technical Summary

Technical Problem

The manual process of correcting defects in vehicle painting is time-consuming and prone to human bias, making it difficult to automate and optimize the repair process efficiently.

Method used

A defect detection and ranking system that uses image capture and analysis to identify defects with coordinates, severity, and type, integrating vehicle specifications and market information to prioritize repairs, and automates the repair process with robotic systems.

Benefits of technology

Enhances repair efficiency by reducing human bias, improving accuracy, and minimizing repair time, ensuring vehicles meet aesthetic standards before sale.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007954264000003
    Figure 0007954264000003
  • Figure 0007954264000004
    Figure 0007954264000004
  • Figure 0007954264000005
    Figure 0007954264000005
Patent Text Reader

Abstract

Providing a vehicle assembly line defect detection and ranking system SOLUTION: The system includes an image capture device that captures a plurality of images of a vehicle on a vehicle assembly line and a defect detector that analyzes the plurality of captured images to detect a plurality of defects on a surface of the vehicle. Each of the plurality of defects has an associated x-y-z coordinate location, a defect type, and a defect severity. The system also includes a data store including vehicle specifications for vehicles on the vehicle assembly line and defect priorities based on the vehicle specifications. The system also includes a defect prioritization generator configured to receive the plurality of defects from the defect detector, retrieve the vehicle specifications and the defect priorities, apply the defect priorities to the plurality of defects, and output a prioritized list of defects. The defect prioritization generator outputs the prioritized list of defects to an output device associated with the vehicle assembly line.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Many different types of vehicles are manufactured around the world every year. Vehicles such as passenger cars, trucks, passenger vehicles, and aircraft often have one or more coatings of paint and clear coat applied during the manufacturing process. The process of correcting defects that occur in the painting process is in many cases a time-consuming manual task.

Summary of the Invention

[0002] A defect detection and ranking system for a vehicle assembly line is provided. The system includes an image capture device that captures a plurality of images of a vehicle on the vehicle assembly line. The system also includes a defect detector that analyzes the plurality of captured images and detects a plurality of defects on the surface of the vehicle based on the analysis. Each of the plurality of defects has an associated x-y-z coordinate position, defect type, and defect severity. The system also includes a data store containing vehicle specifications for the vehicle on the vehicle assembly line and defect priorities based on the vehicle specifications. The system also includes a defect prioritization generator configured to receive the plurality of defects from the defect detector, retrieve the vehicle specifications and the defect priorities, apply the defect priorities to the plurality of defects, and output a prioritized defect list. The defect prioritization generator outputs the prioritized defect list to an output device associated with the vehicle assembly line.

Brief Description of the Drawings

[0003] [Figure 1] 1 is a schematic conceptual diagram of a manufacturing environment for detecting and correcting defects on a vehicle surface. [Figure 2] 1 is a block diagram of a method for defect detection and remediation in a manufacturing environment. [Figure 3] 1 is a block diagram illustrating a vehicle remediation system. [Figure 4] 1 is a block diagram of a defect detection and ranking system. [Figure 5] This is a block diagram of a networked vehicle repair system. [Figure 6] This is a block diagram showing a method for prioritizing defects for repair. [Figure 7] This shows the vehicle repair system. [Figure 8] This is an example user interface for a vehicle repair system. [Figure 9] This is a block diagram of the defect detection and ranking system architecture. [Figure 10] An example of a mobile device that can be used in the embodiment shown in the previous figure is shown. [Figure 11] An example of a mobile device that can be used in the embodiment shown in the previous figure is shown. [Figure 12] This is a block diagram of a computing environment that can be used in the embodiment shown in the previous figure. [Figure 13A] Examples of defects and illustrative feedback are shown, which will be explained in more detail in the embodiments. [Figure 13B] Examples of defects and illustrative feedback are shown, which will be explained in more detail in the embodiments. [Figure 14A] Examples of defects and illustrative feedback are shown, which will be explained in more detail in the embodiments. [Figure 14B] Examples of defects and illustrative feedback are shown, which will be explained in more detail in the embodiments. [Figure 15A] Examples of defects and illustrative feedback are shown, which will be explained in more detail in the embodiments. [Figure 15B] Examples of defects and illustrative feedback are shown, which will be explained in more detail in the embodiments. [Figure 16] Examples of defects and illustrative feedback are shown, which will be explained in more detail in the embodiments.

[0004] Structural modifications to the embodiments described herein can be made without departing from the scope of the art of this disclosure. The figures are not necessarily to a constant scale. Similar numbers used in the figures indicate similar components. However, the use of a number to indicate a component in a given figure is not intended to limit the component in another figure that is numbered the same. [Modes for carrying out the invention]

[0005] The automation of vehicle manufacturing processes has been ongoing for decades. The vehicle manufacturing industry has begun to introduce robots to replace manual work that was previously performed by human workers. Automation can increase the efficiency of manufacturing processes and reduce workplace accidents by using robots to accomplish difficult, dangerous, or time-consuming tasks. Robots can also improve precision and reduce waste.

[0006] However, one area that has proven difficult to automate is the repair of defects, particularly surface defects that can be caused or exacerbated during the painting process. As will be explained in more detail with respect to the figure, a vehicle is often coated with several paints, which may include several coatings of primer, color, and clear coat. Defects on the surface of a vehicle can be caused by impurities, coagulation, and flow problems in the painting process, as well as drying irregularities.

[0007] Vehicle buyers have aesthetic standards that the vehicle they intend to purchase must meet. For example, someone buying a new passenger car expects to be certain that the car is free from scratches, dents, or other defects. Significant time and effort are devoted to inspecting passenger cars and repairing any defects found before they leave the manufacturing facility.

[0008] The embodiments described herein illustrate defect detection and repair processes when applied to the manufacture of automobiles such as passenger cars and trucks. However, it is clearly intended that many of the embodiments described herein are equally applicable to other vehicles such as boats and other vessels, passenger cars and locomotives, airplanes and other flying vehicles, and rockets.

[0009] As used herein, the term “paint” is intended to include primers, pigment layers, clear coats, and any other coatings that may be applied to passenger vehicles.

[0010] As used herein, the term “defect” is intended to refer to any defect that may be present on a vehicle after the painting process. Examples of defects include, but are not limited to, scratches, dents, debris trapped within the paint coating, excess or uneven application of paint, haze, inconsistent gloss, or other surface-based defects.

[0011] Figure 1 is a conceptual diagram illustrating an exemplary manufacturing environment for detecting and correcting defects on a vehicle surface. In manufacturing environment 100, vehicle 10 is first inspected for defects at station 110, a repair plan is generated, and then it is sent to station 150 for repair. In many manufacturing environments, stations 110 and 150 are sequential so that defects are identified and then repaired. However, in some embodiments, these steps can be performed at least partially simultaneously.

[0012] Vehicle 10 has several defects 20, 12, 14, 16, 22, and 18 after the painting process. Depending on the manufacturing facility, there may be more or fewer defects, and the defects may be located in different places and of different types and severity.

[0013] Each defect on the vehicle 10 has a defect type (e.g., scratch or dent) and a severity level (e.g., severe or minor). The defect also has a location on the vehicle 10. Depending on the type of the vehicle 10, different defects may be more or less significant according to their type, severity, and location. For example, for a small passenger car 10, a defect 18 near the door handle is far more likely to be a concern for potential customers than a defect 12 on the roof of the passenger car 10 or a defect 16 under the front bumper.

[0014] The time available during manufacturing to detect and repair defects is limited. If more defects are detected than can be repaired in a timely manner, the vehicle 10 may need to be diverted for repair instead of being repaired during assembly.

[0015] As shown in Figure 1, the defect detection station 110 includes a vision system having one or more cameras 40 and one or more light sources 50a, 50b, and the like. The vision system operates by detecting and subsequently mapping irregularities on the surface of the vehicle 10. In some embodiments, the defect detection station 110 is part of a moving assembly line, as indicated by the conveyor belt 30 and the transport mechanism 32. However, in other embodiments, the defect detection station 110 accommodates the vehicle 10 in a stationary position.

[0016] Defect detection systems are known in the art. For example, U.S. Patent Application Publication No. 2019 / 0096057 filed on May 9, 2018, and No. 2017 / 0277979 filed on March 22, 2016, both owned by Innovision Software Solutions Inc., describe several exemplary defect detection systems.

[0017] Once defects are identified, they are repaired at a repair station 150. The repair station 150 may include a robot 160 that can move using joints 164. Abrasive material can be applied to the vehicle using an applicator 162.

[0018] However, although the robot 160 is shown to perform repairs in FIG. 1, in many cases, repair plans are formulated by a human operator. In many cases, at least part of the repair is also performed by a human. However, human operators may introduce bias into the repair process. For example, a human operator may prefer to start at the front of the vehicle, for example, repair defect 16, and end at the rear by repairing defect 20 last. Similarly, a human operator may preferentially choose to repair defect 12 on the top of the vehicle before defects 18 or 22.

[0019] Using the robotic repair system 160 can also provide advantages over human repair technicians. The robotic repair system 160 uses a coordinate-based positioning system provided from the defect detection station 110 to access and repair defects. Humans rely on vision to locate defects. However, when a polishing pad with an abrasive or other abrasive material is applied, the defect may no longer be visible to the human eye. The robotic repair system 160 does not need to aim at the defect during repair, but instead relies on coordinates to ensure that the repair is performed at the correct location. This results in more reliable repairs, and in many cases a smaller repair area, because the robot 160 is more accurate.

[0020] During vehicle manufacturing, time is of the utmost importance, and selecting which defects to repair and which to leave as-is can make the difference between whether a vehicle is declared ready for sale or not.

[0021] Innovation is needed to determine a repair plan 120 for repairing detected defects on the vehicle surface. The repair plan must consider known information about the vehicle 10 (make, model, color, and paint application parameters), as well as information about the detected defects (number, location, severity) and the destination market (country, distributor, prospective customer) in order to determine which defects should be repaired and in what order. The system can also provide recommendations or instructions to the robotic repair system 160 regarding which polishing products to use to repair the defects.

[0022] Figure 2 is a block diagram of a defect detection and repair method in which embodiments of the present invention may be useful. Method 200 is applicable to a wide variety of vehicles, including passenger cars, trucks, boats, airplanes, trains, helicopters, and the like.

[0023] In block 210, a vehicle is manufactured. Manufacturing includes the vehicle's structural assembly 212. Manufacturing also includes one or more coats of paint 214. After the paint has been applied, the vehicle undergoes a baking process 216 to facilitate uniform paint drying. Movement between the painting station and the baking station may introduce dust or other debris into the vehicle's surface, which may result in defects. In addition, the vehicle may be scratched or dented during assembly 212, painting 214, or baking 216. The paint may also be applied unevenly, resulting in excessive paint buildup in some areas, an inconsistent surface, haze, or an inconsistent gloss.

[0024] In block 220, defects are detected. The defect detection system often includes one or more light sources that illuminate the vehicle. A scanning system is also included, as shown in block 222. The scanning system detects defects on the paint surface and determines the location of each defect, as shown in block 224. A type is identified for each defect, as shown in block 226. For example, a defect may be identified as, for example, a dent, scratch, or excess paint on the surface. A defect may also be identified as a raised surface, such as excess paint or debris trapped in the paint, or a concave surface, such as a dent or scratch. In some embodiments, the location of a defect includes the coordinate location of the identified defect on the vehicle, as well as the affected area. For example, a small piece of debris may affect only one square centimeter of paint, while a larger piece may affect one square inch. A defect is also classified as having a severity level of 228.

[0025] In blocks 230 and 240, identified defects are repaired. Traditionally, defects are repaired by human repair technicians. Identified defects are provided to the technician from a visual system, and the technician determines which defects to repair based on printouts and visual inspection. Often, the visual inspection is used to determine whether the vehicle can be repaired during the assembly process or whether the vehicle needs to be detoured for more intensive repair. Detours result in longer assembly times and more man-hours for the vehicle, leading to higher costs. The systems and methods described herein are designed to increase the efficiency of the repair process. In addition, the systems and methods designed herein also facilitate the use of robotic repair systems, thereby increasing the accuracy of repairs, reducing the amount of repaired products required, and shortening the overall repair time.

[0026] In block 230, defects are often sanded using abrasive products in combination with water, abrasives, or other fluids. The defective area can be cleaned before or after sanding. After water or another fluid is applied, the defective area can be wiped dry.

[0027] In block 240, the sanded defects are polished. Polishing may involve using a polishing or non-polishing product in combination with water, an abrasive, or another fluid. Before or after polishing, the defect area may be cleaned or wiped dry. After polishing, the repaired defects can be inspected as shown in block 242. This may be done as part of the repair verification process.

[0028] In Block 250, the vehicle is sent out of the assembly process for sale. This often involves transporting the vehicle to the dealer across state or national borders. Upon arrival at the dealer, the vehicle is often inspected again, as shown in Block 252. Because inspection at the point of sale is expected, the selection of which defects to repair during Blocks 230 and 240 is crucial to ensure that the vehicle is not deemed unacceptable by the dealer.

[0029] Figure 3 is a block diagram of a vehicle repair system. In some embodiments, the vehicle repair system 300 can be implemented as part of a vehicle assembly and manufacturing system.

[0030] In one embodiment, the vehicle specification 310 can be stored in a data store. The vehicle specification is available to the vehicle currently being evaluated by the defect detection and ranking system 350. The vehicle specification includes, for example, the vehicle make 312, the model 314, and the color 316. For example, a black Honda® CR-V® may be on the assembly line. The vehicle may have specific specifications, for example, the number of doors and their positions 322 (such as a two-door or four-door passenger car or truck). Different automobiles may have different bonnet detail details 324 or trunk detail details 326. For example, for aesthetic and aerodynamic reasons, different passenger car makes and models may have bonnets of different designs with different contours. However, many embodiments described herein refer to passenger cars and trucks, and many embodiments are equally applicable to other types of vehicles. For example, an airplane or helicopter may have specifications relating to nose design 328 and tail design 332. Other specifications 318 may also be available for a given vehicle.

[0031] The defect detection and ranking system 350 includes a defect detection system 352 and a defect prioritization system 352. System 350 receives vehicle specifications 310, and based on several vehicle specifications and defects detected by the defect detection system 352, the defect prioritization system provides a set of prioritized defects for repair to the defect repair system 360.

[0032] For at least some vehicles, market information 340 may also be important to the prioritization system 354. Market information may include end destination 342, passenger car grade 344, customer 346, or other information. For example, a passenger car destined for the Japanese or European market may need to have fewer untreated defects to be considered acceptable by the dealer. A passenger car with a higher passenger car grade 344 may also have a higher standard for repaired defects, as well as fewer untreated defects, if present. For example, a Lamborghini® is considered a higher passenger car grade than a Ford® Ranger®. Customer information may also be available and important for prioritizing defects. For example, customers in the Japanese market are generally shorter than customers in the US market. Therefore, passenger cars destined for different markets may have different defect prioritizations depending on the expected customer range.

[0033] The defect detection system 352 is configured to detect a set of defects related to the vehicle. The defect detection system 352 is configured to detect defects using a visual system. The defect detection system 352 outputs a set of defects to the defect prioritization system 354. Each defect in the set of defects is associated with a defect type, defect location, and defect severity.

[0034] The defect prioritization system 354 receives a set of defects and prioritizes them for repair based on at least some of the specifications 310 and market information 340. The defect prioritization system 354 can also make decisions on which defects should be repaired and which should be left unrepaired. The defect prioritization system can also provide recommendations on whether the vehicle should be repaired as part of the assembly line process or detoured for a separate repair process. Detours may be recommended automatically based on defect thresholds, such as the length of the detected defect or the size of the trapped fragment. Detours may also be recommended based on the number of defects, the number of defect types, or other thresholds.

[0035] Depending on the vehicle specifications 310 and market information 340, the defect prioritization system 354 generates a defect repair list or defect repair sequence, ranking defects differently depending on the specifications from which they are taken. For example, for a taller passenger car, defects on the roof have a lower priority. For a silver passenger car, scratches are less noticeable and therefore have a lower priority than scratches on a red passenger car. On a black surface such as the hood or trunk of a black passenger car, all surface defects are more noticeable and therefore have a higher priority for repair. Defects on the driver's side door, especially around the handle, have a higher priority than defects in other locations on the vehicle. Defects may have a lower priority if they are below the exterior trim line than if they are above the exterior trim line. A defect on the hood of a clear-coated jet-black painted sedan may be far more visible than the same defect on the hood of the exact same white passenger car. A defect in the underside of a vehicle on a truck is far more readily accepted than the same defect in a passenger car, due to the truck's perceived purpose and durability, as well as its rough expected lifespan.

[0036] The defect detection and ranking system 350 also communicates with a defect repair system 360 in some embodiments. In some embodiments, the defect repair system 360 may be a robotic repair system. Information regarding the performance of the robotic repair unit, such as the time it takes to repair a given defect and the time it takes to move between defects, may also be considered by the defect prioritization system 354 when creating a defect repair plan. The defect repair system may have a sanding robot 362, which includes an abrasive applicator 364 and a water and / or abrasive dispenser 366. The defect repair system may also have a polishing robot 370, which includes an abrasive applicator 372 and a water and / or abrasive fluid dispenser 374. The sanding robot 362 and the polishing robot 370 may be separate robotic systems or may be part of a single robotic repair system. The defect repair system 360 may also have a verification system 380 that can inspect the repaired defects for quality.

[0037] Figure 4 is a block diagram of the defect detection and ranking system. The defect detection and ranking system 400 is similar to the defect detection and ranking system 350 in Figure 3. The defect detection and ranking system includes a defect detection system 410 that outputs a preliminary list 420 of defects and locations, and a defect prioritization system 430 that outputs a prioritization list 480 of defects and locations.

[0038] The defect detection system 410 includes one or more cameras 412, one or more illumination systems 414, and an image stitching and defect detection system 416. The illumination system 414 illuminates a portion of the vehicle while the cameras 412 capture images or video of the vehicle. The image stitching and defect detector 416 includes a processor that examines the captured images or video to detect defects on the imaged surfaces of the vehicle. Examining may, in some embodiments, include stitching the images.

[0039] The detected defects are output from the defect detection system 410 as a defect list 420. The preliminary defect list includes the location classification and severity for each defect. The defect detector 416 may output the location in coordinate system format. The coordinate system format of the preliminary defect list 420 can be provided to the repair system 494 for final repair.

[0040] The defect prioritization system 430 receives the preliminary defect list 420. The defect prioritization system uses the information stored in the vehicle information database 440, the defect database 450, and the feedback database 470 to evaluate and prioritize the preliminary defect list 420.

[0041] The vehicle information database 440 contains information about the vehicle being evaluated. A given vehicle has a model 444, a color 446, detailed specifications 442, and other features 448.

[0042] The defect database 450 includes defects identified in the preliminary list of defects 420. The identification and repair system can be used to identify and repair many different types of surface defects. Some examples of defects that can be identified using the defect detection system 410 include scratches 452, dents 454, excess paint 456, haze 462, irregular surface 464, inconsistent gloss 466, or other defects 458.

[0043] In some embodiments, the defect detection and ranking system 400 includes some machine learning capabilities so that the generated repair plans can be improved over time. The user-driven feedback database 470 receives and stores user feedback provided about previous vehicles in order to improve future use.

[0044] For example, the user can provide feedback regarding the classified defect severity 472, for instance, indicating that the severity was incorrect. In addition, feedback may be provided regarding the assigned defect type 474. Vehicle details 476 may also be corrected using the user feedback interface. Defect location 478, as well as potential areas affected by a given defect, can also be corrected. Generally, defect details are identified based on images of the vehicle captured by the defect detection system 410. User feedback helps ensure that defect identification, classification, and location information become more accurate over time.

[0045] A list of corrected defects 480 is provided. The list of corrected defects 480 may be presented on the user interface so that the operator can review and provide feedback 482, and the feedback 482 may be incorporated into a user-driven feedback database 470 for future use.

[0046] If defects are found in a vehicle, a repair decision 490 must be made. If there are few defects and / or if they can be easily repaired, the vehicle can proceed through the field repair process 494. However, if there are too many defects or if the defects cannot be easily repaired on the assembly line, the vehicle may be detoured for more intensive repair.

[0047] In one embodiment, a list of adjusted defects 480 is provided along with recommendations for repair. The recommendations may be presented to the user, for example, at a repair decision point 490. The user can review the repair recommendations and either, for example, repair the vehicle on-site as recommended, or detour it, or the user may override the recommendations by not agreeing with them.

[0048] The adjusted defect list 480 may be provided as a prioritized list of defects and defect locations. The prioritization may include a recommended order of repair for defects that require repair. The prioritization may also include one or more defects that do not require repair. For defects for which repair is recommended, the recommendation may also include a product recommendation 496 for the repair process.

[0049] Figure 5 is a block diagram of a networked vehicle repair system. In one embodiment, the defect detection and ranking system 500 is accessible on the network 500. As used herein, the term network 500 includes wired networks, wireless networks, or cloud-based networks.

[0050] The defect detection system 510 inspects the vehicle using a visual system 511. The visual system 511 includes one or more light sources for illuminating the vehicle and one or more image capture systems, such as cameras or video cameras. Defects are identified based on the captured images.

[0051] For each defect, the type of defect is identified using the defect type classifier 515. The defect type classifier 515 identifies each defect based on the image of the defect captured by the vision system 511. For example, the defect type classifier 516 can detect surface discontinuities corresponding to scratches. Dents or excess paint can be detected by shadows that should not be present on the surface. For each detected defect, the defect severity classifier 514 assigns a severity level. For example, deep scratches are more severe than shallow scratches. Long scratches are more severe than short scratches.

[0052] The coordinate system identifier 512 identifies the vehicle's coordinate system. The defect location identifier 516 can use the vehicle's coordinate system to identify the defect center and defect area for at least some defects. For example, in the case of a fragment trapped during the painting process, the defect center and the total area that needs to be sanded during repair can be identified. In the case of a scratch, the defect center, as well as its length and direction, can be identified.

[0053] The defect prioritization system 520 retrieves identified defects from the defect detection system 510 using the defect list searcher 522. The identified defects can be provided along with their location, type, and severity. The defect prioritization system also retrieves vehicle specifications from the vehicle specification database 540 using the vehicle specification searcher 521.

[0054] The vehicle specification database 540 stores information about the vehicle in the manufacturing assembly process. The vehicle specification includes a three-dimensional map of the vehicle's surface, which can be matched to a defect coordinate system. A vehicle currently being evaluated for repair determination may have a model 541, one or more colors 542, and one or more options 543 (e.g., for automobiles, two-door or four-door, spoiler, door trim, hood trim, etc.). The vehicle may also have a destination 544, which may be an intended market or, more specifically, a distributor. In addition, each vehicle may have a paint type 546 and a paint application process 545. For example, most vehicles have several layers of applied paint, as well as several layers of clear coat. One or more primer layers may also be applied. Each layer has an associated thickness and curing method. Curing may include air-drying the vehicle or applying heat or another curing process to ensure that the layers are completely dry between coats.

[0055] The vehicle specification searcher 521 retrieves information, including a surface map of the vehicle, from the vehicle specification database 540. The different specifications retrieved allow the defect prioritization system 520 to rank the retrieved defects differently. For example, scratches are much more visible to customers on a black vehicle than on a lighter color. Excess paint is not visible on silver or other metallic colors.

[0056] The defect prioritization searcher 523 retrieves prioritization information from the defect prioritization database 530. The retrieved priorities may also be based on vehicle specification information retrieved by the vehicle specification searcher 521. Different vehicle models have different model priorities 531. For example, trucks have different priorities than sedans, etc., regarding what defects must be repaired. Vehicles with different color schemes also have different color priorities 532. Different destination markets may also have different market priorities 534. Different classes of vehicles also have different vehicle class priorities 535. For example, yachts have different defect priorities and criteria than low-end motorboats.

[0057] In one embodiment, the data in the defect prioritization database 530 can be stored as a lookup table 536. However, other storage formats 537 are also conceivable.

[0058] The repair option searcher 526 retrieves information from the repair database 550. For example, it can retrieve current information about the repair system 560 for an identified defect. The repair system 560 includes telemetric measurements 562 associated with the defect, including the location of the defect, and telemetric measurements of the movement of the repair robot after an attempt at robotic repair has been made. The repair system 560 also includes images 564 associated with the repair, including images before the repair and images after the repair taken during the verification of the repair. Other information 568 can also be retrieved. The repair system 560 can also retrieve information about previous defect repairs that may be relevant to the current repair. For example, telemetric measurements 562 for a previously repaired scratch on the hood may be referenced by the defect prioritization generator 528 when determining the repair plan and estimated repair time. The repair system 560 for a given defect may also include products 566 used in the repair. For example, the abrasive grit, the type and amount of abrasive, the amount of water or other fluid supplied, or other product parameters may be stored as products 566 associated with the defect repair.

[0059] The repair option searcher 526 can also retrieve other information from the repair database 550, such as available sanding materials 552 and sanding robot parameters 554. For example, different sanding grits may be available, and / or different fluids to support sanding may be available. In addition, different sanding materials 556 may be available along with sanding robot parameters 558.

[0060] The robot status searcher 527 communicates with the repair system 570 to determine the status of the repair robot. For example, if additional sanding material or abrasive material is needed, or if there are reported errors, the robot status searcher 527 can retrieve such information and present it to the user interface 480.

[0061] The defect prioritization generator 528 automatically generates a prioritized defect list using the specifications retrieved by the vehicle specification searcher 521, the defect prioritization information retrieved by the defect prioritization searcher 523, and the defect list searcher 522. For example, the defect prioritization generator 528 can select defects in the high-priority area as defects that must be repaired, and can then order those defects in a repair plan based on information retrieved from the repair option searcher 526 and the robot status searcher 527. For example, using known telemetry information 562 from previous repairs, the defect prioritization generator can optimize the robot's movement path to address the highest-priority defects in the shortest time.

[0062] The networked repair system 500 may include a user interface 580. The user interface 580 can enable operators of the repair section of the assembly line to interact with the defect prioritization system 520. In particular, the user interface 580 can present a prioritized defect list 582 and repair recommendations 584. The repair recommendations 584 may include a repair plan that includes the order of defects to be repaired and / or recommended products, as well as telemetry, product quantity to be supplied, dwell time, and other parameters for each defect. Repair prioritization can also be determined by the availability of repair cycles available on the assembly line versus stop stations versus offline repair stations. If a large number of non-critical defects are identified, but there is a high demand for vehicle production during its production run, defect repair may be given a lower priority. Alternatively, if stop stations and offline repair stations are unavailable or full, defect repair may also be given a lower priority.

[0063] The user interface 580 can also enable the operator to interact with the repair system 560, for example, to view defect images 564, product usage during repair 566, or other information 568 related to the repair. In addition, the user interface 580 may allow the operator to provide feedback on the prioritized defect list 582 and the repair recommendations 584. For example, the operator can change the priority of defects on the prioritized defect list 582, or enter a different repair plan 584 for a given defect.

[0064] Figure 6 is a block diagram of a method for prioritizing defects for repair. In one embodiment, method 600 is performed automatically by the processor of a repair prioritization system.

[0065] In block 610, a prioritization specification is retrieved. For example, the prioritization specification includes information about the vehicle 612 being evaluated. For example, the vehicle's make, model, color, detail options, etc., may be retrieved. A prioritization specification 614 is also retrieved, which includes vehicle grade priorities, destination priorities, and other relevant priorities for a given vehicle. A restoration specification 616 is also retrieved, which includes information about previous restorations of a similar vehicle. In addition, previous user feedback may also be retrieved.

[0066] In block 620, a list of detected defects is received. Defects are detected on the vehicle using a vision system, which often includes one or more light sources and one or more cameras that capture images of the vehicle. The vision system also includes a processor that detects defects based on the captured images of the vehicle. The list of defects is often provided as a set of coordinates associated with each defect. Each defect may also include a defect type and / or severity ranking.

[0067] Block 630 generates a list of prioritized defects. The list of prioritized defects is generated automatically by applying the extracted prioritized specifications to the list of extracted defects. For example, depending on the vehicle, different characteristics may result in different defects being given higher priority. Defect prioritization takes into account color 632, defect location 634, relative position of the defect to the vehicle surface 636, defect type 638, captured defect image 642, destination market 644, and / or vehicle grade 646. Prioritization may also include ordering defects in repair order. Prioritization may also include grouping defects into groups such as “repair” or “not repair” depending on the extracted prioritization specifications.

[0068] The generated prioritized defect list is provided in block 640. The prioritized defect list can be provided to an assembly line operator for review, for example, on a user interface. Providing the prioritized defect list may also include providing a repair robot with instructions to repair at least some of the defects on the prioritized defect list.

[0069] In block 650, feedback is received. In one embodiment, feedback is received from the operator via the user interface based on a prioritized list of defects. However, in some embodiments, feedback is provided only after the repair is complete.

[0070] The advantages of using the systems and methods described herein include improved efficiency in defect repair. Human-based repair plans often involve biases, such as limiting the necessary human movement around the vehicle and ensuring that defects are not overlooked. For this reason, humans often repair defects on the vehicle in a regular manner, from top to bottom and front to rear. By using robots to repair defects and having an automated system for prioritizing defects for repair, the bias of human repair technicians is eliminated without overlooking defects that need to be repaired. Defects are also objectively considered and ranked by measurable criteria, thereby eliminating the subjectivity of human repair technicians. In addition, the time required to repair defects is reduced. Blocks 610-640 of Method 600 can be achieved in less than about 5 minutes in some embodiments. In some embodiments, blocks 610-640 can be achieved in less than about 1 minute. This results in a more objective repair plan based on customizable criteria, which is delivered faster than what can be done by human inspection.

[0071] Figure 7 shows a vehicle repair system. System 700 shows one exemplary system in which the systems and methods described herein can be used. However, the systems and methods described herein are not limited to the system shown in Figure 7.

[0072] The controller 710 provides commands to the robot motion controller 730, which in turn moves the robot 720. The movement of the robot 720 allows the polishing tool 724 to come into contact with the surface of the vehicle. The polishing tool 724 may include a sanding or polishing tool. The polishing tool 724 may also include a fluid dispenser that can supply water, abrasive, or other fluids. A force compliance mechanism 722 can adjust the amount of force applied to the surface by the polishing tool 724. Different amounts of force over different dwell times remove different amounts of paint from the surface of the vehicle. Figure 7 shows only one robot 720 for simplification, but in other embodiments, there may be more robots or additional polishing tools 724.

[0073] In some embodiments, a verification robot 740 is also present in the system 700. However, although the verification robot 740 is shown as a separate system that receives control signals from the controller 710, it is also understood that in some embodiments, the verification system may be included as part of the repair robot 720.

[0074] In one embodiment, the controller 710 generates a repair plan for the repair robot 720. The repair plan includes an order of defects to be repaired based on a prioritized list of defects to be repaired. In one embodiment, the controller 710 can also generate a list of prioritized defects based on information received regarding the vehicle. Information about the vehicle and information about the vehicle's repair priorities are provided by the system 750. The system 750 may also include a user interface so that the operator can review the list of prioritized defects and / or the repair plan and provide feedback to improve future repair prioritization.

[0075] Figure 8 shows an exemplary user interface for a vehicle repair system. Figure 8 shows a computer 800 with a user interface display screen 802. Screen 802 may be a touchscreen or a pen-enabled interface that receives input from a pen or stylus. The computer 800 may also use a virtual keyboard on the screen. Naturally, the computer 800 may be attached to a keyboard or other user input device via a suitable mounting mechanism, such as a wireless link or a USB port. The computer 800 may also, exemplary, receive voice input. Although the computer 800 is shown as a tablet, in other embodiments the computer 800 may also be a desktop computer or a laptop computer.

[0076] On screen 802, in one embodiment, there is a preliminary list 804 of coordinates. These coordinates may be those output from a defect detection system before prioritization. In a conventional repair system, these coordinates would serve as a reference for human repair technicians to repair the vehicle.

[0077] In one embodiment, there is also a prioritized list of defects 806, which includes the location, type, and severity of each defect. In some embodiments, each defect in the list 806 may be selectable so that the operator can see which priorities contributed to the ranking. Each defect in the list 806 may also, in one embodiment, be configured as a form of user feedback.

[0078] Information about the vehicle being evaluated can also be presented in information box 810. For example, a silver Honda CR-V for a dealer in Tokyo, Japan, is presented.

[0079] As shown in block 820, repair recommendations can also be provided. For example, based on the detected defects, the mentioned vehicle can be repaired on-site. In some embodiments, the repair recommendations 820 may be selectable so that the operator can examine the repair instructions sent to the repair robot, including telemetry and the polishing products to be used.

[0080] Figure 9 is a block diagram of a defect detection and ranking system architecture. The remote server architecture 900 shows one embodiment of the implementation of the defect detection and ranking system 910. As an example, the remote server architecture 900 can provide computing, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system delivering the service. In various embodiments, the remote server can deliver services over a wide area network such as the Internet using an appropriate protocol. For example, the remote server can deliver applications over a wide area network and access them via a web browser or any other computing component. The software or components shown or described in Figures 1 to 8, and the corresponding data, can be stored on a remotely located server. Computing resources within the remote server environment can be integrated at a remote data center location, or the computing resources can be distributed. The remote server infrastructure can deliver services over a shared data center, even if it appears to the user as a single access point. Thus, the components and functions described herein can be provided from a remotely located remote server using the remote server architecture. Alternatively, the components and functions described herein may be provided by a conventional server installed directly on a client device or by other means.

[0081] In the embodiment shown in Figure 9, some items are the same as those shown in the previous figure. Specifically, Figure 9 shows that the defect detection and ranking system can be located at the remote server location 902. Thus, the computing device 920 accesses these systems via the remote server location 902. The operator 950 can similarly access the user interface 922 using the computing device 920.

[0082] Figure 9 also illustrates another embodiment of the remote server architecture. Figure 9 shows that some elements of the system described herein are located at the remote server location 902, but others are not. For example, the storage devices 930, 940, or 960, or the repair system 970, are located at a location separate from location 902 and can be accessed via the remote server at location 902. Regardless of the location of the storage devices 930, 940, or 960, or the repair system 970, they may be accessed directly by the computing device 720, via a network (either a wide area network or a local area network), hosted and provided as a service at a remote site by a service, or accessed by a connected service located at a remote location. Furthermore, data can be stored at substantially any location and accessed intermittently by interested parties, or transferred to interested parties. For example, physical carriers can be used instead of, or in addition to, electromagnetic carriers.

[0083] Furthermore, it should be noted that the elements or parts thereof shown in Figures 1 to 5 can be placed on a wide variety of different devices. Some of these devices include servers, desktop computers, laptop computers, tablet computers, or other mobile devices such as palmtop computers, mobile phones, smartphones, multimedia players, and personal digital assistants.

[0084] Figures 10 and 11 show examples of mobile devices that can be used in the embodiments shown in the previous figures.

[0085] Figure 10 is a simplified block diagram of one exemplary embodiment of a handheld or mobile computing device that can be used as a user or client handheld device 16 (for example, as the computing device 920 in Figure 9) to deploy the system (or a part thereof). For example, the mobile device can be deployed within the operator compartment of the computing device 920 for use in generating, processing, or displaying data. Figure 11 is another embodiment of the handheld or mobile device.

[0086] Figure 10 provides a general block diagram of the components of client device 1016, which can perform and interact with some of the components shown in Figures 1 to 5, or perform some and interact with some of them. Device 1016 is provided with a communication link 1013 that enables a handheld device to communicate with other computing devices and, in some embodiments, provides a channel for automatically receiving information, such as by scanning. Examples of communication links 1013 include enabling communication via one or more communication protocols, such as a wireless service used to provide cellular access to a network, and a protocol that provides a local wireless connection to a network.

[0087] In other embodiments, the application may be received on a removable Secure Digital (SD) card connected to interface 1015. Interface 1015 and communication link 1013 communicate with processor 1017 (which may also be an embodiment of a processor) along bus 1019, which is also connected to memory 1021 and input / output (I / O) components 1023, as well as clock 1025 and position system 1027.

[0088] In one embodiment, the I / O component 1023 is provided to facilitate input and output operations, and the device 1016 may include input components such as buttons, touch sensors, optical sensors, microphones, touchscreens, proximity sensors, accelerometers, and orientation sensors, and output components such as display devices, speakers, and / or printer ports. Other I / O components 1023 can be used in a similar manner.

[0089] Clock 1025 includes, exemplarily, a real-time clock component that outputs the time and date. Clock 1025 can also provide timing functions for processor 1017.

[0090] Exemplary, the position system 1027 includes a component that outputs the current geographical location of device 1016. This could be, for example, a global positioning system (GPS) receiver, a LORAN system, an autonomous navigation system, a cellular triangulation system, or other positioning system. The position system 1027 may also include, for example, mapping software or navigation software that generates a desired map, navigation route, and other geographical functions.

[0091] Memory 1021 stores the operating system 1029, network settings 1031, applications 1033, application configuration settings 1035, data store 1037, communication drivers 1039, and communication configuration settings 1041. Memory 1021 may include all types of tangible volatile and nonvolatile computer-readable memory devices. Memory 1021 may also include computer storage media (described below). Memory 1021 stores computer-readable instructions that, when executed by processor 1017, cause the processor to perform steps or functions that the computer will perform according to the instructions. Processor 1017 can be activated by other components, and their functions can also be facilitated.

[0092] Figure 11 shows that the device may be a smartphone 1071. The smartphone 1071 has a touch-sensitive display 1073 that displays icons or tiles or other user input mechanisms 1075. Mechanisms 1075 can be used by the user to run applications, make phone calls, perform data transfer operations, etc. Generally, the smartphone 1071 is built on a mobile operating system and provides more advanced computing power and connectivity than a feature phone.

[0093] Please note that other forms of device 1016 are possible.

[0094] Figure 12 is a block diagram of a computing environment that can be used in the embodiment shown in the previous figure.

[0095] Figure 12 is an example of a computing environment in which elements of Figures 1 to 5, or parts thereof (for example), can be deployed. Referring to Figure 12, an exemplary system for implementing several embodiments includes a general-purpose computing device in the form of a computer 1110. The components of the computer 1110 may include, but are not limited to, a processing unit 1120 (which may include a processor), system memory 1130, and a system bus 1121 that connects various system components, including the system memory, to the processing unit 1120. The system bus 1121 may be any of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, and a local bus using any of various bus architectures. The memory and programs described with respect to Figures 1 to 5 can be deployed in the corresponding parts of Figure 12.

[0096] Computer 1110 typically includes various computer-readable media. Computer-readable media may be any available media that can be accessed by computer 1110 and include both volatile / non-volatile media and removable / non-removable media. Computer-readable media may include computer storage media and communication media, not as an example but as an exception. Computer storage media are different from and do not include modulated data signals or carrier waves. Computer storage media include hardware storage media that include both volatile / non-volatile media and removable / non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computer 1110. Communication media include any information delivery medium that can embody computer-readable instructions, data structures, program modules, or other data within a transport mechanism. The term “modulated data signal” means a signal having one or more of its properties that are set or modified to encode information within the signal.

[0097] The system memory 1130 includes computer storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 1131 and random access memory (RAM) 1132. A basic input / output system (BIOS) 1133, which contains basic routines that help transfer information between elements within the computer 1110 during startup, is typically stored in the ROM 1131. The RAM 1132 typically contains data and / or program modules that are immediately accessible and / or currently running by the processing unit 1120. As an example, but not an limitation, Figure 12 shows an operating system 1134, an application program 1135, other program modules 1136, and program data 1137.

[0098] Computer 1110 may also include other removable / non-removable and volatile / non-volatile computer storage media. As just one example, Figure 12 shows a hard disk drive 1141 that reads from or writes to a non-removable, non-volatile magnetic medium, a non-volatile magnetic disk 1152, an optical disk drive 1155, and a non-volatile optical disk 1156. The hard disk drive 1141 is typically connected to the system bus 1121 via a non-removable memory interface such as interface 1140, and the optical disk drive 1155 is typically connected to the system bus 1121 via a removable memory interface such as interface 1150.

[0099] Alternatively or additionally, the functions described herein can be performed, at least in part, by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (e.g., ASICs), application-specific standard products (e.g., ASSPs), system-on-a-chip systems (SOCs), and complex programmable logic devices (CPLDs).

[0100] The drives described above and shown in Figure 12, and their associated computer storage media, provide storage for computer-readable instructions, data structures, program modules, and other data of the computer 1110. In Figure 12, for example, the hard disk drive 1141 is shown as storing the operating system 1144, application programs 1145, other program modules 1146, and program data 1147. Note that these components may be the same as or different from the operating system 1134, application programs 1135, other program modules 1136, and program data 1137.

[0101] The user can input commands and information to the computer 1110 via input devices such as a keyboard 1162, a microphone 1163, and a pointing device 1161 such as a mouse, trackball, or touchpad. Other input devices (not shown) include joysticks, gamepads, satellite receivers, and scanners. These and other input devices are often connected to the processing unit 1120 via a user input interface 1160 coupled to the system bus, but may be connected by other interfaces and bus structures. A visual display 1191 or other types of display devices are also connected to the system bus 1121 via interfaces such as a video interface 1190. In addition to the monitor, the computer may also include other peripheral output devices such as a speaker 1197 and a printer 1196, which can be connected via an output peripheral interface 1195.

[0102] Computer 1110 operates in a networked environment using logical connections such as a Local Area Network (LAN) or Wide Area Network (WAN) to one or more remote computers, such as remote computer 1180.

[0103] When used in a LAN network environment, computer 1110 is connected to LAN 1171 via a network interface or adapter 1170. When used in a WAN network environment, computer 1110 typically includes a modem 1172 or other means for establishing communication over WAN 1173, such as the Internet. In a networked environment, program modules can be stored in remote memory storage devices. Figure 12 shows, for example, that a remote application program 1185 may reside on a remote computer 1180.

[0104] It should also be noted that the different embodiments described herein can be combined in different ways. That is, parts of one or more embodiments can be combined with parts of one or more other embodiments. All of this is intended herein.

[0105] Embodiment Embodiment 1 is a defect detection and ranking system for a vehicle assembly line. The system includes an image capture device that captures multiple images of vehicles on the vehicle assembly line. The system also includes a data store containing vehicle specifications and defect priorities based on the vehicle specifications for the vehicles on the vehicle assembly line. The system also includes a defect detector that analyzes the multiple captured images and, based on the analysis, detects multiple defects on the surface of the vehicle. Each detection detector associates the xyz coordinate position, defect type, and defect severity for each of the multiple defects. The system also includes a defect prioritization generator configured to receive the multiple defects from the defect detector, retrieve the vehicle specifications and defect priorities, apply the defect priorities to the multiple defects, and generate a list of prioritized defects. The defect prioritization generator outputs the list of prioritized defects to an output device associated with the vehicle assembly line.

[0106] Embodiment 2 includes the features described in Embodiment 1, except that the type of defect is one of scratches, dents, or excess paint.

[0107] Embodiment 3 includes the features described in Embodiment 1 or 2, but the plurality of defects include a first defect having a first defect location and a second defect having a second defect location. The defect priority is the defect location. The first defect is given higher priority than the second defect.

[0108] Embodiment 4 includes the features described in Embodiment 3, but the vehicle is a passenger car, the first defect is located on the driver's side door of the passenger car, and the second defect is located on the roof of the passenger car.

[0109] Embodiment 5 includes the features described in any one of Embodiments 1 to 4, except that the vehicle specification is the first vehicle color. One of the multiple defect types is a scratch. The defect priority of a scratch is higher for the first vehicle color than for the second vehicle color.

[0110] Embodiment 6 includes the features described in Embodiment 5, but the first color is black and the second color is silver.

[0111] Embodiment 7 includes the features described in Embodiments 1 to 6, but also includes a sensor configured to identify a vehicle. The defect prioritization generator automatically retrieves the vehicle specifications and defect priorities based on the identified vehicle.

[0112] Embodiment 8 includes the features described in Embodiment 7, but the defect prioritization generator automatically applies vehicle specifications and defect priorities when multiple defects are received.

[0113] Embodiment 9 includes the features described in any one of Embodiments 1 to 8, but the defect prioritization generator also provides a list of prioritized defects along with a recommendation for repair. The recommendation for repair is either to repair the vehicle on-site or to divert the vehicle for repair.

[0114] Embodiment 10 includes the features described in Embodiment 9, but the repair recommendation also includes a directive not to repair a subset of multiple defects.

[0115] Embodiment 11 includes the features described in Embodiment 10, but the repair recommendation also includes an instruction to the repair robot to repair at least one defect. The instruction includes a remote measurement to the repair robot.

[0116] Embodiment 12 includes the features described in Embodiment 11, but the instructions also include the polishing product for at least one defect, the force to be applied, the trajectory, and the dwell time.

[0117] Embodiment 13 includes the features described in any one of Embodiments 1 to 12, wherein the defect priority is selected from model priority, color priority, defect location priority, destination priority, vehicle grade priority, vehicle paint priority, and paint application priority.

[0118] Embodiment 14 includes the features described in any one of Embodiments 1 to 13, wherein the captured images are stored in a repair system database. The defect prioritization generator retrieves images of defects from the repair system database. The defect prioritization generator prioritizes the defects based at least partially on the retrieved images.

[0119] Embodiment 15 includes the features described in any one of Embodiments 1 to 14, but the output device includes an input device configured to receive user feedback metrics. The user feedback metrics change the ranking of defects in a prioritized list of defects.

[0120] Embodiment 16 includes the features described in Embodiment 15, but the user feedback metrics are stored in a data store.

[0121] Embodiment 17 includes the features described in any one of Embodiments 1 to 16, but the vehicle specification includes a three-dimensional map of the vehicle's surface.

[0122] Embodiment 18 is a computer-based method for ranking detected surface defects on a vehicle on an assembly line. The method includes the step of receiving vehicle identification. The method also includes the step of using a defect list lookup to access a defect database stored in defect memory and receiving a plurality of detected surface defects on the vehicle. Each of the plurality of detected surface defects has a defect location, a defect type, and a defect severity. The method also includes the step of using a specification lookup to access a specification database stored in specification memory and automatically retrieving a first vehicle specification and a second vehicle specification based on the vehicle identification using a vehicle specification lookup. The method also includes the step of using a defect priority lookup to access a defect priority database stored in priority memory and automatically retrieving a first defect priority based on the first vehicle specification and a second defect priority based on the second vehicle specification. The method also includes the step of using a defect priority generator to generate a defect priority list by applying the first and second defect priorities to a plurality of defects. The defect priority list includes ranking multiple detected surfaces such that a first defect is ranked higher than a second defect among multiple defects, based on applying a first and second defect priority to the defect location, defect type, and defect severity. The method also includes the step of providing the defect priority list as output to an output device associated with the assembly line. The steps of receiving identification, receiving multiple detected surface defects, retrieving vehicle specifications, generating a defect priority list, and providing the defect priority list are performed by a non-temporary computer-readable storage medium containing instructions, which, when executed by at least one processor of a computing device, cause at least one processor to perform the steps of defect list lookup, vehicle specification lookup, defect priority list, and providing the defect priority list.

[0123] Embodiment 19 includes the features described in Embodiment 18, wherein the first and second vehicle specifications are selected from the group consisting of vehicle model, vehicle color, vehicle options, vehicle destination, vehicle paint type, and vehicle paint application.

[0124] Embodiment 20 includes the features described in Embodiment 18 or 19, wherein the first and second defect priorities are selected from the group consisting of defect location priority, defect classification priority, vehicle color priority, vehicle grade priority, and relative defect location priority.

[0125] Embodiment 21 includes the features described in any one of Embodiments 18 to 20, but also includes the step of imaging a vehicle. The imaging step includes capturing multiple images using a camera and stitching the multiple images together using a processor. The method also includes the step of detecting multiple surface defects on the vehicle. The detection step includes the step of the processor analyzing the stitched images for potential defects. The defect detection processor identifies defect features. Defect features include the defect location and defect type of each detected surface defect.

[0126] Embodiment 22 includes the features described in any one of Embodiments 18 to 21, but the output device has a screen, and the defect priority list is provided on the screen.

[0127] Embodiment 23 includes the features described in any one of Embodiments 81 to 22, but the output device also has an input device configured to receive user feedback regarding a defect priority list. The user feedback is automatically associated with a first vehicle specification and a second vehicle specification.

[0128] Embodiment 24 includes the features described in Embodiment 23, except that the vehicle is the first vehicle. Upon receiving identification of a second vehicle having the first vehicle specifications and the second vehicle specifications, user feedback is extracted and applied to a second set of defects associated with the second vehicle.

[0129] Embodiment 25 includes the features described in any one of Embodiments 18 to 24, but the output device also has an input device configured to receive user feedback regarding a defect priority list. The user feedback is automatically associated with a first defect priority and a second defect priority.

[0130] Embodiment 26 includes the features described in Embodiment 25, except that the vehicle is the first vehicle. Upon receiving the identification of a second vehicle having a first defect priority and a second defect priority, user feedback is retrieved and applied to the second set of defects associated with the second vehicle.

[0131] Embodiment 27 includes the features described in any one of Embodiments 18 to 26, wherein the first vehicle specification is the vehicle model. The first defect priority is the defect location. The first detected defect location and vehicle model are prioritized higher than a second detected defect having a second location.

[0132] Embodiment 28 includes the features described in any one of Embodiments 18 to 27, wherein the first vehicle specification is the first vehicle color. The defect priority is the defect type. A first detected defect having the first defect type is given a higher priority than a second detected defect having the second defect type.

[0133] Embodiment 29 includes the features described in Embodiment 28, but for a second vehicle having a second vehicle color different from the first vehicle color, the priority of the second detected defect is higher than that of the first detected defect.

[0134] Embodiment 30 is a vehicle repair prioritization system for an assembly line. The system includes a vehicle specification database containing the specifications of vehicles on the assembly line, the database including a first specification and a second specification. The first specification is the vehicle color index of the vehicle. The system also includes a defect database containing a list of defects detected on the vehicle. Each detected defect has an associated defect location and defect type. The system also includes a non-temporary computer-readable medium on which instructions are stored, the instructions being configured to retrieve the first specification and the second specification, and a first priority based on the first specification and a second priority based on the second specification when executed by the processor. The first priority is the color priority. The computer-readable medium is also configured to generate a list of prioritized defects by retrieving the list of detected defects, applying the first priority and the second priority to each defect on the list of defects to determine the priority of the defects to be repaired, and to output the list of prioritized defects to an output device associated with the assembly line.

[0135] Embodiment 31 includes the features described in Embodiment 30, but the system is a networked system.

[0136] Embodiment 32 includes the features described in Embodiment 31, but the system also includes a repair system that communicates with a vehicle repair prioritization system.

[0137] Embodiment 33 includes the features described in Embodiment 32, but the repair system includes a repair system database that stores marks of previously repaired defects. The marks include at least one of pre-repair images, post-repair images, remote measurements of robotic repairs, and the type and amount of polishing product applied.

[0138] Embodiment 34 includes the features described in any one of Embodiments 30 to 33, but the processor automatically retrieves the specifications, retrieves the detected defects, and generates a prioritized list of defects based on the received indicators that the vehicle is ready for repair.

[0139] Embodiment 35 includes the features described in Embodiment 34, but the indicator is received from a sensor that detects a vehicle on the assembly line.

[0140] Embodiment 36 includes the features described in any one of Embodiments 30 to 35, but the list of detected defects is provided by a defect detection system including a camera and a processor configured to analyze images captured by the camera. The processor is configured to detect defects on the vehicle based on the analysis and to assign a defect location and a defect type to the defects.

[0141] Embodiment 37 includes the features described in any one of Embodiments 30 to 36, wherein the second vehicle specification is selected from the group consisting of vehicle model, vehicle destination, vehicle paint type, vehicle paint application parameters, or vehicle options.

[0142] Embodiment 38 includes the features described in any one of Embodiments 30 to 37, wherein the prioritized list is an ordered list of defects to be repaired.

[0143] Embodiment 39 includes the features described in any one of Embodiments 30 to 38, but the prioritized list includes a first subset of detected defects designated to be repaired and a second subset of defects designated not to be repaired.

[0144] Embodiment 40 includes the features described in any one of Embodiments 30 to 39, but the processor is also configured to output a recommendation for vehicle repair. The repair recommendation is either to repair the vehicle on the assembly line or to divert the vehicle for repair.

[0145] Embodiment 41 includes the features described in Embodiment 40, but the repair recommendation also includes remote measurements to a repair robot for repairing multiple defects on a prioritized list of defects.

[0146] Embodiment 42 includes the features described in Embodiment 41, but the remediation recommendation also includes product recommendations for defects on a prioritized list of defects.

[0147] Embodiment 43 includes the features described in Embodiment 42, but the product recommendation is based in part on the previous repair history of the vehicle having the first and second specifications.

[0148] Embodiment 44 includes the features described in Embodiment 42, but the product recommendation is based in part on the previous repair history of the vehicle having a defect similar to that defect.

[0149] Embodiment 45 includes the features described in Embodiment 42, but the repair recommendation is communicated to the repair robot.

[0150] Embodiment 46 includes the features described in any one of Embodiments 30 to 45, but the prioritized list of defects is output to a repair robot assigned to repair the vehicle.

[0151] Embodiment 47 includes the features described in any one of Embodiments 30 to 46, except that the output device is a robotic repair unit. The robotic repair unit is configured to perform repairs based on a prioritized list of defects. [Examples]

[0152] Example 1: Defect detection Typical sources of defects include particles trapped within the applied paint, aggregation of paint components, scratches, dents, or excess paint. Figure 11A shows a defect caused by particles of inclusion in a highly reflective paint, with a size of less than one-tenth of a millimeter. Figure 11A shows a magnified, microscopic view of the defect on a black painted surface with a clear coat.

[0153] Figure 11B shows a similar defect within the reflection on the upper surface. The defect is detectable and appears as a shadow in the image.

[0154] For ease of understanding, the following examples relate to a small set of defects on a vehicle. However, it should be understood that at many assembly sites, a given vehicle may have numerous surface defects that need to be evaluated and then prioritized for repair.

[0155] Example 2: Prioritizing defects Table 1 below shows an exemplary list of defects output from the defect detection system. Defects are output through a coordinate system along with their volume and type.

[0156] [Table 1]

[0157] In Table 1, x, y, and z provide the location in three-dimensional space. Volume represents the size of the defect, and the center of the defect in space is related to the generalized volume. In this case, the types used are d-drop, s-scratch, and c-dent.

[0158] In Table 1, defects are initially prioritized based on their location. However, users can provide feedback that larger defects should be prioritized above their location on the vehicle. This allows the prioritization to be changed, and the set of defects is re-prioritized based on size, as shown in Table 2.

[0159] [Table 2]

[0160] Example 3: Defect Prioritization Table Not all defects require repair. Part of the prioritization scheme includes steps to examine defects and remove those that do not require repair. Figure 12A shows a sample output of detected defects returned to the lookup table. The array, from left to right, includes the sample ID, x, y, and z coordinates, the defect size in millimeters, and the defect classification.

[0161] Defects with a value of 0.01 or greater are removed. In addition, surface boundary defects 205, 314, and 400 are high-priority defects. As shown in Figure 12B, the removed defects are not presented on the list of prioritized defects.

[0162] Example 4: Incorporating user feedback Figures 13A and 13B illustrate an example of how user feedback is incorporated into prioritizing defects for repair. Figure 13A shows the initial list of defects, and Figure 13B shows the list of prioritized defects. Defects classified as "d" were inputs of user feedback indicating that the paint and defect location on a given vehicle should be given a lower priority. Defects with ID numbers 4 and 6 were also indicated as needing to be prioritized based on direct human inspection of the defects.

[0163] Example 5: On-site repair or detour Determining early whether a vehicle requires more extensive repairs than can be performed on the assembly line can save time in the long run. Prioritizing defects before repair work begins and detecting defects that require extensive repairs on the vehicle allows the assembly line to continue running. Figures 14-16 show a portion of the set of defects for prioritization. The prioritization rule in Example 3 is applied to obtain the set in Figure 14, but additional prioritization also exists, which automatically classifies vehicles as needing to be diverted for repair if a defect of 0.5 mm or larger is detected. Defect ID number 16 has such a size, so the vehicle is diverted for repair.

Claims

[Claim 1] A defect detection and ranking system for a vehicle assembly line, An image capture device that captures multiple images of vehicles on the vehicle assembly line, A data store including the vehicle specifications of the vehicle on the vehicle assembly line and the defect priority based on the vehicle specifications, A defect detector that analyzes the plurality of captured images and, based on the analysis, detects a plurality of defects on the surface of the vehicle, wherein each of the plurality of defect detectors associates the x-y-z coordinate position, defect type, and defect severity with respect to each of the plurality of defects. A defect prioritization generator, The defect detector receives the plurality of defects, Extract the vehicle specifications and the defect priority, Applying the aforementioned defect priority order to the plurality of defects, Generate a list of prioritized defects. A defect prioritization generator configured as follows, Equipped with, The defect prioritization generator outputs the prioritized list of defects to an output device associated with the vehicle assembly line. The vehicle specification includes a three-dimensional map of the surface of the vehicle. Defect detection and ranking system.

Citation Information

Patent Citations

  • Surface defect inspection device

    JP1993322543A

  • Method and apparatus for inspecting defect on surface to be inspected

    JP1999118730A

  • System, method and program for inspecting and maintaining lease vehicle

    JP2007066116A

  • Display device, display method, program allowing computer to execute the display method, and recording medium recorded with the program

    JP2008052611A

  • Defect classification method and device thereof

    JP2009103508A