Automated vehicle repair system

Abstract

Providing a vehicle assembly line defect detection and ranking system SOLUTION: The system includes an image capture device that captures a plurality of images of a vehicle on a vehicle assembly line and a defect detector that analyzes the plurality of captured images to detect a plurality of defects on a surface of the vehicle. Each of the plurality of defects has an associated x-y-z coordinate location, a defect type, and a defect severity. The system also includes a data store including vehicle specifications for vehicles on the vehicle assembly line and defect priorities based on the vehicle specifications. The system also includes a defect prioritization generator configured to receive the plurality of defects from the defect detector, retrieve the vehicle specifications and the defect priorities, apply the defect priorities to the plurality of defects, and output a prioritized list of defects. The defect prioritization generator outputs the prioritized list of defects to an output device associated with the vehicle assembly line.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Many different types of vehicles are manufactured worldwide each year. Vehicles, such as cars, trucks, coaches, and airplanes, often have one or more coatings of paint and clear coat applied during the manufacturing process. The process of correcting defects that occur in the painting process is often a time-consuming manual task. Summary of the Invention

[0002] A defect detection and ranking system for a vehicle assembly line is provided. The system includes an image capture device that captures multiple images of a vehicle on the vehicle assembly line. The system also includes a defect detector that analyzes the multiple captured images and detects multiple defects on a surface of the vehicle based on the analysis. Each of the multiple defects has an associated x-y-z coordinate location, a defect type, and a defect severity. The system also includes a data store that includes vehicle specifications for the vehicles on the vehicle assembly line and a defect priority based on the vehicle specification. The system also includes a defect prioritization generator configured to receive the multiple defects from the defect detector, retrieve the vehicle specification and the defect priority, apply the defect priority to the multiple defects, and output a list of prioritized defects. The defect prioritization generator outputs the list of prioritized defects to an output device associated with the vehicle assembly line. [Brief explanation of the drawings]

[0003] [Figure 1] 1 is a conceptual diagram of a manufacturing environment for detecting and correcting defects on a vehicle surface. [Figure 2] FIG. 1 is a block diagram of a method for defect detection and repair in a manufacturing environment. [Figure 3] FIG. 1 is a block diagram illustrating a vehicle repair system. [Figure 4] FIG. 1 is a block diagram of a defect detection and ranking system. [Figure 5] FIG. 1 is a block diagram of a networked vehicle repair system. [Figure 6] FIG. 1 is a block diagram of a method for prioritizing defects for repair. [Figure 7] 1 shows a vehicle repair system. [Figure 8] 1 is an exemplary user interface of a vehicle repair system. [Figure 9] FIG. 1 is a block diagram of a defect detection and ranking system architecture. [Figure 10] 1 illustrates an example of a mobile device that can be used in the embodiment illustrated in the previous figure. [Figure 11] 1 illustrates an example of a mobile device that can be used in the embodiment illustrated in the previous figure. [Figure 12] FIG. 1 is a block diagram of a computing environment that can be used in the embodiments shown in the previous figures. [Figure 13A] 1 shows defects and exemplary feedback that are described in more detail in the Examples. [Figure 13B] 1 shows defects and exemplary feedback that are described in more detail in the Examples. [Figure 14A] 1 shows defects and exemplary feedback that are described in more detail in the Examples. [Figure 14B] 1 shows defects and exemplary feedback that are described in more detail in the Examples. [Figure 15A] 1 shows defects and exemplary feedback that are described in more detail in the Examples. [Figure 15B] 1 shows defects and exemplary feedback that are described in more detail in the Examples. [Figure 16] 1 shows defects and exemplary feedback that are described in more detail in the Examples.

[0004] Structural changes to the embodiments described in this disclosure may be made without departing from the scope of the technology of the present disclosure. The figures are not necessarily drawn to scale. Like numbers used in the figures refer to like components. However, the use of a number to refer to a component in a given figure is not intended to limit the component in another figure bearing the same number. DETAILED DESCRIPTION OF THE INVENTION

[0005] The automation of vehicle manufacturing processes has been underway for decades. The vehicle manufacturing industry has begun to introduce robots to replace manual tasks previously performed by human workers. Automation can increase the efficiency of the manufacturing process and reduce workplace accidents by using robots to accomplish difficult, dangerous, or time-consuming tasks. Robots can also improve precision and reduce waste.

[0006] However, one area that has proven difficult to automate is the repair of defects, particularly surface defects that may be introduced or developed during the painting process. As explained in more detail with respect to the figures, vehicles are often coated with several layers of paint, which may include several coats of primer, color, and clear coat. Defects on the vehicle's surface can be caused by contaminants, clumping, and flow issues in the painting process, as well as drying irregularities.

[0007] Vehicle buyers have aesthetic standards that prospective vehicles must meet. For example, someone buying a new car expects to be sure that the car is free of scratches, dents, or other defects. Significant time and effort is spent inspecting the car and repairing any defects that are detected before it leaves the manufacturing facility.

[0008] The examples described herein describe the defect detection and repair process as it applies to the manufacture of automobiles, such as cars and trucks, however, it is expressly contemplated that many of the embodiments described herein are equally applicable to other vehicles, such as boats and other watercraft, passenger cars and locomotives, airplanes and other flying vehicles, rockets, etc.

[0009] As used herein, the term "paint" is intended to include primers, pigment layers, clear coats, and any other coating that may be applied to a passenger vehicle.

[0010] As used herein, the term "defect" is intended to refer to any imperfection that may be present on a vehicle after the painting process. For example, defects may include, but are not limited to, scratches, dents, debris trapped within the paint coating, excess or unevenly applied paint, haze, inconsistent gloss, or other surface-based imperfections.

[0011] 1 is a conceptual diagram illustrating an exemplary manufacturing environment for detecting and correcting defects on a vehicle surface. In manufacturing environment 100, vehicle 10 is first inspected for defects at station 110, a repair plan is generated, and then the vehicle is sent for repair at station 150. In many manufacturing environments, stations 110 and 150 are sequential, such that defects are identified and then repaired. However, in some embodiments, these steps can occur at least partially simultaneously.

[0012] After the painting process, vehicle 10 has multiple defects 20, 12, 14, 16, 22, and 18. Depending on the manufacturing facility, there may be more or fewer defects, and the defects may be located in different locations and of different types and severity.

[0013] Each defect on the vehicle 10 has a defect type (e.g., scratch or dent) and a severity (e.g., major or minor). The defect also has a location on the vehicle 10. Depending on the type of vehicle 10, different defects may be more or less important depending on their type, severity, and location. For example, in a small passenger car 10, a defect 18 near the door handle is much more likely to be a problem to a potential customer than a defect 12 on the roof of the passenger car 10 or a defect 16 under the front bumper.

[0014] The time available during production to detect and repair defects is limited: if more defects are detected than can be repaired in a timely manner, the vehicle 10 may need to be diverted for repair instead of being repaired during assembly.

[0015] 1, the defect detection station 110 includes a vision system having one or more cameras 40 and one or more light sources 50a, 50b, etc. The vision system operates by detecting and subsequently mapping irregularities in the surface of the vehicle 10. In some embodiments, the defect detection station 110 is part of a moving assembly line, as indicated by the conveyor belt 30 and transport mechanism 32. However, in other embodiments, the defect detection station 110 includes the vehicle 10 in a stationary position.

[0016] Defect detection systems are known in the art. For example, U.S. Patent Application Publication Nos. 2019 / 0096057, filed May 9, 2018, and 2017 / 0277979, filed March 22, 2016, both owned by Innovision Software Solutions Inc., describe some exemplary defect detection systems.

[0017] Once defects are identified, they are repaired at a repair station 150. The repair station 150 may include a robot 160 that can move using joints 164. An abrasive material may be applied to the vehicle using an applicator 162.

[0018] However, while robot 160 is shown in FIG. 1 to accomplish the repair, in many cases the repair plan is developed by a human operator. In many cases, at least a portion of the repair is also performed by a human. However, the human operator may introduce bias into the repair process. For example, a human operator may prefer to start at the front of the vehicle and finish at the rear by, for example, repairing defect 16 and repairing defect 20 last. Similarly, a human operator may preferentially choose to repair defect 12, which is on the top of the vehicle, before defects 18 or 22.

[0019] Using a robotic repair system 160 can also provide advantages over a human repair technician. The robotic repair system 160 uses a coordinate-based positioning system provided by the defect detection station 110 to approach and repair defects. Humans rely on vision to identify the location of defects. However, when a polishing pad with abrasives or other abrasive materials is applied, the defects may no longer be visible to the human eye. The robotic repair system 160 does not need to aim at the defect during repair, but instead relies on coordinates to ensure the repair is made in the correct location. This results in more reliable repairs and often smaller repair areas because the robot 160 is more precise.

[0020] During the manufacture of a vehicle, time is of the essence and choosing which defects to repair and which to leave alone can make the difference between a vehicle being declared ready for sale or not.

[0021] Innovation is required to determine a repair plan 120 for repairing detected defects on the vehicle surface. The repair plan must take into account known information about the vehicle 10 (make, model, color, and paint application parameters), as well as information about the detected defects (number, location, severity) and the destination market (country, distributor, potential customer) to determine which defects to repair and in what order. The system can also make recommendations or provide instructions to the robotic repair system 160 regarding which abrasive products to use to repair the defects.

[0022] 2 is a block diagram of a defect detection and repair method in which embodiments of the present invention may be useful. The method 200 is applicable to a wide variety of vehicles, including cars, trucks, boats, planes, trains, helicopters, etc.

[0023] At block 210, a vehicle is manufactured. Manufacturing includes structural assembly 212 of the vehicle. Manufacturing also includes one or more applications of paint 214. After the paint is applied, the vehicle undergoes a baking process 216 to promote uniform paint drying. Travel between the painting and baking stations can introduce dust or other debris into the vehicle surface, which can result in defects. Additionally, the vehicle can be scratched or dented during assembly 212, painting 214, or baking 216. Paint can also be applied unevenly, resulting in excess paint buildup in some areas, an inconsistent surface, haze, or inconsistent gloss.

[0024] In block 220, defects are detected. A defect detection system often includes one or more light sources that illuminate the vehicle. A scanning system may also be included, as shown in block 222. The scanning system detects defects in the paint surface and determines the location of each defect, as shown in block 224. A type is identified for each defect, as shown in block 226. For example, a defect may be identified as, for example, a dent, a scratch, or excess paint on the surface. A defect may also be identified as a raised surface, for example, excess paint or debris trapped in the paint, or a concave surface, such as a dent or scratch. In some embodiments, the defect location includes the coordinate location of the identified defect on the vehicle as well as the affected area. For example, a small piece of debris may only affect one square centimeter of paint, while a larger piece may affect one square inch. The defect is also classified as having a severity 228.

[0025] In blocks 230 and 240, the identified defects are repaired. Traditionally, defects are repaired by a human repair technician. The identified defects are provided to the technician from the vision system, and the technician decides which defects to fix based on printouts and visual inspection. Often, based on the visual inspection, a decision is made as to whether the vehicle can be repaired during the assembly process or whether the vehicle needs to be diverted for a more intensive repair. Diversions result in longer assembly times and more man-hours for the vehicle, resulting in higher costs. The systems and methods described herein are designed to increase the efficiency of the repair process. Additionally, the systems and methods designed herein also facilitate the use of robotic repair systems, thereby increasing repair accuracy, reducing the amount of repair product required, and shortening overall repair time.

[0026] At block 230, the defect is sanded, often using an abrasive product in combination with water, an abrasive, or another fluid. The defective area can be cleaned before or after sanding. After the water or another fluid is applied, the defective area can be wiped dry.

[0027] At block 240, the sanded defect is polished. Polishing may include using an abrasive or non-abrasive product in combination with water, an abrasive, or another fluid. Before or after polishing, the defect area may be washed or wiped dry. After polishing, the repaired defect may be inspected, as shown in block 242. This may be done as part of a repair verification process.

[0028] At block 250, the vehicle is released from the assembly process for sale. This often involves shipping the vehicle across state or country lines to a dealer. When the vehicle arrives at the dealer, it is often inspected again, as shown in block 252. Because inspection is expected at the point of sale, the selection of which defects to repair during blocks 230 and 240 is important to ensure that the vehicle is not rejected by the dealer.

[0029] 3 is a block diagram illustrating a vehicle repair system 300. In some embodiments, the vehicle repair system 300 can be implemented as part of a vehicle assembly and manufacturing system.

[0030] Vehicle specifications 310, in one embodiment, may be stored in a data store. Vehicle specifications are available for the vehicle currently being evaluated by the defect detection and ranking system 350. Vehicle specifications include, for example, vehicle make 312, model 314, and color 316. For example, a black Honda® CR-V® may be on the assembly line. A vehicle may have certain specifications, such as the number of doors and their position 322 (such as a two-door or four-door model car or truck). Different automobiles may have different hood details 324 or trunk details 326. For example, for aesthetic and aerodynamic reasons, different makes and models of cars have different hood designs with different contours. However, while many examples described herein refer to cars and trucks, many embodiments are equally applicable to other vehicle types. For example, an airplane or helicopter may have specifications regarding nose design 328 and tail design 332. Other specifications 318 may also be available for a given vehicle.

[0031] Defect detection and ranking system 350 includes defect detection system 352 and defect prioritization system 352. System 350 receives vehicle specification 310, and based at least on some of the vehicle specification and defects detected by defect detection system 352, the defect prioritization system provides a prioritized set of defects for repair to defect repair system 360.

[0032] For at least some vehicles, market information 340 may also be important to the prioritization system 354. Market information may include final destination 342, vehicle grade 344, customer 346, or other information. For example, vehicles for the Japanese or European markets may need to have fewer outstanding defects to be considered acceptable by a dealer. Vehicles with a higher vehicle grade 344 may also have a higher standard for repaired defects, as well as fewer outstanding defects, if any. For example, a Lamborghini™ is considered a higher vehicle grade than a Ford™ Ranger™. Customer information may also be available and important for prioritizing defects. For example, customers in the Japanese market are generally shorter than customers in the U.S. market. Thus, vehicles for different markets may have different defect prioritization depending on the expected customer's visibility range.

[0033] Defect detection system 352 is configured to detect a set of defects associated with the vehicle. Defect detection system 352 is configured to detect the defects using a vision system. Defect detection system 352 outputs the set of defects to defect prioritization system 354. Each defect in the set of defects is associated with a defect type, a defect location, and a defect severity.

[0034] The defect prioritization system 354 receives the set of defects and orders the defects for repair based on at least some of the specifications 310, market information 340. The defect prioritization system 354 can also make decisions about which defects should be repaired and which defects should remain unrepaired. The defect prioritization system can also provide recommendations about whether a vehicle should be repaired as part of the assembly line process or whether it should be diverted for a separate repair process. Diversions may be recommended automatically based on defect thresholds, such as the length of a detected flaw or the size of trapped debris. Diversions may also be recommended based on the number of defects, the number of types of defects, or another threshold.

[0035] Depending on the vehicle specifications 310 and market information 340, the defect prioritization system 354 generates a defect repair list or defect repair order that ranks defects differently depending on the retrieved specifications. For example, for a taller passenger car, defects on the roof are a lower priority. For a silver passenger car, scratches are less noticeable and a lower priority than scratches on a red passenger car. On black surfaces, such as the hood or trunk of a black passenger car, all surface defects are more noticeable and a higher priority for repair. Defects on the driver's side door, especially around the handle, are a higher priority than defects elsewhere on the vehicle. Defects may be a lower priority if they are below the exterior trim line than if they are above the exterior trim line. A defect on the hood of a clear-coated jet-black painted sedan may be much more visible than the same defect on the hood of an identical passenger car that is white. A defect in the underside of a vehicle on a truck may be much more readily accepted than the same defect on a passenger car due to the truck's perceived purpose and durability, and rough expected lifespan.

[0036] The defect detection and ranking system 350 also communicates with a defect repair system 360 in some embodiments. In some embodiments, the defect repair system 360 may be a robotic repair system. Information regarding the performance of the robotic repair unit, such as the time to repair a given defect and the time to move between defects, may also be considered by the defect prioritization system 354 when creating a defect repair plan. The defect repair system may include a sanding robot 362 including an abrasive applicator 364 and a water and / or abrasive dispenser 366. The defect repair system may also include a polishing robot 370 having an abrasive applicator 372 and a water and / or abrasive fluid dispenser 374. The sanding robot 362 and the polishing robot 370 may be separate robotic systems or may be part of a single robotic repair system. The defect repair system 360 may also include a verification system 380 that can inspect the repaired defects for quality.

[0037] Figure 4 is a block diagram of a defect detection and ranking system. Defect detection and ranking system 400 is similar to defect detection and ranking system 350 of Figure 3. The defect detection and ranking system includes a defect detection system 410 that outputs a preliminary list of defects and locations 420, and a defect prioritization system 430 that outputs a prioritized list of defects and locations 480.

[0038] The defect detection system 410 includes one or more cameras 412, one or more lighting systems 414, and an image stitching and defect detection system 416. The lighting system 414 illuminates a portion of the vehicle while the camera 412 captures images or video of the vehicle. The image stitching and defect detector 416 includes a processor that reviews the captured images or video to detect defects on the imaged surface of the vehicle. Reviewing may, in some embodiments, include stitching the images.

[0039] The detected defects are output from the defect detection system 410 as a list of defects 420. The preliminary list of defects includes, for each defect, a location classification and a severity level. The defect detector 416 may output the locations in a coordinate system format. The coordinate system format of the preliminary defect list 420 may be provided to a repair system 494 for final repair.

[0040] Defect prioritization system 430 receives preliminary defect list 420. The defect prioritization system evaluates and prioritizes preliminary defect list 420 using information stored in vehicle information database 440, defect database 450, and feedback database 470.

[0041] The vehicle information database 440 contains information about the vehicles being evaluated. A given vehicle has a model 444, a color 446, a detailed specification 442, and other characteristics 448.

[0042] The defect database 450 includes the defects identified in the preliminary list of defects 420. The identification and repair system can be used to identify and repair many different types of surface defects. Some examples of defects that can be identified using the defect detection system 410 include scratches 452, dents 454, excess paint 456, haze 462, irregular surfaces 464, inconsistent gloss 466, or other defects 458.

[0043] In some embodiments, the defect detection and ranking system 400 includes some machine learning capabilities so that the generated repair plans can be improved over time. A user-driven feedback database 470 receives and stores user feedback provided about previous vehicles to improve future use.

[0044] For example, a user may provide feedback regarding the classified defect severity 472, e.g., indicating that the severity was incorrect. Additionally, feedback may be provided regarding the assigned defect type 474. Vehicle details 476 may also be corrected using the user feedback interface. Defect location 478, as well as the potential area affected by a given defect, may also be corrected. Generally, defect details are identified based on images of the vehicle captured by the defect detection system 410. User feedback helps ensure that defect identification, classification, and location information becomes more accurate over time.

[0045] A list of reconciled defects 480 is provided, which may be presented on a user interface for an operator to review and provide feedback 482, which may be incorporated into a user-driven feedback database 470 for future use.

[0046] Once defects are found on a vehicle, a repair decision 490 must be made. If there are few defects and / or they are easily repaired, the vehicle can go through an on-site repair process 494. However, if there are too many defects or the defects cannot be easily repaired on the assembly line, the vehicle can be diverted for a more intensive repair.

[0047] In one embodiment, the reconciled defect list 480 is provided along with repair recommendations, which may be presented, for example, to a user at a repair decision point 490. The user can review the repair recommendations and, for example, either repair the vehicle on-site or divert it as recommended, or the user can disagree with the recommendation and override it.

[0048] The adjusted defect list 480 may be provided as a prioritized list of defects and defect locations. The prioritization may include a recommended repair order for defects requiring repair. The prioritization may also include one or more defects that do not need to be repaired. For defects for which repair is recommended, the recommendation may also include a product recommendation 496 for the repair process.

[0049] 5 is a block diagram of a networked vehicle repair system. The defect detection and ranking system 500 is accessible, in one embodiment, over a network 500. As used herein, the term network 500 includes a wired network, a wireless network, or a cloud-based network.

[0050] The defect detection system 510 inspects the vehicle using a vision system 511. The vision system 511 includes one or more light sources for illuminating the vehicle and one or more image capture systems, such as cameras or video cameras. Defects are identified based on the captured images.

[0051] For each defect, a defect type is identified using defect type identifier 515. Defect type identifier 515 identifies each defect based on an image of the defect captured by vision system 511. For example, defect type identifier 516 can detect surface discontinuities corresponding to scratches. Dents or excess paint can be detected by shadows that should not be present on the surface. For each detected defect, defect severity identifier 514 assigns a severity level. For example, deep scratches are more severe than shallow scratches. Long scratches are more severe than short scratches.

[0052] The coordinate system identifier 512 identifies the coordinate system of the vehicle. The defect location identifier 516 can use the coordinate system of the vehicle to identify the defect center and defect area for at least some defects. For example, in the case of a piece of debris trapped during the painting process, the center of the defect can be identified, as well as the total area that needs to be sanded during repair. In the case of a scratch, the center of the defect can be identified, as well as its length and direction.

[0053] The defect prioritization system 520 retrieves identified defects from the defect detection system 510 using a defect list retriever 522. The identified defects may be provided with location, type, and severity. The defect prioritization system also retrieves vehicle specifications from a vehicle specification database 540 using a vehicle specification retriever 521.

[0054] A vehicle specification database 540 stores information about vehicles in the manufacturing assembly process. The vehicle specification includes a three-dimensional map of the vehicle's surface to which a defect coordinate system can be matched. A vehicle currently being evaluated for repair decisions can have a model 541, one or more colors 542, and one or more options 543 (e.g., for an automobile, two or four doors, spoilers, door trim, hood ornaments, etc.). A vehicle can also have a destination 544, which may be an intended market or, more specifically, a dealer. Additionally, each vehicle can have a paint type 546 and a paint application process 545. For example, most vehicles have several layers of applied paint, as well as several layers of clear coat. One or more primer layers may also be applied. Each layer has an associated thickness and curing method. Curing may include air-drying the vehicle or subjecting it to a heat or other curing process to ensure that the layers are completely dry between applications.

[0055] Vehicle specification retriever 521 retrieves information including a surface map of the vehicle from vehicle specification database 540. Different specifications retrieved may cause the defect prioritization system 520 to rank the retrieved defects differently. For example, scratches are much more visible to a customer on a black vehicle than on a lighter color. Excess paint is not visible on silver or other metallic colors, etc.

[0056] The defect prioritization retriever 523 retrieves prioritization information from a defect prioritization database 530. The retrieved priorities may also be based on the vehicle specification information retrieved by the vehicle specification retriever 521. Different vehicle models have different model priorities 531. For example, a truck will have a different priority than a sedan, etc., regarding what defects must be repaired. Vehicles with different color schemes will also have different color priorities 532. Different destination markets may also have different market priorities 534. Different classes of vehicles will also have different vehicle class priorities 535. For example, a yacht will have different defect priorities and criteria than a low-end powerboat.

[0057] In one embodiment, the data in the defect prioritization database 530 may be stored as a lookup table 536. However, other storage formats 537 are also envisioned.

[0058] The repair option retriever 526 retrieves information from the repair database 550. For example, current information regarding a repair chain 560 for an identified defect may be retrieved. The repair chain 560 includes telemetry 562 associated with the defect, including the defect's location, and telemetry regarding the repair robot's movements after a robotic repair attempt has been made. The repair chain 560 also includes images 564 associated with the repair, including pre-repair images and post-repair images taken during repair verification. Other information 568 may also be retrieved. The repair chain 560 may also retrieve information about previous defect repairs that may be relevant to the current repair. For example, telemetry 562 about a previously repaired scratch on the hood may be referenced by the defect prioritization generator 528 when determining a repair plan and estimated repair time. The repair chain 560 for a given defect may also include a product 566 used in the repair. For example, the abrasive grit, the type and amount of abrasive, the amount of water or other fluid supplied, or other product parameters may be stored as a product 566 associated with the defect repair.

[0059] The repair option retriever 526 can also retrieve other information from the repair database 550, such as available sanding abrasives 552 and sanding robot parameters 554. For example, different abrasive grits may be available and / or different fluids that support sanding may be available. Additionally, different sanding abrasives 556 may be available along with sanding robot parameters 558.

[0060] The robot status retriever 527 communicates with the reconditioning system 570 to determine the status of the reconditioning robot. For example, if additional sanding or abrasive material is needed or if there are any errors reported, the robot status retriever 527 can retrieve such information and present it on the user interface 480.

[0061] The defect prioritization generator 528 automatically generates a prioritized defect list using the specifications retrieved by the vehicle specification retriever 521, the defect prioritization information retrieved by the defect prioritization retriever 523, and the list of defects retrieved by the defect list retriever 522. For example, the defect prioritization generator 528 can select defects in the high priority area as defects that must be repaired and can order those defects in a repair plan based on information retrieved from the repair option retriever 526 and the robot status retriever 527. For example, using known telemetry information 562 from previous repairs, the defect prioritization generator can optimize the robot travel path to address the highest priority defects in the shortest amount of time.

[0062] The networked repair system 500 may include a user interface 580. The user interface 580 may enable an operator of the repair portion of the assembly line to interact with the defect prioritization system 520. Among other things, the user interface 580 may present a prioritized defect list 582 and repair recommendations 584. The repair recommendations 584 may include a repair plan including the order of defects to repair and / or recommended products, as well as telemetry for each defect, the amount of product to be delivered, dwell time, and other parameters. Repair prioritization may also be determined by the availability of repair cycles available on the assembly line versus a stop station versus an offline repair station. If a large number of non-critical defects are identified but there is high demand for vehicle production during that production run, defect repair may be deprioritized. Alternatively, defect repair may also be reprioritized if the stop station and offline repair station are unavailable or full.

[0063] User interface 580 may also allow an operator to interact with repair lineage 560, such as viewing defect images 564, product usage during repair 566, or other information 568 related to the repair. Additionally, using user interface 580, an operator may be able to provide feedback regarding prioritized defect list 582 and repair recommendations 584. For example, the operator may be able to change the priority of defects on prioritized defect list 582 or enter a different repair plan 584 for a given defect.

[0064] 6 is a block diagram of a method for prioritizing defects for repair. Method 600, in one embodiment, is performed automatically by a processor of a repair prioritization system.

[0065] In block 610, a prioritization specification is retrieved. For example, the prioritization specification includes information about the vehicle 612 being evaluated. For example, the vehicle's make, model, color, detailing options, etc. may be retrieved. A prioritization specification 614 is also retrieved, including vehicle grade priority, destination priority, and other related priorities for the given vehicle. A repair specification 616 is also retrieved, including information about previous repairs of similar vehicles. In addition, previous user feedback may also be retrieved.

[0066] At block 620, a list of detected defects is received. The defects are detected on the vehicle using a vision system that often includes one or more light sources and one or more cameras that capture images of the vehicle. The vision system also includes a processor that detects the defects based on the captured images of the vehicle. The list of defects is often provided as a set of coordinates associated with each defect. Each defect may also include a defect type and / or severity ranking.

[0067] At block 630, a prioritized list of defects is generated. The prioritized list of defects is generated automatically by applying the retrieved prioritization specifications to the retrieved list of defects. For example, depending on the vehicle, different characteristics may result in different defects being prioritized higher. Prioritization of the defects may consider color 632, defect location 634, relative position of the defect to the surface of the vehicle 636, defect type 638, captured defect image 642, destination market 644, and / or vehicle grade 646. Prioritization may include ordering the defects in repair order. Prioritization may also include grouping the defects into groups such as "repair" or "do not repair" depending on the retrieved prioritization specifications.

[0068] The generated prioritized defect list is provided at block 640. The prioritized defect list may be provided to an assembly line operator for review, for example, on a user interface. Providing the prioritized defect list may also include providing instructions to a repair robot to repair at least some of the defects on the prioritized defect list.

[0069] At block 650, feedback is received. In one embodiment, the feedback is received from an operator via a user interface based on the prioritized defect list. However, in some embodiments, the feedback is provided only after the repairs are completed.

[0070] Benefits of using the systems and methods described herein include improved efficiency of defect repair. Human-based repair plans often include biases, such as limiting the necessary human movement around and in front of the vehicle and ensuring defects are not overlooked. For this reason, humans often repair defects on vehicles in a methodical manner, from top to bottom and front to back. Using a robot to repair defects and having an automated system to prioritize defects for repair eliminates the bias of human repair technicians without overlooking defects that need to be repaired. Defects are also considered and ranked objectively by measurable criteria, thereby eliminating the subjectivity of human repair technicians. Additionally, the time required to repair defects is reduced. Blocks 610-640 of method 600 can be accomplished in less than about five minutes in some embodiments. In some embodiments, blocks 610-640 can be accomplished in less than about one minute. This results in a more objective repair plan based on customizable criteria that is delivered faster than by human review.

[0071] 7 illustrates a vehicle repair system. System 700 illustrates one exemplary system in which the systems and methods described herein may be used. However, the systems and methods described herein are not limited to the system illustrated in FIG. 7.

[0072] The controller 710 provides instructions to a robot motion controller 730, which moves the robot 720. The movement of the robot 720 allows a sanding tool 724 to contact the surface of the vehicle. The sanding tool 724 may include a sanding or polishing tool. The sanding tool 724 may also include a fluid dispenser capable of dispensing water, abrasive, or other fluid. A force compliance mechanism 722 can adjust the amount of force applied to the surface by the sanding tool 724. Different amounts of force over different dwell times remove different amounts of paint from the vehicle surface. While FIG. 7 shows only one robot 720 for simplicity, in other embodiments, there may be more robots or additional sanding tools 724.

[0073] In some embodiments, a verification robot 740 is also present in the system 700. However, although the verification robot 740 is shown as a separate system that receives control signals from the controller 710, it is also understood that in some embodiments, the verification system may be included as part of the repair robot 720.

[0074] The controller 710, in one embodiment, generates a repair plan for the repair robot 720. The repair plan includes an order for the defects to be repaired based on a prioritized list of the defects to be repaired. The controller 710, in one embodiment, can also generate the prioritized list of defects based on information received about the vehicle. The information about the vehicle and the vehicle's repair priority comes from the system 750. The system 750 can also include a user interface that allows an operator to review the prioritized list of defects and / or the repair plan and provide feedback to improve future repair prioritization.

[0075] FIG. 8 is an exemplary user interface for a vehicle repair system. FIG. 8 shows a computer 800 shown with a user interface display screen 802. The screen 802 may be a touch screen or a pen-enabled interface that receives input from a pen or stylus. The computer 800 may also use an on-screen virtual keyboard. Of course, the computer 800 may be attached to a keyboard or other user input device via a suitable attachment mechanism, such as, for example, a wireless link or a USB port. The computer 800 may also illustratively receive voice input. While the computer 800 is shown as a tablet, in other embodiments, the computer 800 may also be a desktop or laptop computer.

[0076] On the screen 802, in one embodiment, there is a preliminary list 804 of coordinates. These coordinates may be the coordinates output from the defect detection system prior to prioritization. In a traditional repair system, these coordinates would serve as a reference for a human repair technician to repair the vehicle.

[0077] Also, in one embodiment, there is a prioritized defect list 806 that includes the location, defect type, and severity for each defect. In some embodiments, each defect in the list 806 may be selectable, allowing the operator to see which priority contributed to the ranking. Each defect in the list 806 may also be configurable, in one embodiment, as a form of user feedback.

[0078] Information about the vehicle being evaluated may also be presented in information box 810. For example, a silver Honda™ CR-V is presented for a dealer in Tokyo, Japan.

[0079] Repair recommendations may also be provided, as shown in block 820. For example, based on the detected defects, the cited vehicle may be repaired on-site. Repair recommendations 820 may, in some embodiments, be selectable, allowing an operator to review the repair instructions sent to the repair robot, including telemetry and the abrasive product used.

[0080] FIG. 9 is a block diagram of a defect detection and ranking system architecture. Remote server architecture 900 illustrates one embodiment of an implementation of defect detection and ranking system 910. As an example, remote server architecture 900 can provide computational, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system delivering the services. In various embodiments, the remote server can deliver services over a wide area network, such as the Internet, using an appropriate protocol. For example, the remote server can deliver applications over the wide area network, which can be accessed via a web browser or any other computing component. The software or components shown or described in FIGS. 1-8 and corresponding data can be stored on a server at a remote location. Computing resources in a remote server environment can be consolidated at a remote data center location, or the computing resources can be distributed. The remote server infrastructure can deliver services through a shared data center, even if it appears to users as a single point of access. Thus, the components and functionality described herein can be provided from a remote server at a remote location using a remote server architecture. Alternatively, the components and functionality described herein may be provided by a conventional server installed directly or otherwise on the client device.

[0081] In the example shown in Figure 9, some items are similar to those shown in previous figures. Figure 9 specifically shows that defect detection and ranking systems can be located at a remote server location 902. Thus, computing device 920 accesses those systems via remote server location 902. Operator 950 can similarly access user interface 922 using computing device 920.

[0082] FIG. 9 also illustrates another example of a remote server architecture. FIG. 9 illustrates that it is contemplated that some elements of the system described herein are located at the remote server location 902, while others are not. By way of example, storage devices 930, 940, or 960, or repair system 970, may be located at a location separate from location 902 and accessed via a remote server at location 902. Regardless of where storage devices 930, 940, or 960, or repair system 970, are located, they may be accessed directly by computing device 720, over a network (either a wide area network or a local area network), hosted at a remote site by a service, offered as a service, or accessed by a connectivity service residing at a remote location. Additionally, data may be stored in virtually any location and intermittently accessed by or transferred to interested parties. For example, a physical carrier may be used instead of or in addition to an electromagnetic wave carrier.

[0083] It should also be noted that the elements of Figures 1-5, or portions thereof, may be located on a wide variety of different devices, some of which include servers, desktop computers, laptop computers, tablet computers, or palmtop computers, as well as other mobile devices such as mobile phones, smartphones, multimedia players, and personal digital assistants.

[0084] 10-11 show examples of mobile devices that can be used in the embodiments shown in the previous figures.

[0085] Figure 10 is a simplified block diagram of one illustrative embodiment of a handheld or mobile computing device that may be used as a user or client handheld device 16 (e.g., as computing device 920 of Figure 9) on which the present system (or portions thereof) may be deployed. For example, a mobile device may be deployed within the operator compartment of computing device 920 for use in generating, processing, or displaying data. Figure 11 is another example of a handheld or mobile device.

[0086] Figure 10 provides a general block diagram of components of a client device 1016 that may execute and interact with, or execute and interact with, some of the components shown in Figures 1-5. A communication link 1013 is provided in the device 1016 that allows the handheld device to communicate with other computing devices and, in some embodiments, provides a channel for automatically receiving information, such as by scanning. Examples of communication link 1013 include those that enable communication via one or more communication protocols, such as wireless services used to provide cellular access to networks, as well as protocols that provide local wireless connections to networks.

[0087] In another embodiment, the application may be received on a removable Secure Digital (SD) card connected to interface 1015. Interface 1015 and communication link 1013 communicate with processor 1017 (which may also embody a processor) along bus 1019, which is also connected to memory 1021 and input / output (I / O) components 1023, as well as clock 1025 and position system 1027.

[0088] I / O components 1023, in one embodiment, are provided to facilitate input and output operations, and device 1016 may include input components such as buttons, touch sensors, optical sensors, microphones, touch screens, proximity sensors, accelerometers, orientation sensors, etc., and output components such as display devices, speakers, and / or printer ports. Other I / O components 1023 may be used as well.

[0089] The clock 1025 illustratively includes a real-time clock component that outputs the time and date. The clock 1025 may also provide timing functions for the processor 1017.

[0090] Illustratively, the location system 1027 includes components that output the current geographic location of the device 1016. This may include, for example, a global positioning system (GPS) receiver, a LORAN system, a dead-receipt system, a cellular triangulation system, or other positioning systems. The location system 1027 may also include, for example, mapping or navigation software that generates desired maps, navigation routes, and other geographic features.

[0091] The memory 1021 stores an operating system 1029, network settings 1031, applications 1033, application configuration settings 1035, data store 1037, communication drivers 1039, and communication configuration settings 1041. The memory 1021 may include all types of tangible, volatile and non-volatile computer-readable memory devices. The memory 1021 may also include computer storage media (described below). The memory 1021 stores computer-readable instructions that, when executed by the processor 1017, cause the processor to perform computer-implemented steps or functions in accordance with the instructions. The processor 1017 may be activated by other components to facilitate their functions as well.

[0092] 11 shows that the device may be a smartphone 1071. The smartphone 1071 has a touch-sensitive display 1073 that displays icons or tiles or other user input mechanisms 1075. The mechanisms 1075 can be used by a user to run applications, make phone calls, perform data transfer operations, etc. Generally, smartphones 1071 are built on a mobile operating system and offer more advanced computing power and connectivity than feature phones.

[0093] It should be noted that other configurations of device 1016 are possible.

[0094] FIG. 12 is a block diagram of a computing environment that can be used in the embodiments shown in the previous figures.

[0095] FIG. 12 is an example of a computing environment in which elements of FIGS. 1-5, or portions thereof (for example), may be deployed. Referring to FIG. 12, an exemplary system for implementing some embodiments includes a general-purpose computing device in the form of a computer 1110. Components of the computer 1110 may include, but are not limited to, a processing unit 1120 (which may include a processor), a system memory 1130, and a system bus 1121 that couples various system components including the system memory to the processing unit 1120. The system bus 1121 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The memory and programs described with respect to FIGS. 1-5 may be deployed in the corresponding portions of FIG. 12.

[0096] The computer 1110 typically includes a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by the computer 1110 and includes both volatile and nonvolatile media and removable and non-removable media. By way of example, and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media is distinct from and does not include a modulated data signal or carrier wave. Computer storage media includes hardware storage media, including both volatile and nonvolatile media and removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage devices, or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computer 1110. Communication media may embody computer-readable instructions, data structures, program modules, or other data in a transport mechanism and include any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0097] The system memory 1130 includes computer storage media in the form of volatile and / or nonvolatile memory such as read only memory (ROM) 1131 and random access memory (RAM) 1132. A basic input / output system 1133 (BIOS), containing the basic routines that help to transfer information between elements within the computer 1110, such as during start-up, is typically stored in the ROM 1131. The RAM 1132 typically contains data and / or program modules that are immediately accessible to and / or presently being operated on by the processing unit 1120. By way of example, and not limitation, FIG. 12 illustrates operating system 1134, application programs 1135, other program modules 1136, and program data 1137.

[0098] Computer 1110 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, Figure 12 illustrates hard disk drive 1141, which reads from and writes to non-removable, non-volatile magnetic media: nonvolatile magnetic disk 1152, optical disk drive 1155, and nonvolatile optical disk 1156. Hard disk drive 1141 is typically connected to system bus 1121 through a non-removable memory interface, such as interface 1140, and optical disk drive 1155 is typically connected to system bus 1121 by a removable memory interface, such as interface 1150.

[0099] Alternatively, or additionally, the functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-a-Chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0100] The drives and their associated computer storage media, discussed above and illustrated in Figure 12, provide storage of computer-readable instructions, data structures, program modules, and other data for the computer 1110. In Figure 12, for example, hard disk drive 1141 is illustrated as storing operating system 1144, application programs 1145, other program modules 1146, and program data 1147. Note that these components can either be the same as or different from operating system 1134, application programs 1135, other program modules 1136, and program data 1137.

[0101] A user may enter commands and information into the computer 1110 through input devices such as a keyboard 1162, a microphone 1163, and a pointing device 1161, such as a mouse, trackball, or touch pad. Other input devices (not shown) may include a joystick, game pad, satellite receiver, scanner, or the like. These and other input devices are often connected to the processing unit 1120 through a user input interface 1160 that is coupled to the system bus, but may also be connected by other interface and bus structures. A visual display 1191 or other type of display device is also connected to the system bus 1121 via an interface, such as a video interface 1190. In addition to the monitor, computers may also include other peripheral output devices such as speakers 1197 and printer 1196, which may be connected through an output peripheral interface 1195.

[0102] The computer 1110 operates in a networked environment using logical connections, such as a local area network (LAN) or a wide area network (WAN), to one or more remote computers, such as a remote computer 1180.

[0103] When used in a LAN networking environment, the computer 1110 is connected to the LAN 1171 through a network interface or adapter 1170. When used in a WAN networking environment, the computer 1110 typically includes a modem 1172 or other means for establishing communications over the WAN 1173, such as the Internet. In a networked environment, program modules may be stored in remote memory storage devices. Figure 12 illustrates, for example, that remote application programs 1185 may reside on the remote computer 1180.

[0104] It should also be noted that the different embodiments described herein can be combined in different ways, i.e., portions of one or more embodiments can be combined with portions of one or more other embodiments, all of which are contemplated herein.

[0105] Embodiment Embodiment 1 is a defect detection and ranking system for a vehicle assembly line. The system includes an image capture device that captures a plurality of images of vehicles on the vehicle assembly line. The system also includes a data store that includes vehicle specifications for the vehicles on the vehicle assembly line and defect priorities based on the vehicle specifications. The system also includes a defect detector that analyzes the plurality of captured images and detects a plurality of defects on a surface of the vehicle based on the analysis. Each defect detector associates, for each of the plurality of defects, an x-y-z coordinate location, a defect type, and a defect severity. The system also includes a defect prioritization generator configured to receive the plurality of defects from the defect detector, extract the vehicle specifications and the defect priorities, apply the defect priorities to the plurality of defects, and generate a list of prioritized defects. The defect prioritization generator outputs the list of prioritized defects to an output device associated with the vehicle assembly line.

[0106] Embodiment 2 includes the features described in embodiment 1, except that the defect type is one of a scratch, a dent, or excess paint.

[0107] Example 3 includes the features of example 1 or 2, except that the plurality of defects includes a first defect having a first defect location and a second defect having a second defect location. The defect priority is the location of the defect. The first defect is prioritized higher than the second defect.

[0108] Embodiment 4 includes the features described in embodiment 3, except that the vehicle is a passenger car, the first defect is located on a driver's side door of the passenger car, and the second defect is located on a roof of the passenger car.

[0109] A fifth embodiment includes any one of the features of the first to fourth embodiments, except that the vehicle specification is a first vehicle color, one defect type of the plurality of defects is a scratch, and the scratch has a higher defect priority for the first vehicle color than for the second vehicle color.

[0110] Example 6 includes the features described in Example 5, except that the first color is black and the second color is silver.

[0111] Example 7 includes the features described in Examples 1-6, but also includes a sensor configured to identify the vehicle, and the defect prioritization generator automatically derives the vehicle specification and defect priority based on the identified vehicle.

[0112] An eighth embodiment includes the features described in the seventh embodiment, except that the defect prioritization generator automatically applies vehicle specifications and defect priorities when multiple defects are received.

[0113] Example 9 includes the features of any one of Examples 1 to 8, except that the defect prioritization generator also provides a repair recommendation along with the prioritized list of defects, the repair recommendation being either to repair the vehicle on-site or to divert the vehicle for repair.

[0114] A tenth embodiment includes the features described in the ninth embodiment, except that the repair recommendation also includes an instruction not to repair a subset of the plurality of defects.

[0115] Example 11 includes the features of example 10, except that the repair recommendation also includes instructions to a repair robot to repair the at least one defect. The instructions include telemetry to the repair robot.

[0116] Example 12 includes the features described in Example 11, but the instructions also include an abrasive product, a force to apply, a trajectory, and a dwell time for at least one defect.

[0117] Embodiment 13 includes the features described in any one of embodiments 1-12, except that the defect priority is selected from model priority, color priority, defect location priority, destination priority, vehicle grade priority, vehicle paint priority, and paint application priority.

[0118] Example 14 includes the features of any one of Examples 1-13, except that the captured images are stored in a repair lineage database. The defect prioritization generator retrieves images of the defects from the repair lineage database. The defect prioritization generator prioritizes the defects based at least in part on the retrieved images.

[0119] Example 15 includes the features of any one of examples 1-14, except that the output device includes an input device configured to receive an indication of user feedback, which changes the order of the defects in the prioritized list of defects.

[0120] Embodiment 16 includes the features described in embodiment 15, except that the indicator of user feedback is stored in a data store.

[0121] A seventeenth embodiment includes the features of any one of the first to sixteenth embodiments, except that the vehicle specification includes a three-dimensional map of the vehicle's surface.

[0122] Embodiment 18 is a computer-implemented method for ranking detected surface defects on a vehicle on an assembly line. The method includes receiving an identification of the vehicle. The method also includes accessing a defect database stored in a defect memory using a defect list searcher to receive a plurality of detected surface defects on the vehicle. Each of the plurality of detected surface defects has a defect location, a defect type, and a defect severity. The method also includes accessing the specification database stored in the specification memory using a specification searcher and automatically retrieving a first vehicle specification and a second vehicle specification based on the identification of the vehicle using the vehicle specification searcher. The method also includes accessing a defect priority database stored in a priority memory using a defect priority searcher to automatically retrieve a first defect priority based on the first vehicle specification and a second defect priority based on the second vehicle specification. The method also includes generating a defect priority list by applying the first defect priority and the second defect priority to the plurality of defects using a defect prioritization generator. The defect priority list includes a ranking of the plurality of detected surfaces such that a first defect of the plurality of defects is ranked higher than a second defect of the plurality of defects based on applying a first defect priority and a second defect priority to the defect location, the defect type, and the defect severity. The method also includes providing the defect priority list as output to an output device associated with the assembly line. The steps of receiving the identification, receiving the plurality of detected surface defects, retrieving a vehicle specification, generating the defect priority list, and providing the defect priority list are performed by a non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing device, cause the at least one processor to perform a defect list searcher step, a vehicle specification searcher step, a defect priority list step, and a defect priority list providing step.

[0123] Example 19 includes the features of example 18, except that the first vehicle specification and the second vehicle specification are selected from the group consisting of a vehicle model, a vehicle color, a vehicle option, a vehicle destination, a vehicle paint type, and a vehicle paint application.

[0124] Embodiment 20 includes the features described in embodiment 18 or 19, except that the first defect priority and the second defect priority are selected from the group consisting of defect location priority, defect classification priority, vehicle color priority, vehicle grade priority, and relative defect location priority.

[0125] Example 21 includes the features of any one of Examples 18-20, but also includes imaging the vehicle. The imaging includes capturing a plurality of images using a camera and stitching the plurality of images using a processor. The method also includes detecting a plurality of surface defects on the vehicle. The detecting includes analyzing the stitched images by a processor for potential defects. The defect detection processor identifies defect characteristics. The defect characteristics include a defect location and a defect type for each detected surface defect.

[0126] Embodiment 22 includes the features of any one of embodiments 18 to 21, except that the output device comprises a screen, and the defect priority list is provided on the screen.

[0127] Example 23 includes the features of any one of Examples 81 to 22, except that the output device also includes an input device configured to receive user feedback regarding the defect priority list, the user feedback being automatically associated with the first vehicle specification and the second vehicle specification.

[0128] Example 24 includes the features of example 23, except that the vehicle is a first vehicle. Upon receiving an identification of a second vehicle having the first vehicle specification and the second vehicle specification, user feedback is retrieved and applied to a second plurality of defects associated with the second vehicle.

[0129] Example 25 includes the features of any one of Examples 18 to 24, except that the output device also includes an input device configured to receive user feedback regarding the defect priority list, the user feedback being automatically associated with the first defect priority and the second defect priority.

[0130] Example 26 includes the features of example 25, except that the vehicle is a first vehicle. Upon receiving an identification of a second vehicle having a first defect priority and a second defect priority, user feedback is retrieved and applied to a second plurality of defects associated with the second vehicle.

[0131] Example 27 includes the features of any one of Examples 18 to 26, except that the first vehicle specification is a vehicle model, the first defect priority is a defect location, and the prioritization of a first detected defect location and vehicle model is higher than for a second detected defect having a second location.

[0132] Example 28 includes the features of any one of Examples 18-27, except that the first vehicle specification is a first vehicle color. The defect priority is a defect type. A first detected defect having a first defect type is prioritized higher than a second detected defect having a second defect type.

[0133] Embodiment 29 includes the features described in embodiment 28, except that for a second vehicle having a second vehicle color different from the first vehicle color, the second detected defect is prioritized higher than the first detected defect.

[0134] Embodiment 30 is a system for prioritizing vehicle repairs on an assembly line. The system includes a vehicle specification database containing specifications for vehicles on the assembly line, the database including a first specification and a second specification. The first specification is a vehicle color index for the vehicle. The system also includes a defect database containing a list of detected defects on the vehicle. Each detected defect has an associated defect location and defect type. The system also includes a non-transitory computer-readable medium having instructions stored thereon, which, when executed by a processor, are configured to: retrieve the first specification and the second specification; and derive a first priority based on the first specification and a second priority based on the second specification. The first priority is a color priority. The computer-readable medium is also configured to retrieve the list of detected defects, apply the first priority and the second priority to each defect on the list of defects to determine a priority of the defects to be repaired, thereby generating a prioritized list of defects, and output the prioritized list of defects to an output device associated with the assembly line.

[0135] Embodiment 31 includes the features described in embodiment 30, but the system is a networked system.

[0136] Embodiment 32 includes the features described in embodiment 31, but the system also includes a repair system in communication with the vehicle repair prioritization system.

[0137] Example 33 includes the features of example 32, except that the repair system includes a repair pedigree database that stores indicia of previously repaired defects, the indicia including at least one of a before-repair image, an after-repair image, telemetry of the robotic repair, and the type and amount of abrasive product applied.

[0138] Embodiment 34 includes the features described in any one of embodiments 30 to 33, except that the processor automatically retrieves the specifications, retrieves the detected defects, and generates a prioritized list of defects based on received indications that the vehicle is ready for repair.

[0139] Example 35 includes the features described in example 34, except that the indication is received from a sensor that detects a vehicle on the assembly line.

[0140] Example 36 includes the features of any one of Examples 30-35, except that the list of detected defects is provided from a defect detection system including a camera and a processor configured to analyze images captured by the camera, and the processor is configured to detect defects on the vehicle and assign defect locations and defect types to the defects based on the analysis.

[0141] Embodiment 37 includes the features of any one of embodiments 30-36, except that the second vehicle specification is selected from the group consisting of a vehicle model, a vehicle destination, a vehicle paint type, a vehicle paint application parameter, or a vehicle option.

[0142] Embodiment 38 includes the features of any one of embodiments 30 to 37, except that the prioritized list is an ordered list of defects to be repaired.

[0143] Embodiment 39 includes the features described in any one of embodiments 30 to 38, except that the prioritized list includes a first subset of detected defects designated to be repaired and a second subset of defects designated not to be repaired.

[0144] Example 40 includes the features of any one of Examples 30 to 39, but the processor is also configured to output a recommendation for repairing the vehicle, the recommendation being either to repair the vehicle on the assembly line or to divert the vehicle for repair.

[0145] Embodiment 41 includes the features described in embodiment 40, but the repair recommendation also includes telemetry to a repair robot for repairing multiple defects on the prioritized defect list.

[0146] Example 42 includes the features described in example 41, except that the repair recommendations also include product recommendations for defects on the prioritized defect list.

[0147] Embodiment 43 includes the features described in embodiment 42, except that the product recommendation is based in part on the previous repair history of a previous vehicle having the first specification and the second specification.

[0148] Embodiment 44 includes the features described in embodiment 42, except that the product recommendation is based in part on the previous repair history of previous vehicles with defects similar to the defect.

[0149] Embodiment 45 includes the features described in embodiment 42, except that the repair recommendations are communicated to a repair robot.

[0150] Embodiment 46 includes the features of any one of embodiments 30 to 45, except that the prioritized list of defects is output to a repair robot assigned to repair the vehicle.

[0151] Example 47 includes the features of any one of examples 30 to 46, except that the output device is a robotic repair unit. The robotic repair unit is configured to perform repairs based on the prioritized list of defects. [Example]

[0152] Example 1: Detecting defects Common sources of typical defects include particles trapped within the applied paint, agglomerations of paint components, scratches, dents, or excess paint. Figure 11A shows a defect caused by a highly reflective paint contaminant particle less than one-tenth of a millimeter in size. Figure 11A shows a magnified defect shown under a microscope on a black paint surface with a clear coat.

[0153] Figure 11B shows a similar defect in the top surface reflection. The defect is detectable and appears as a shadow in the image.

[0154] For ease of understanding, the following examples relate to a small set of defects on a vehicle. However, it should be understood that at many assembly sites, a given vehicle may have numerous surface defects that need to be evaluated and then prioritized for repair.

[0155] Example 2: Prioritizing Defects Table 1 below shows an exemplary defect list output from the defect detection system. The defects are output through a coordinate system along with the defect volume and defect type.

[0156] [Table 1]

[0157] In Table 1, x, y, and z provide the location in three-dimensional space. Volume represents the size of the defect, and the center of the defect in space relates to the generalized volume. The types used in this case are d - drop, s - scratch, and c - dent.

[0158] In Table 1, the defects are originally prioritized based on defect location. However, the user can provide feedback to prioritize larger size defects over their location on the vehicle. This allows the prioritization to change, resulting in a re-prioritized set of defects based on size, resulting in Table 2.

[0159] [Table 2]

[0160] Example 3: Defect Prioritization Table Not all defects require repair. Part of the prioritization scheme includes reviewing the defects and removing defects that do not require repair. Figure 12A shows a sample output of detected defects returned in a lookup table. From left to right, the array includes the sample ID, x, y, z coordinates, size of the defect in millimeters, and defect classification.

[0161] Defects with a value of 0.01 or greater are removed. In addition, surface-bound defects 205, 314, and 400 are high-priority defects. As shown in FIG. 12B, removed defects are not presented on the list of prioritized defects.

[0162] Example 4: Incorporating user feedback 13A and 13B show an example of how user feedback is incorporated into the prioritization of defects for repair. FIG. 13A shows the initial list of defects, and FIG. 13B shows the prioritized list of defects. Defects classified as "d" were user feedback input that should be deprioritized for a given vehicle paint and defect location. Defects with ID numbers 4 and 6 were also indicated as needing to be prioritized based on direct human review of the defects.

[0163] Example 5: On-site repair or bypass Determining early that a vehicle requires more extensive repair than can be performed on the assembly line can save time in the long run. Prioritizing defects to detect defects requiring extensive vehicle repair before repair work begins allows the assembly line to keep moving. Figures 14-16 show a portion of a set of defects for prioritization. The prioritization rules of Example 3 are applied to obtain the set of Figure 14, but additional prioritization is also present, which automatically classifies a vehicle as needing to be diverted for repair if a defect of 0.5 mm or greater in size is detected. Because defect ID number 16 has such a size, the vehicle is diverted for repair.

Claims

1. 1. A vehicle assembly line defect detection and ranking system, comprising: an image capture device that captures a plurality of images of vehicles on the vehicle assembly line; a data store containing vehicle specifications for the vehicles on the vehicle assembly line and defect priorities based on the vehicle specifications; a defect detector that analyzes the plurality of captured images and detects a plurality of defects on a surface of the vehicle based on the analysis, each defect detector associating with each of the plurality of defects an x-y-z coordinate location, a defect type, and a defect severity; 1. A defect prioritization generator comprising: receiving the plurality of defects from the defect detector; Retrieving the vehicle specification and the defect priority; applying the defect priorities to the plurality of defects; Generate a prioritized list of defects, a defect prioritization generator configured to: Equipped with the defect prioritization generator outputs the list of prioritized defects to an output device associated with the vehicle assembly line. Defect detection and ranking system.

2. The defect detection and ranking system of claim 1 , wherein the defect type is one of a scratch, a dent, or excess paint.

3. 3. The defect detection and ranking system of claim 1, wherein the plurality of defects includes a first defect having a first defect location and a second defect having a second defect location, the defect priority is a location of the defect, and the first defect is prioritized higher than the second defect.

4. 4. The defect detection and ranking system of claim 3, wherein the vehicle is a passenger car, the first defect is located on a driver's side door of the passenger car, and the second defect is located on a roof of the passenger car.

5. 5. The defect detection and ranking system of claim 1, wherein the vehicle specification is a first vehicle color, one defect type of the plurality of defects is a blemish, and the defect priority of the blemish is higher for the first vehicle color than for a second vehicle color.

6. 6. The defect detection and ranking system of claim 5, wherein the first color is black and the second color is silver.

7. 7. The defect detection and ranking system of claim 1, further comprising a sensor configured to identify the vehicle, and wherein the defect prioritization generator automatically derives the vehicle specification and the defect priority based on the identified vehicle.

8. The defect detection and ranking system of claim 7 , wherein the defect prioritization generator automatically applies the vehicle specification and the defect priorities when the plurality of defects is received.

9. 9. The defect detection and ranking system of claim 1, wherein the defect prioritization generator also provides a repair recommendation along with the list of prioritized defects, the repair recommendation being either to repair the vehicle on-site or to divert the vehicle for repair.

10. The defect detection and ranking system of claim 9 , wherein the repair recommendation also includes instructions not to repair a subset of the plurality of defects.

11. The defect detection and ranking system of claim 10 , wherein the repair recommendation also includes instructions to a repair robot to repair at least one defect, the instructions including telemetry to the repair robot.

12. The defect detection and ranking system of claim 11 , wherein the instructions also include an abrasive product, a force to apply, a trajectory, and a dwell time for the at least one defect.

13. 13. The defect detection and ranking system of claim 1, wherein the defect priority is selected from model priority, color priority, defect location priority, destination priority, vehicle grade priority, vehicle paint priority, and paint application priority.

14. 14. The defect detection and ranking system of claim 1, wherein the captured images are stored in a repair lineage database, the defect prioritization generator retrieves images of defects from the repair lineage database, and the defect prioritization generator prioritizes the defects based at least in part on the retrieved images.

15. 15. The defect detection and ranking system of claim 1, wherein the output device includes an input device configured to receive an indication of user feedback, the indication of user feedback changing a ranking of defects in the prioritized list of defects.

16. The defect detection and ranking system of claim 15 , wherein the indicators of user feedback are stored in the data store.

17. The defect detection and ranking system of any preceding claim, wherein the vehicle specification comprises a three-dimensional map of the surface of the vehicle.

18. 1. A computer-implemented method for ranking detected surface defects on a vehicle on an assembly line, comprising: receiving an identification of the vehicle; accessing a defect database stored in a defect memory using a defect list searcher to receive a plurality of detected surface defects on the vehicle, each of the plurality of detected surface defects having a defect location, a defect type, and a defect severity; accessing a specification database stored in a specification memory using a specification retriever, and automatically retrieving a first vehicle specification and a second vehicle specification based on the identification of the vehicle using the vehicle specification retriever; using a defect priority retriever to access a defect priority database stored in a priority memory to automatically retrieve a first defect priority based on the first vehicle specification and a second defect priority based on the second vehicle specification; generating, using a defect prioritization generator, a defect priority list by applying the first defect priority and the second defect priority to the plurality of defects, the defect priority list including a ranking of the plurality of detected surfaces such that a first defect of the plurality of defects is ranked higher than a second defect of the plurality of defects based on applying the first defect priority and the second defect priority to the defect location, the defect type, and the defect severity; providing the defect priority list as an output to an output device associated with the assembly line; on a processor, the steps of receiving an identification, receiving a plurality of detected surface defects, retrieving a vehicle specification, generating a defect priority list, and providing a defect priority list are performed by a non-transitory computer-readable storage medium containing instructions that, when executed by at least one processor of a computing device, cause the at least one processor to perform the defect list retriever step, the vehicle specification retriever step, the defect priority list step, and the defect priority list providing step. method.

19. 20. The method of claim 18, wherein the first vehicle specification and the second vehicle specification are selected from the group consisting of a vehicle model, a vehicle color, a vehicle option, a vehicle destination, a vehicle paint type, and a vehicle paint application.

20. 20. The method of claim 18 or 19, wherein the first defect priority and the second defect priority are selected from the group consisting of defect location priority, defect classification priority, vehicle color priority, vehicle grade priority, and relative defect location priority.

21. imaging the vehicle, the imaging step including capturing a plurality of images using a camera and stitching the plurality of images using a processor; detecting the plurality of surface defects on the vehicle, the detecting step including the processor analyzing the stitched images for potential defects, the defect detection processor identifying defect features including the defect location and the defect type for each detected surface defect; The method of any one of claims 18 to 20, further comprising:

22. A method according to any one of claims 18 to 21, wherein the output device comprises a screen and the defect priority list is provided on the screen.

23. 23. The method of any one of claims 18 to 22, wherein the output device also has an input device configured to receive user feedback of the defect priority list, and wherein the user feedback is automatically associated with the first vehicle specification and the second vehicle specification.

24. 24. The method of claim 23, wherein upon receiving an identification of a second vehicle having the first vehicle specification and the second vehicle specification, the vehicle is a first vehicle, the user feedback is retrieved and applied to a second plurality of defects associated with the second vehicle.

25. 25. The method of claim 18, wherein the output device also comprises an input device configured to receive user feedback of the defect priority list, the user feedback being automatically associated with the first defect priority and the second defect priority.

26. 26. The method of claim 25, wherein the vehicle is a first vehicle and upon receiving an identification of a second vehicle having the first defect priority and the second defect priority, the user feedback is retrieved and applied to a second plurality of defects associated with the second vehicle.

27. 27. The method of any one of claims 18 to 26, wherein the first vehicle specification is a vehicle model, the first defect priority is a defect location, and the prioritization of a first detected defect location and vehicle model is higher than for a second detected defect having a second location.

28. 28. The method of any one of claims 18 to 27, wherein the first vehicle specification is a first vehicle color, the defect priority is a defect type, and the prioritization of a first detected defect having a first defect type is higher than a second detected defect having a second defect type.

29. 29. The method of claim 28, wherein for a second vehicle having a second vehicle color different from the first vehicle color, the prioritization of the second detected defect is higher than the first detected defect.

30. 1. An assembly line vehicle repair prioritization system, comprising: a vehicle specification database containing specifications of vehicles on the assembly line, the database including a first specification and a second specification, the first specification being a vehicle color index for the vehicle; a defect database containing a list of detected defects on the vehicle, each detected defect having an associated defect location and defect type; A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by a processor, Taking the first specification and the second specification; extracting a first priority based on the first specification, the first priority being a color priority, and a second priority based on the second specification; Retrieving the list of detected defects; generating a prioritized list of defects by applying the first priority and the second priority to each defect on the list of defects to determine a priority of the defects to be repaired; outputting the prioritized defect list to an output device associated with the vehicle assembly line; a non-transitory computer-readable medium configured to: A system comprising:

31. 31. The system of claim 30, wherein the system is a networked system.

32. 32. The system of claim 31, wherein the system also includes a repair system in communication with the vehicle repair prioritization system.

33. 33. The system of claim 32, wherein the repair system comprises a repair lineage database that stores indicia of previously repaired defects, the indicia including at least one of a before-repair image, an after-repair image, remote measurements of the robotic repair, and the type and amount of abrasive product applied.

34. 34. The system of claim 30, wherein the processor automatically retrieves the specifications, retrieves the detected defects, and generates a prioritized list of defects based on received indications that the vehicle is ready for repair.

35. 35. The system of claim 34, wherein the indication is received from a sensor that detects the vehicle on an assembly line.

36. 36. The system of any one of claims 30 to 35, wherein the list of detected defects is provided from a defect detection system including a camera and a processor configured to analyze images captured by the camera, and wherein the processor is configured to detect defects on the vehicle and assign defect locations and defect types to the defects based on the analysis.

37. The system of any one of claims 30 to 36, wherein the second vehicle specification is selected from the group consisting of a vehicle model, a vehicle destination, a vehicle paint type, a vehicle paint application parameter, or a vehicle option.

38. The system of any one of claims 30 to 37, wherein the prioritized list is an ordered list of defects to be repaired.

39. 39. The system of claim 30, wherein the prioritized list includes a first subset of detected defects designated to be repaired and a second subset of defects designated not to be repaired.

40. 40. The system of any one of claims 30 to 39, wherein the processor is also configured to output a recommendation for repairing the vehicle, the recommendation for repair being either to repair the vehicle on the assembly line or to divert the vehicle for repair.

41. 41. The system of claim 40, wherein the repair recommendation also includes telemetry to a repair robot to repair a plurality of defects on the prioritized list of defects.

42. 42. The system of claim 41, wherein the repair recommendations also include product recommendations for defects on the prioritized list of defects.

43. 43. The system of claim 42, wherein the product recommendation is based in part on a previous repair history of a previous vehicle having a first specification and a second specification.

44. 43. The system of claim 42, wherein the product recommendation is based in part on previous repair history of previous vehicles having a defect similar to the defect.

45. 43. The system of claim 42, wherein the repair recommendation is communicated to a repair robot.

46. A system according to any one of claims 30 to 45, wherein the prioritized list of defects is output to a repair robot assigned to repair the vehicle.

47. 47. The system of any one of claims 30 to 46, wherein the output device is a robotic repair unit, the robotic repair unit configured to perform repairs based on the prioritized list of defects.

Images