Method and system for replacement and repair of vehicle components

Abstract

A vehicle repair system and method for detecting and assessing vehicle damage and guiding the fitting of replacement parts to the vehicle based on the manufacturer's specifications. A computer visualization system identifies the area of damage by collecting a 3D datafile of the damaged vehicle with a 3D scanner and matching it to a 3D datafile of the OEM vehicle standard. The gathered data is used in conjunction with the visualization system that matches the 3D file to the physical vehicle in virtual space allowing the technician to see and interact with an exact match of the physical vehicle on the computer system in virtual space.

Description

METHOD AND SYSTEM FOR REPLACEMENT AND REPAIR OF VEHICLE COMPONENTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to United States provisional patent application US 63 / 486,268 filed 22 February 2023, which is hereby incorporated by reference herein in its entirety.BACKGROUND

[0002] Repairs to various components of damaged vehicles must be conducted accurately to ensure an adequate and safe repair. The specification of vehicle component repairs is dictated by the vehicle manufacturer and repairs or replacement of damaged vehicle components are generally maintained within the original equipment manufacturing (OEM) specifications and standards such that the damaged vehicle is repaired to a high standard. The standards required and auditing of vehicle repairs is becoming more relevant as the accuracy of vehicle safety detection systems such as ADAS (advanced driver assistance system), which assist drivers in driving and parking functions, are becoming more widely used and are dependent on the specific localization and orientation of various component parts. In addition, modem electronic systems can also be used to predict how a vehicle will absorb an impact, and factoring in the specifications of vehicle structural components is especially important in ensuring the desired engineered effect of absorption upon a collision. After a collision, all aspects of vehicle damage must be considered, and all components must be repaired or replaced to return the vehicle to a roadworthy safety standard. Likewise, ADAS systems are reliant upon the structure of the vehicle being aligned or repaired back to Original Equipment Manufacturers (OEM) specifications and must be realigned or repaired to minimum acceptable tolerances of measurement in order for these systems to operate safely.

[0003] The assessment and repair of damaged vehicle components after a collision or damage event can be executed in a number of ways, including using a frame rack to pull components back to their original shape using force to push or pull damaged parts of components. The removal, adjustment, and / or replacement of the component can be done using a specific fixed or universal jigging system, or the replacement and removal of damaged components can be done with the guidance of a 2D or 3D visual display.

[0004] With the replacement of steel components, which are used on older vehicles, with modem aluminum and polymer or printed structural components in more and more vehicles, techniques in vehicle and vehicle component repair have had to adapt. Aluminum has some advantages compared to steel in vehicle parts, most importantly that it is lightweight, easy to mold and not as prone to rust as steel. However, aluminum has a lower fatigue index than steel and will crack and break after a certain amount of bending. As such, aluminum parts generally cannot be forced back to their original shape once deformed because of the nature of aluminum, and deformed aluminum parts are more likely to fracture or break compared to more malleable and durable steel. Techniques that were previously used to push or pull a steel component back to its original shape are not adaptable to more breakable aluminum components. Aluminum vehicle components are also generally riveted and bonded together to allow a more sustainable and workable method for the replacement of the components.

[0005] As a required practice as set out by the OEM when working with aluminum vehicle parts, repairs to vehicles must engage in using a fixed or universal jigging system to assist with the replacement of the components. Fixed and universal jigging systems are used to create a jig that can hold that vehicle component in place at the exact location specified by the OEM. A fixed jigging system is a system that is limited to small movements and fixed jigging systems are generally designed for specific vehicle models for use at specific locations on the vehicle. In contrast, universal jigging systems are designed to allow the user to add various pieces to make a custom jig that is designed for a specific vehicle and location. Both types of these jigging systems are often designed by the manufacturer of the jigging system with the assistance or guidance of the OEM vehicle manufacturer. As new vehicle models are released these jigging systems must be customized to fit specific new model components for specific locations on the vehicle.

[0006] Each jigging system must be, and is designed for, use on a frame rack system and all jigging systems require a user to use jigs with a frame rack system to support the placement of the jigs. One example of a frame rack levelling bench is described in US7,810,245B2 to Henblad et al. which provides a frame rack as a rigid steel platform that the vehicle can drive on, or be pulled on using a winch. The frame rack is designed to attach a pulling arm for pulling and / or pushing damaged components as well as the attachment of jigs for aligning vehicle components. The process of repairing vehicle components using a jigging system is cumbersome and requiresmany steps. In one common example of vehicle repair using a jigging system, the user must first disassemble the vehicle and its components before the vehicle is loaded onto the jigging system. Because the user must place the vehicle on a frame rack and a jigging system must be used, areas of the vehicle that are undamaged must also be disassembled to allow all the adequate jigs to be placed in accordance with the manufacturer of the jigging system. The disassembly of the vehicle is usually done on a two post vehicle hoist first, then the vehicle is driven, lifted, pushed and / or pulled onto the frame rack. Once on the frame rack, the user must align the vehicle at a specific location that will allow for the use of the jigging system. The vehicle must also be levelled and areas of the vehicle around the damaged area must also be set on jigs as to level and align the vehicle so the jigging system can be used to support the desired repair. The building of the jigging system and the set up of the vehicle can also take many hours as the vehicle must be moved from the hoist and set up on the frame rack and jigs.

[0007] As the user changes and fits a new vehicle component for the damaged vehicle, the new component must be placed on the jigging system and measured as it is moved. The measurements must match the measurements of the vehicle manufacturer and jigging system manufacturer. Typical measuring systems are done in 3D with a probe system, such as the Car- O-Liner® vision system. This example of a probe system can be used by inserting a laser probe into one of a few specific locations in the vehicle frame as indicated in a two dimensional (2D) vehicle map shown on a display screen. The system displays the vehicle map to a technician and the technician will identify at which location the probe is measuring. These measurement points are limited to identifiable areas of the vehicle such as bolts, bolt holes and manufacturer jig holes. A measurement is then taken by the probe by pressing the capture button at the specified predetermined location as indicated. The probe system can connect to a computer wirelessly such as via Bluetooth and the measurements are compared to a data set in a software having the OEM standard measurements for the same vehicle type. The results are then displayed on the 2D vehicle map as a measurement of the deviation distance from the standard, where higher numbers are indicative of more deviation and therefore more damage. In this way, collected data identifies how far from the OEM standard the probe measurements are at the predetermined location. In order to make the system useful the user must find three undamaged points of reference of the vehicle to center the measuring probe to the vehicle, creating a field or frame of reference so that the probe system can create a map for localization. In order to get a comparativemeasuring data set, the user must measure the vehicle only at specific and predetermined measuring points as identified by the system that the measuring system manufacturer has premeasured in their facility.

[0008] There remains a need for a flexible vehicle repair system that can assess structural damage to a vehicle and provide a universal jigging system that is not limited to a repair on a frame rack or with probes at specific locations. There also remains a need for a vehicle repair system that can give the user feedback on a part as it is being orientated back to its original proper position and show the user live measurements as the part is being aligned to assist with efficient and accurate vehicle component replacement and vehicle repair.

[0009] This background information is provided for the purpose of making known information believed by the applicant to be of possible relevance to the present invention. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art against the present invention.SUMMARY OF INVENTION

[0010] An object of the present invention is to provide a vehicle imaging, assessment, and repair system that can assist the user in replacing vehicle components to their original location on a vehicle while providing the user live feedback on measurements and placement and avoiding unnecessary disassembly of undamaged vehicle components.

[0011] In an aspect there is provided a computer implemented method comprising: scanning a damaged vehicle with a three dimensional (3D) scanner to obtain a 3D datafile of the damaged vehicle; matching the 3D datafile of the damaged vehicle to a 3D datafile of an original equipment manufacturing (OEM) vehicle standard by matching the make and model of the damaged vehicle, the 3D datafile of the OEM vehicle standard associated with a plurality of vehicle parts for the OEM vehicle standard; creating an image overlay of the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard; identifying an area of damage on the damaged vehicle by comparing the 3D datafile for the OEM vehicle standard to the 3D datafile of the damaged vehicle and identifying a repair area where a measurement threshold between the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard is greater than within a desired tolerance; identifying at least one replacement part in the repair area; and during a repair: imaging a location of the replacement part on the damaged vehicle; andwhen the replacement part is within a placement tolerance on the damaged vehicle such that the replacement part can be aligned on the damaged vehicle.

[0012] In another aspect there is provided a computer implemented method comprising: scanning a damaged vehicle with a three dimensional (3D) scanner to obtain a 3D datafile of the damaged vehicle; matching the 3D datafile of the damaged vehicle to a 3D datafile of an original equipment manufacturing (OEM) vehicle standard by matching the make and model of the damaged vehicle, the 3D datafile of the OEM vehicle standard associated with a plurality of vehicle parts for the OEM vehicle standard; creating an image overlay of the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard; identifying an area of damage on the damaged vehicle by comparing the 3D datafile for the OEM vehicle standard to the 3D datafile of the damaged vehicle and identifying a repair area where a measurement threshold between the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard is greater than within a desired tolerance; identifying at least one replacement part in the repair area; and during a repair: imaging a location of the replacement part on the damaged vehicle; and identifying when the replacement part is within a placement tolerance on the damaged vehicle such that the replacement part can be aligned on the damaged vehicle.

[0013] In an embodiment, the 3D datafile of the damaged vehicle has a scanning accuracy better than 5mm.

[0014] In another embodiment, the method further comprises, during the repair, dynamically tracking the replacement part relative to the damaged vehicle with the 3D scanner.

[0015] In another embodiment, the 3D datafile is a point cloud file, mesh file, standard tessellation language (STL) file, or any other datafile that can represent an object in 3D space within a desired scanning accuracy.

[0016] In another embodiment, the scan is completed after a damaged component has been removed.

[0017] In another embodiment, the 3D datafile of the damaged vehicle has a scanning accuracy greater than 5 mm.

[0018] In another embodiment, the desired tolerance is less than or within about 5mm.

[0019] In another embodiment, the method further comprises displaying a location of the replacement part and damaged vehicle in the image overlay on a graphical user interface.

[0020] In another embodiment, the method further comprises, during the repair, moving the replacement part relative to the damaged vehicle and tracking movement of the replacement part with the 3D scanner.

[0021] In another embodiment, the method further comprises providing an indication when the replacement part is aligned on the vehicle.

[0022] In another embodiment, the indication is represented on a graphical user interface as one or more of color, pattern, and highlighting.

[0023] In another embodiment, the indication comprises one or more of sound, haptic feedback, and displayed distance measurements between the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard.

[0024] In another embodiment, the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard can be manipulated in three dimensions on a graphical user interface.

[0025] In another embodiment, the 3D scanner uses one or more of time-coded structured light, time of flight scanning, white light scanning, light projector imaging, laser vision, laser displacement scanning, active stereo random pattern projector (RPP) scanning, laser triangulation 3D scanning , laser 3D scanning, infrared scanning, structured light 3D scanning, photogrammetry, video photogrammetry, stereo photogrammetry, audio or ultrasound scanning, contact-based 3D scanning, and laser pulse-based 3D scanning.

[0026] In another embodiment, creating an image overlay comprises aligning the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard comprises using a matching algorithm.

[0027] In another embodiment, matching the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard using a matching algorithm.

[0028] In another embodiment, creating an image overlay datafile comprises aligning the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard and displaying on a display one or more of edge distribution, texture, pattern, absolute measurement of deviation, and shape.

[0029] In another aspect there is provided a vehicle repair system comprising: a visualization system comprising one or more 3D scanners for capturing a 3D image of a damaged vehicle or part thereof; a processing system connected to the visualization system for receiving the 3D image and creating a 3D datafile of the damaged vehicle; a vehicle and parts database comprising a plurality of 3D datafiles of original equipment manufacturing (OEM) vehicles and parts sorted by make and model; a graphical user interface connected to the processing system for receiving and displaying the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard, wherein during repair the visualization system dynamically updates a location of a replacement part relative to the damaged vehicle in a repair area and provides an indication of the relative location of the replacement part to the damaged vehicle.

[0030] In another aspect there is provided a vehicle repair system comprising: a visualization system comprising one or more 3D scanners for capturing a 3D image of a damaged vehicle or part thereof; a processing system connected to the visualization system for receiving the 3D image and creating a 3D datafile of the damaged vehicle; and a vehicle and parts database comprising a plurality of 3D datafiles of original equipment manufacturing (OEM) vehicles and parts sorted by make and model; wherein during repair the visualization system dynamically updates a location of a replacement part relative to the damaged vehicle in a repair area and provides an indication of the relative location of the replacement part to the damaged vehicle.

[0031] In another embodiment, the system further comprises a graphical user interface connected to the processing system for receiving and displaying the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard.

[0032] In another embodiment, the graphical user interface is displayed on a 2D display or 3D display. In another embodiment, the system further comprises an augmented reality display device for displaying at least part of the graphical user interface.

[0033] In another embodiment, the processing system can create a mapping of a virtual replacement by aligning, with the visualization system, the location of the replacement part with a location of the physical replacement part on the damaged vehicle.

[0034] In another embodiment, the processing system can eliminate objects physically blocking the repair area and the graphical user interface can show the repair area and virtual replacementpart without the eliminated object based on alignment and measurement of the replacement part and repair area.

[0035] In an embodiment the system further comprises a 3D printer capable of receiving the 3D datafile of the damaged vehicle and printing a suitable part for replacement.

[0036] In another embodiment, the processing system can create a virtual replacement part in the graphical user interface by aligning, with the visualization system, the location of the replacement part with a location of the physical replacement part on the damaged vehicle.

[0037] In another embodiment, the processing system can eliminate objects physically blocking the repair area and the graphical user interface can show the repair area and virtual replacement part without the eliminated object based on alignment and measurement of the replacement part and repair area.

[0038] In another embodiment, the visualization system comprises one or more 3D scanner which use one or more of time-coded structured light, time of flight, laser vision, laser displacement, active stereo random pattern projector (RPP), laser triangulation 3D scanning , laser 3D scanning, structured light 3D scanning, photogrammetry, contact-based 3D scanning, and laser pulse-based 3D scanning.

[0039] In another embodiment, the graphical user interface is displayed on a 2D display or 3D display.

[0040] In another embodiment, the system further comprises an augmented reality display device for displaying at least part of the graphical user interface. In another embodiment the images processed by the user interface can be seen through augmented reality devices such as, but not limited to augmented reality glasses.BRIEF DESCRIPTION OF THE FIGURES

[0041] For a better understanding of the present invention, as well as other aspects and further features thereof, reference is made to the following description which is to be used in conjunction with the accompanying figures which illustrate embodiments or aspects of the invention, where:

[0042] Figure 1 is an example system diagram and embedded flow chart of an embodiment of the present system and method;

[0043] Figure 2 is flowchart illustrating the alignment and placement of a replacement part onto a vehicle;

[0044] Figure 3 illustrates an example system for capturing 3D vehicle scan data and transferring the data to a processing system;

[0045] Figure 4 illustrates an example of the visualization system comprising a 3D scanning system and a processing system;

[0046] Figure 5 illustrates an example of a virtual overlay of an OEM vehicle standard file matched to a 3D datafile of the damaged vehicle;

[0047] Figure 6 illustrates an image of a vehicle structure with OEM factory specification reference loci in a point cloud;

[0048] Figure 7 illustrates an example window from a graphical user interface showing a vehicle with a component distance measurement;

[0049] Figure 8 illustrates an example of a method to choose repair procedures using an interactive graphical user interface;

[0050] Figure 9 illustrates an example graphical representation of the interactive graphical user interface display when the replacement part is being manipulated in space;

[0051] Figure 10 illustrates an example visualization system with a wearable device and graphical user interface;

[0052] Figure 11 is an illustration of a universal jigging system holding a replacement part;

[0053] Figure 12 illustrates an image of virtual frame and frame rail moving in virtual space;

[0054] Figure 13 illustrates an image of an aligned frame and frame rail in virtual space;

[0055] Figure 14 illustrates a side view of a frame rail in virtual space at a location distant from a frame; and

[0056] Figure 15 illustrates an image of a frame rail accurately placed and affixed to the frame using the present system.

[0057] As used in the specification and claims, the singular forms “a”, “an” and “the” include plural references unless the context clearly dictates otherwise.

[0058] The term “comprise” and any of its derivatives (e.g. comprises, comprising) as used in this specification is to be taken to be inclusive of features to which it refers, and is not meant to exclude the presence of any additional features unless otherwise stated or implied. The term “comprising” as used herein will also be understood to mean that the list following is non- exhaustive and may or may not include any other additional suitable items, for example one or more further feature(s), component(s) and / or element(s) as appropriate.

[0059] As used herein, the terms “having,” “including” and “containing,” and grammatical variations thereof, are inclusive or open-ended and do not exclude additional, unrecited elements and / or method steps, and that that the list following is non-exhaustive and may or may not include any other additional suitable items, for example one or more further feature(s), component(s) and / or element(s) as appropriate. A composition, device, article, system, use, process, or method described herein as comprising certain elements and / or steps may also, in certain embodiments consist essentially of those elements and / or steps, and in other embodiments consist of those elements and / or steps and additional elements and / or steps, whether or not these embodiments are specifically referred to.

[0060] As used herein, the term “about” refers to an approximately + / - 10% variation from a given value. It is to be understood that such a variation is always included in any given value provided herein, whether or not it is specifically referred to. The recitation of ranges herein is intended to convey both the ranges and individual values falling within the ranges, to the same place value as the numerals used to denote the range, unless otherwise indicated herein.

[0061] The use of any examples or exemplary language, e.g. “such as”, “exemplary embodiment”, “illustrative embodiment" and “for example” is intended to illustrate or denote aspects, embodiments, variations, elements or features relating to the invention and not intended to limit the scope of the invention.

[0062] As used herein, the terms “connect” and “connected” refer to any direct or indirect physical association between elements or features of the present disclosure. Accordingly, these terms may be understood to denote elements or features that are partly or completely contained within one another, attached, coupled, disposed on, joined together, in communication with, operatively associated with, etc., even if there are other elements or features intervening between the elements or features described as being connected.

[0063] As used herein, the term “3D datafile” or “three dimensional datafile” refers to any set of data that describes a three dimensional shape, such as a vehicle, vehicle components, or parts thereof, or the surrounding environment. The 3D datafiles used in the present system and method can be, for example, point cloud files, mesh files, standard tessellation language (STL) files, or any other datafile that can represent an object in 3D space within a required accuracy tolerance. It is understood that mesh or geometric (STL) files can be converted back and forth with point cloud files. The terms mesh file and point cloud are used herein as two example 3D datafiles, and it is understood that other 3D datafile types may also be used. It is understood that the terms “3D datafile” or “three dimensional datafile” may also refer to representations of the data sets in virtual space as well as representing captured data describing an actual vehicle, and that references to comparisons between or alignments or matching of datafiles may refer to partial or whole representations of the datafiles, including visual representations in virtual space.

[0064] Herein is described a method and system for the repair or replacement of vehicle components. The vehicle repair system and method detects and assesses vehicle damage and guides the repair of the vehicle and fitting of replacement parts to the vehicle based on the manufacturer’s specifications. A computer visioning or visualization system connected to one or more 3D scanners identifies the area of damage by collecting 3D data from a damaged vehicle and matching it to the 3D datafile of the OEM vehicle standard. The 3D data collected by the 3D scanner(s) is used in conjunction with the visualization system that matches the imaged 3D datafile of the damaged vehicle to the physical vehicle in virtual space and also real space around the damaged vehicle allowing the technician to see and interact with the damaged vehicle and / or a virtual model of the physical vehicle on the computer system in virtual space compared to a virtual model of the OEM vehicle standard. Likewise, the technician can interact with virtual physical replacement parts and the physical vehicle in virtual space, allowing matching of the movement of the physical replacement part on the computer display during the actual repair of the damaged vehicle. This allows the user to fit and replace vehicle components with accuracy on the physical vehicle while being guided by the computer program and the virtual image files such that they see all movement and accurate placement in 3D on the computer display.

[0065] The present system receives a 3D datafile (e.g. from a 3D scan) of a damaged vehicle and matches the received 3D datafile to a 3D datafile of the OEM vehicle standard to create an overlay datafile of the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard. The captured 3D image of the damaged vehicle can be done before or after the damaged components have been removed from the physical vehicle, and damaged parts can further be scanned independently and stored as discrete 3D datafiles for archival or comparison purposes. The 3D datafile of the OEM vehicle standard is a complete file and can also include any missing parts that have been removed from the physical vehicle which are represented on the 3D captured file. Because the 3D datafile of the damaged vehicle can be received from 3D scans taken after parts are removed and the 3D datafile of the OEM vehicle standard has the parts included, there may be a difference between the two files. The difference of the two files can be assigned as the area for repair that requires the replacement or repair of a damaged part on the physical vehicle. One or more camera or 3D scanner together with the visualization system can be used to create a virtual space around the physical vehicle and to add the physical vehicle into the virtual space. The computer system can also align and match the 3D datafiles to the physical vehicle in virtual space at scale, providing confirmation of fit prior to an actual repair.

[0066] The present system can be used to align replacement parts to the damaged vehicle in virtual space using the 3D datafile as a reference. To do this, a virtual replacement part file can be loaded into the computer system at scale. The user generally uses a universal jigging system to hold a physical replacement part for placement on the damaged vehicle. As the physical replacement part is moved into the field of view of the camera or 3D scanning system, the computer system can automatically match it to the virtual replacement part file from a parts database associated with the make and model of the OEM vehicle standard and place it in virtual space. The user can then manipulate the physical part to fit onto the physical vehicle for repair. As the physical part is moved, the camera or 3D scanning system will track its movement in relation to the damaged vehicle. The user can use the display from the computer system to see where the part is in relation to the damaged vehicle, thus tracking its movement and allowing the user to align the replacement part to the exact location specified by the vehicle manufacturer. As the part is moved nearer its desired location the computer system will give the user feedback as to how close or far away the replacement part is from where it must be placed. Once thereplacement part is in place within an acceptable placement tolerance and held by the jigging system the replacement part can be secured to the damaged vehicle.

[0067] The present system can also be used to generate repair procedures for the user to follow during the repair. The repair procedures are based on steps and instructions generally created by the OEM vehicle manufacturer that must be followed to adequately and safely repair the vehicle. To access a repair procedure for the particular repair area, the user can load the 3D virtual vehicle file before the damaged vehicle has been worked on. A part or location of the vehicle can be chosen by manipulating the vehicle reference file in 3D and clicking on the desired part to retrieve all information relevant to repairing or replacing the desired part, as well any procedure that may be needed to ready the vehicle for general repairs will then be populated by the system. Alternatively, an identification of the repair area on the damaged vehicle can be identified by a matching of the 3D datafile of the OEM standard to the 3D datafile created by 3D scanning of the damaged vehicle and the processing system can provide a repair protocol for repairing the particular repair area. Currently, the manufacturer of the jigging system does not have the data to use the system until months after the new vehicle is released to the public, creating a time when new vehicle models cannot be repaired. The present system can be generated rapidly upon vehicle release to enable vehicle damage assessment, repair, and parts replacement for even the newest vehicles.

[0068] Figure 1 is an example system diagram and embedded flow chart of an embodiment of the present system and method 100. The flowchart of work flow in visualization system 4 represents the steps of creating a 3D datafile, such as a point cloud or mesh file of the damaged vehicle, and aligning it with the physical vehicle in virtual space to have it ready to be repaired and fit with a replacement part. The technician can load a 3D virtual baseline file or 3D datafile of the OEM vehicle standard from a vehicle and parts database, where the OEM vehicle standard represents the standard vehicle of the same year, make and model as the damaged vehicle into the software before beginning work on the vehicle. Preferably a list of specific parts to be selected is stored in a vehicle and parts database based on one or more of the vehicle year, make, model and is mapped to the repair area on the damaged vehicle. The 3D datafile of the damaged vehicle can then be compared to the 3D datafile of the OEM vehicle standard that is identified in the vehicle and parts database by one or more of vehicle model, vehicle year, vehicle make, andvehicle features, and the replacement part virtual file can be chosen by clicking an area of the 3D virtual vehicle standard file.

[0069] The captured damaged vehicle 3D shape is preferably represented as a point cluster or 3D point cloud or as a geometric or mesh file, forming a shape directly representative of the damaged vehicle. Preferably the scanning and visualization of the damaged vehicle is accurate to better than 5mm, and optimally accurate to better than 2mm compared to the actual damaged vehicle. The 3D datafile of the damaged vehicle can capture an area on the vehicle with the damaged components removed or still on the vehicle. The computer system can automatically recognise which virtual part file is needed to be loaded and replaced based on the differentiating areas of the 3D vehicle scan in the repair area and loaded virtual vehicle standard file, or the replacement part virtual file can be chosen from a list of relevant parts associated with the vehicle standard file.

[0070] The baseline OEM datafile, also referred to as the 3D datafile of the OEM vehicle standard, can be used as reference 3D information 110 for the user to reference steps and details of procedures to repair the vehicle. These procedures can include, but are not limited to, information on how to prepare the vehicle for repair, how to repair specific parts, how to disconnect battery packs, and any information pertinent to a repair. In a processing system 16, upon loading or retrieval of the 3D datafile of the OEM vehicle standard parts of the vehicle are displayed and one or more different areas of the vehicle can be selected to reference for repair. The 3D vehicle standard file can also optionally be edited or customized to show only which parts are needed for alignment before it is aligned to the captured 3D file of the damaged vehicle. The technician or automated selection by the system based on the repair area can choose any part or area for repair or replacement from the 3D datafile of the OEM vehicle standard 112 whereupon the computer processing system will generate a repair procedure related to the repair of that chosen part or specific area 114. The user can then begin work on the damaged vehicle. The user may also decide to scan the vehicle with a 3D scanner with visualization system 4 before the parts for repair are removed for archival reference. A 3D image of the damaged vehicle can then be captured with one or more 3D scanner 120 by visualization system 4. The desired parts for repair or replacement may be removed before or after scanning, or multiple 3D scans of the damaged vehicle may be done both before and after removal of the damaged parts. The OEM baseline or 3D datafile of the OEM standard vehicle can then be used as a referencefile to create a overlay image and datafile of the repair area. The overlay images will align the damaged areas of the captured 3D file to the OEM standard file and can show all damaged areas as a different color or texture. One or more 3D data files of the damaged area and / or damaged vehicle as captured by the 3D scanner(s) will then be uploaded 122 into the visualization system 4 which is functionally connected to the processing system 16. The processing system will then align the captured 3D datafile with the 3D OEM vehicle standard file to create an overlay datafile 124.

[0071] The present system provides for merging and aligning two 3D vehicle files to showcase a repair area where a vehicle component is needed for replacement. The user can thereby scan the damaged vehicle with the damaged components for replacement removed and the replacement parts identified. The technician can then match the 3D scan of the damaged vehicle to the vehicle standard file in the computer system. The chosen OEM standard undamaged file can show the parts that have been removed from the physical vehicle as represented on the 3D datafile of the damaged vehicle. A 3D datafile of the damaged vehicle will represent the physical damaged vehicle and the areas where the physical replacement parts are needed. The overlay of the 3D scan of the damaged vehicle and the virtual vehicle standard file will represent the parts needed for replacement or reassembly.

[0072] As the vehicle scan and the 3D datafile of the OEM vehicle standard are merged and / or aligned together to capture the 3D shape and the 3D datafile of the OEM vehicle standard, the difference between the two datafiles can be displayed on a graphical user interface identifying at least one of, for example, curvature, smoothness, and geometrical edge similarities of the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard. When using mesh files or point clouds, the captured 3D vehicle shape can be represented as point or plane cluster representation, providing a 3D datafile shape representative of the damaged vehicle. Mesh and point cloud files can be obtained using accurate 3D scanners that are accurate to about at least 5mm, about at least 2 mm, and preferably about at least 0.5 mm of the actual vehicle measurements. The matching areas of the mesh or point cloud overlay can be shown as, for example, differences in color, texture, or line characteristics, and the differentiating areas can be shown as areas of a different color, texture, or line characteristic optionally with or outlined area that can be translucent. The differentiating areas of the 3D datafile of the damaged vehicle can represent the vehicle components that are needed for replacement and alignment. A camera or3D scanning system can capture exact dimensions of the physical damaged vehicle and create a virtual space to place the captured data from the physical vehicle into. This virtual space can be used to create an environment to place the 3D datafile of the damaged vehicle into the exact coordinates and location of the physical vehicle and represent the coordinates in the computer system to the scale of the physical vehicle.

[0073] During the repair the 3D scanning system is placed in an area near the vehicle that can capture the physical vehicle 126 in the area of damage and the required replacement part in 3D space. Once the damaged vehicle has been visualized, the 3D datafile obtained can be placed into virtual space in the computer processing system 128. The computer visualization system 4 will then align the 3D datafile of the damaged vehicle with the copy of the damaged vehicle in virtual space 130 to a copy of the undamaged vehicle standard in virtual space. The computer visualization system can also capture the real-world environment around the physical vehicle as well as the physical vehicle and place it into virtual space and scale the images to reflect the actual size of the physical vehicle thus creating an exact virtual digital copy of the physical vehicle. The virtual digital copy of the replacement part can then be loaded into the virtual space 132 created by the camera system and computer system. A virtual replacement part can be loaded into the virtual space in the computer system to scale of the actual part with the virtual 3D datafile of the damaged vehicle. To do this, the visualization system 4 creates a virtual space to scale of the actual space that the damaged vehicle occupies together with objects in real space. The computer program in conjunction with the visualization system 4 can automatically recognize and match the physical replacement part to the virtual replacement part by recognizing and matching the physical traits of the real part to the physical traits of the virtual part by recognizing similar curvature, smoothness, geometrical edge similarities, dimensions, distances between planes and / or points, and texture clusters. The placement of the physical part and physical damaged vehicle are represented to scale in virtual space on the user interface display when the camera system is placed near it. The technician can adjust the location of the physical part near the desired area of the physical vehicle and track the movement to precise measurements on the computer display. The technician can also track the part to the differentiating area of the overlay datafile that represents the area for replacement on a graphical user interface.

[0074] As the part gets nearer the desired area, the computer system can relay an indication that the replacement part is out of range, within range, or in alignment on the damaged vehicle. This can be done by changing an indication of proximity of the replacement part to its correctly aligned position on the actual damaged vehicle, and how close the replacement part is to the OEM specified position. For example, a color, shading, texture, or silhouette on the user interface on the virtual image of the damaged vehicle can change based on a scale of measurement as the part is nearer or in its desired location. When the part is moved to the exact location specified by the vehicle manufacturer a feedback in the form of, but not limited to, the color green could appear on the computer display indicating a good part alignment.Customizable jigs can be used to hold the part in place as the user reorients the part to its position. As the user chooses the virtual replacement file the system will automatically populate an OEM repair procedure for part needed to be replaced. At this point the user can save their progress 140 with parameters related to the repair.

[0075] Figure 2 is flowchart illustrating the alignment and placement of a replacement part onto a vehicle. To align a replacement part on a vehicle the technician would place the physical replacement part into the field of view of the 3D scanning system 202 wherein the computer system automatically matches the physical part to the virtual part in virtual space 204. The technician then assembles a jig system, such as a universal jigging system in a way that is suitable to hold the desired replacement part 206. The jigging system would then allow the user to move the part while attached to the jigging system and to lock it in place at any point. The physical replacement part can then be moved and tracked in virtual space 208 where the technician can see the movement of the physical part reflected on the display or graphical user interface of the computer system. The movement of the physical part in relation to the physical vehicle is replicated to scale on the computer system to the virtual part and virtual 3D datafile of the damaged vehicle in virtual space. As the replacement part is placed onto the vehicle and moved to its desired location 210 the computer system will give the user feedback when the part reached certain parameters 212. The feedback to the user from the computer system can be, but is not limited to, color change based on measured parameters on an x,y,z axis wherein the color could change based on the how close the part is to a certain parameter, haptic feedback that can be, but is not limited to, the form of a sound or vibration, or other visual or audio cues. When thepart is in its desired location the technician can then fasten the replacement part to the vehicle 214.

[0076] Figure 3 is an example system for capturing 3D vehicle scan data and transferring the data to a processor. Assessing damage to a vehicle is done by scanning the vehicle with a three dimensional (3D) scanner 18, such as an optical scanner, to generate a damaged vehicle point cloud image of the vehicle with the damaged parts removed. The 3D scanner 18 connected to the processing system 16 is configured to scan a damaged vehicle and create a 3D datafile of the damaged vehicle 30 such as a point cloud cluster or mesh file of like shapes of the damaged vehicle and its components in accordance to the described embodiments to create the 3D datafile of the damaged vehicle 30, which in this case is a point cloud. The damaged vehicle includes a repair area 52 that can be identified in the image overlay of the damaged vehicle 30 where physical parts, such as damaged parts, have been removed. The 3D scanner 18 may implement various methods for gathering data including, but not limited to, one or more of laser scanning, white light scanning, light projector imaging, camera photography, video photography, contactbased 3D scanning, time-coded structured light scanning, stable light source scanning, time of flight scanning, active stereo random pattern projector (RPP) scanning, infrared scanning, audio or ultrasound scanning, time-of-flight 3D laser scanning, laser displacement scanning, triangulation based 3D laser scanning, structured-light 3D scanning, modulated light 3D scanning, camera photogrammetry, video photogrammetry, stereo photogrammetry, light projector imaging, and infrared scanning technology. Laser scanning generally captures three- dimensional spatial data using lasers in the form of a point cloud comprised of a set of millions of three-dimensional coordinates (x,y,z) in a 3D space.

[0077] In its typical operation the 3D scanner 18 can gather information relating to the damaged vehicle by generating a replica 3D datafile such as a point cloud cluster that is accurate to greater than, for example, at least 5 mm, or at least 2 mm, and represents the precise shapes of the area of the vehicle with removed parts on an x,y,z scale in three dimensions. Once captured by the 3D scanner 18, the 3D datafile information is then transmitted to the processing system 16 to be stored and analyzed. The user can load an OEM vehicle standard file of the same make and model into the processing system 16 to create, for example, a 3D datafile of the OEM vehicle standard and 3D scanned vehicle image by recognizing areas, in this case, of the point cloud data. Matching of the 3D datafile of the damaged vehicle to the 3D datafile of the OEM standardcan be done using a matching algorithm to match, for example, geographical edge profile, line curvature, smoothness, geometrical edge similarities, dimensions, distances between planes and / or points, and texture clusters of the two files. Areas of differentiating point cloud data can appear as a different color, outline, texture, or shading, and represent parts of the vehicle in the repair area 52 that have been removed and need to be replaced, or of parts that are in need of repair and are damaged.

[0078] The processing system 16 can display this information and can also create a series of reports based on the users preferences. In some embodiments the 3D scanner 18 may be used to scan certain areas of the vehicle. For example, if the damage or removed parts to the vehicle are isolated to the front or rear of the vehicle, the user may choose to scan only those areas. The 3D scanner 18 and associated image processing system 16 can be used to collect a 3D datafile of the damaged vehicle or part thereof, optionally including undamaged portions of the damaged vehicle adjacent the location of damage. In the present system, one or more 3D scanners can be used of one or more different types to collect the desired point cloud image. The collected image can also be cleaned by the computer processing system 16 to remove any unwanted artifacts, such as, for example background, superficial aberrations, and specular reflections. The processing system 16 then compares the received image of the damaged vehicle with the corresponding reference image of the OEM vehicle from the vehicle and parts database along several attributes, e.g., edge distribution, texture, pattern, absolute measurement of deviation, and shape. Using a variety of computer vision techniques, the processing system 16 can recognize where and how the received images depart from the reference images by comparing the point clouds or 3D datafiles, and identifies the corresponding part(s) and / or regions on the damaged vehicle that need repaired or replacement parts. Corresponding areas of the damaged vehicle are then matched in the software to the OEM standard and presented as an overlay of images, for example represented as an STL file format, point cloud, or mesh, that will show few or no differences in areas that have not been affected by damage. Furthermore, areas of differentiating points on the vehicle overlay image will represent parts that are needed to be replaced. These locations in the repair area can be represented by, for example, change of color, line thickness, dot thickness, highlighting, edge distribution, texture, pattern, or other visual differentiators. The absolute measurement of the deviation of the damaged vehicle from the OEM vehicle standard can also be displayed in the repair area in the overlay image and / or the deviation distancesaccessible by mouseover or using a software measurement tool to provide the distance of deviation between a location on the OEM standard vehicle and a location on the damaged vehicle. The image overlay is created by the image processing system by matching an array of points of the 3D datafile of the damaged vehicle using image processing to the 3D datafile of the OEM vehicle standard, optionally using machine learning technologies.

[0079] The 3D datafile, for example a point cloud image or STL file, of the damaged vehicle can be stored in a memory operatively connected to, either wired or wirelessly, or in communication with the 3D scanner 18 and processing system 16. The memory is preferably in communication with the one or more processors or image processing systems to compare the 3D datafile of the damaged vehicle. The display, preferably presented in a graphical user interface, can also be configured such that the user can choose to toggle each digital image on and off. Areas for repair or replacement can be shown as differences in colour, shading, outline, or any other display method that makes clear which is the image of the OEM standard and which is the damaged vehicle. The display can provide a flat 2D vehicle map, however preferably the position of the vehicle image can be adjusted in the display to provide different views of the damaged vehicle compared to the vehicle standard. In another embodiment, the display can be a 3D display or holographic display, optionally used in conjunction with a wearable device, with options to adjust the viewing angle in 3D space so the technician can get a full 3D space representational image of the vehicle damage and repair area 52. The user can have the capability of toggling each image on and off by pressing a virtual button on the image processing system. It has been found that providing technicians with 2D and 3D visualization of the damaged area of a vehicle during a repair enables the technician to understand the repair and improves repair quality and reduces repair time.

[0080] The present system can also apply photogrammetry to assess and measure damage on a vehicle. Photogrammetry is the science and technology of obtaining reliable information about physical objects and the environment through the process of recording, measuring and interpreting photographic images and patterns of electromagnetic radiant imagery and other phenomena. In an embodiment, the 3D imaging visualization system is used to capture data from a damaged vehicle after an accident. The visualization system can use one or more of various scanning technologies to collect a 3D dataset of the damaged vehicle. In a simple embodiment, the visualization system can use a white light scanner or camera. In another embodiment, time-of-flight 3D laser scanning technology employs an active scanner that uses laser light to probe the subject. At the heart of this type of scanner is a time-of-flight laser range finder. The laser range finder finds the distance of a surface by timing the round-trip time of a pulse of light. In another embodiment, triangulation based 3D laser scanning can be used which employs active scanners that use laser light to probe the environment. With respect to time-of-flight 3D laser scanner, a triangulation laser shines a laser on the subject and exploits a camera to look for the location of the laser dot. In another embodiment, structured-light 3D scanning can be used to project a pattern of light on the subject and look at the deformation of the pattern on the subject. The pattern is projected onto the subject using either an LCD projector or other stable light source. A camera, offset slightly from the pattern projector, looks at the shape of the pattern and calculates the distance of every point in the field of view. In another embodiment, modulated light 3D scanning shines a continually changing light at the subject. Usually the light source simply cycles its amplitude in a sinusoidal pattern. A camera detects the reflected light and the amount the pattern is shifted by determines the distance the light travelled. Modulated light also allows the scanner to ignore light from sources other than a laser, so there is no interference. In another embodiment, the imaging system can use stereophotogrammetry based on a block or collection of overlapped images, and can be used for 3D mapping and object reconstruction using a plurality of overlayed and indexed 2D images.

[0081] Figure 4 is an example of the visualization system 4 in the present system connected to a 3D scanning system 40 and a processing system 16. The 3D scanning system 40 captures and creates a virtual space around the damaged vehicle as well as anything else that is placed inside or adjacent to the damaged vehicle, such as, but not limited to, the replacement part 6, and creates a 3D datafile of the damaged vehicle 30. The 3D scanning system 40 creates a virtual space by generating 3D feature points that are then dictated to the computer. A scale 48 in the form of a ruler or any other image or structure that can be read and analysed by the computer system can be used to create the scale of the vehicle in virtual space by placing this scale 48 on or near the physical vehicle. This scale is used by the processing system to calculate the scale of the vehicle by associating the physical traits of the vehicle to the size of the physical scale. The processing system recognizes the physical scale and its details and calculates the volume of the physical objects in virtual space by referencing the physical scale to the physical traits of the physical vehicle. The visual scale can comprise, for example, one or more rulers, images withfeatures of known dimensions, anchor points, illuminated markers, or physical objects with known dimensions that the visualization system can use to normalize the captured image against known measurements. The 3D scanning system 40 can then use stereo images to interpolate the camera position, vehicle dimensions and location, using the location of the scale 48 or normalization marker(s). The 3D scanning system 40 can also use infrared trackers as the scale 48, which can serve as reflective markers to track and place the vehicle in place. The 3D scanning system 40 can further use stereo vision methods and one or more pairs of like or different cameras mounted in accordance with their baseline distance and with their optical axes aligned. An algorithm is then used to calculate relevant information related to the camera size, and scale location, scale spacing, and scale orientation and to create the virtual space and 3D datafile of the damaged vehicle 30.

[0082] The normalization of the replacement part in virtual space is necessary for the repair process of the vehicle in the repair area 52 as the technician’s hands, or other vehicle components could be blocking a partial view of the physical replacement part. In this case, the full view of the physical replacement part is not necessary for the computer’s system to show on the display all aspects of the virtual part and to show the scale of the replacement part. Various methods and photogrammetry camera systems for capturing and creating virtual space can be used, including but not limited to time-coded structured light, time of flight, structured light, laser vision, laser displacement, and active stereo random pattern projector (RPP) methods, which can optionally use light projection technology followed by image capturing. Other Laser scanning technologies may also be used by the 3D scanner, for example Laser triangulation 3D scanning, Laser 3D scanning, structured light 3D scanning, photogrammetry, contact-based 3D scanning, Laser Phase-shift 3D scanning, and Laser pulse-based 3D scanning. The 3D scanning system 40 can be placed anywhere near the damaged vehicle as long as the field of view of the 3D scanning system 40 can capture the relevant working area of the vehicle and as long as it can capture relevant distinctive areas of geometry or markers to capture the image in virtual space. It is understood that the 3D scanning system 40 comprises one or more 3D scanners or cameras capable of capturing the 3D space around the vehicle, and that multiple cameras can be used, or one or more movable cameras, or both. The 3D scanning system 40 relays the relevant information to the computer processing system 16 for application of algorithms to match the captured damaged vehicle 8 to create the 3D datafile of the damaged vehicle. A replacement part6 can then be placed into the field of view of the 3D scanning system 40 and the 3D scanning system 40 will record its shape and location and relay it to the computer processing system 16, wherein software in the computer processing system 16 will place the replacement part 6 into virtual space relative to the 3D datafile of the damaged vehicle 30.

[0083] Figure 5 illustrates an example of a virtual image overlay file for an OEM vehicle standard file matched to a 3D datafile of a damaged vehicle 30. The area of difference 32 is an area of the vehicle where a part is needed to be repaired or replaced and where the vehicle has been damaged. The computer visualization system receives a 3D datafile of a damaged vehicle 30 and matches it to a 3D datafile of an OEM vehicle standard 22 for a vehicle of the same make and model. The 3D datafile of the OEM vehicle standard 22 will be a representation of an area of the vehicle that has the desired parts for replacement assembled and will be at least partially completed and will represent exact measurements set out by the vehicle manufacturer per the OEM vehicle standard. The computer visualization system then matches the 3D datafile of a damaged vehicle 30 to the 3D datafile of the OEM vehicle standard 22 and creates a visually or otherwise identifiable area of difference 32 that represents the desired area of repair by creating an overlay datafile 34 of the 3D datafile of the OEM vehicle standard 22 and the 3D datafile of the damaged vehicle 30 and identifying points of deviation from a threshold. The areas where deviation from the threshold or desired tolerance distance between the OEM standard and the damaged vehicle are then identified as areas of the damaged vehicle requiring repair. The tolerance distance between the OEM standard and the damaged vehicle can be generally set based on the deviation at which point an aesthetic or noticeable change is visible in the damaged vehicle. In locations where the damage is structural or functional, the tolerance distance may be lower as any significant deviation from the OEM standard can result in detrimental effects on the functioning or integrity of the vehicle. The absolute distance between the OEM standard and the damaged vehicle can also determine whether a part needs to be fully replaced, or whether a repair is possible, which is also dependent on the material and vehicle component at the location of the repair area. The identified area of difference 32 can then be used to align the replacement part 6 on the damaged vehicle. During visualization of the replacement part, the overlay datafile 34 can also be further overlaid with the replacement part 6 during placement to ensure that the replacement part 6 is properly aligned prior to securing the part to the vehicle undergoing repair. Additional scans may be taken with the 3D scanner once the replacement part has been placed inits desired location to further check the location of the replacement part is accurate to the vehicle manufacturers standards.

[0084] Various methods for matching, aligning, and calculating distances between anchor points in point clouds and mesh or geometric files can be used in the present method and system. In one example of a matching algorithm, an intrinsic shape signature algorithm can be used to detect key shape feature points using a weighted three-dimensional occupational histogram of the data points within the angular space, which is a view-independent representation of the three- dimensional shape. Then, the point feature histogram can be used to represent the underlying surface model properties at a point whose computation is based on the combination of certain geometrical relations between the point’s nearest k-neighbors. The two-view point clouds can then be robustly matched using a double neighborhood constraint of minimizing the sum of the Euclidean distances between the local neighbors of the point and feature point. This matching method has been found to be resistant to noise, reduces the search range for matching points, and improves the correct feature point matching rate for a weak surface texture. In another example of a matching algorithm, an iterative closest point (ICP) algorithm can be uses to match points in the point cloud surfaces. The optimal transformation matrix of a two-view point cloud can be acquired through continuously reducing the Euclidean distance between the point clouds until the best match between the two views is achieved. Three dimensional (3D) point cloud key point matching algorithms can also use the Nearest Neighbor (NN) and Nearest Neighbor Distance Ratio (NNDR). In these algorithms a key point p from the origin point cloud is selected. Key points in the target point cloud pl, p2, and p3 are detected as the most three similar key points through comparing their similarity of descriptors, and they are sorted according to the degree of similarity. NN directly regards the most similar points pl and p as a corresponding point pair. Depending on the ratio of similarity, NNDR chooses pl or p2 as the corresponding point to p, and preferably only considers the top-two similar key points. In yet another example, a Random Sampling Consensus Algorithm (RANSAC) can be used for eliminating wrong corresponding point pairs. RANSAC chooses three groups of corresponding point pairs for registration and then calculates the distance of the rest of the corresponding point pairs after registration to evaluate the accuracy of the three groups of corresponding point pairs to find the most accurate three groups of corresponding point pairs by iterations. The accuracy of correspondence between the two point clouds can then be improved, specifically by distinguishing inliers from outliers basedon a “hypothesis generation and verification” mechanism. To do this, three or more correspondences are first randomly sampled from the initial correspondence set, a hypothesis is then generated with these samples and candidates agreeing the hypothesis are judged as inliers at this iteration, the above processes are repeated, and the inlier set with the maximum cardinality is served as the final result.

[0085] In addition to directing part replacement, the present method and system can also be used to repair damaged parts and align them back to OEM standards. One or more tools capable of applying force and torque can be used to realign or reshape a component, such as, for example, one or more of a pulling tower, hydraulic piston, hammer, or pliers. These tools can be used by hand or can be fastened by a chain, cable, belt, or other fastening device to the damaged part on the physical vehicle to push or pull the damaged part back into the desired shape or location. During a repair procedure, the 3D scan of the damaged vehicle and vehicle standard can show a damaged part or area by aligning the two files and aligning a percentage of similar points that differentiate at a minimal tolerance and showing these points as a single area of the damaged vehicle that is of OEM specification and undamaged. Points within the point cloud that differentiate more than the specified tolerance can be shown as a damaged area of the damaged vehicle. The computer system can recognise and visually identify the damaged area by associating a color or pattern to the points in, for example, the point cloud of the damaged vehicle that differentiate from the vehicle standard file when overlayed with the point cloud of the OEM vehicle standard. Similar alignment and visual identification can be done with mesh files or STL files. The aligned 3D datafiles can show the similar parts of the damaged vehicle scan and vehicle standard in different orientation by showing the vehicle standard portion of the overlay datafile with a color or pattern and the damaged vehicle parts as another color or pattern. The computer display can then be displayed to a technician to illustrate and point out the differentiating areas of the overlay datafile where the vehicle is damaged and the areas of the overlay datafile that are undamaged. Any image display technology may be used to display the overlay image, including but not limited to a 2D display on a graphical user interface, a 3D stereographic display or holographic display, optionally used in conjunction with a wearable device.

[0086] The technician can then assemble the pulling tower and / or tools to the damaged vehicle and begin reorientation of the damaged part to the desired location. As the technician moves thedamaged part on the damaged vehicle the visualization system will recognize the movement of the damaged part and show the movement on the computer display on the graphical user interface. As the part is moved back toward its desired location, the computer system can optionally give the technician feedback in the form of, but not limited to, a change of color, shading, or texture on the overlay image, or alternatively or in addition, audio feedback indicative of the correct part location. Haptic feedback can also be provided through a wearable device to guide placement of the replacement part in the repair area. The computer display can also show differentiating points of the damaged vehicle point cloud portion of the overlay datafile moving in relation to the differentiating points of the 3D datafile of the OEM standard of the overlay datafile. When the damaged part is reorientated back to the OEM specified position the computer system can also give the user feedback and the vehicle part can be deemed repaired.

[0087] Figure 6 illustrates an image of a vehicle structure with factory specification reference loci in a point cloud. The image of the undamaged vehicle is an image or source 3D composite image file representative of an undamaged vehicle (the 3D datafile of the damaged vehicle). Measurement points also referred to as reference loci show x,y,z coordinate differences in distances and angles, and reference loci can optionally be selected in advance as points of reference on the standard vehicle. The reference loci can be chosen by the user, automatically assigned in the vehicle and parts database, selected by the image processing system according to rules, or a combination thereof. The simplified point cloud shown provides the distances between reference loci in the vehicle, as well as angles between reference loci on the vehicle. In one example, the distances are provided on the right as the expected standard distances between reference loci in an OEM vehicle. The OEM repair procedures for the desired part needed to repair the vehicle can also populated as the user or computer system chooses the virtual replacement part file, and the technician can choose the repair procedure of the replacement part from the virtual 3D file before work has been started on the vehicle by clicking on the desired location of the 3D file related to the area for repair. The technician can also select points on the vehicle and / or replacement part to show measurements of accuracy on the computer display to match with the OEM measurements, where the points of measurement can be chosen from any point on the vehicle and / or replacement part.

[0088] Figure 7 illustrates an example window from a graphical user interface 14 showing a vehicle with a manual measurement 28 that can be taken of various areas of the 3D datafile of the damaged vehicle when the replacement part 6 has been fitted and fastened. Upon confirming that the vehicle has been repaired the user can manipulate various functions of the software to create measuring points for reporting to vehicle manufacturers, insurance providers or customers. These manual measurements can be referenced to the reference loci on the 3D mesh file or point cloud file of the vehicle to ensure the vehicle has been repaired adequately and is within OEM specifications for standards and safety for the vehicle. Manual measurements may be assisted by suggestions or computations provided by the processing system. The graphical user interface display can be, for example a screen that can be carried by the user such as, but not limited to a laptop, smartphone, or tablet, and / or can comprise a wearable device such as augmented reality glasses or headset. Preferably the processor allows the vehicle shown on the user interface to be manipulated in 360 degrees and zoomed in and out in virtual space on the graphical user interface.

[0089] Figure 8 illustrates an example of a method to choose repair procedures by using an interactive graphical user interface 14 and pointer to choose an area of the graphical vehicle standard OEM file. Upon selecting the desired area of the 3D datafile of the OEM vehicle standard 22 the chosen part for replacement 24 can become highlighted or otherwise identified as a means of confirmation. Upon confirmation the user can choose to print, save or interact with a list or parts and / or tasks for the OEM repair procedure related to the chosen part for replacement 24. This chosen part for replacement 24 will be related to the repair and in may cases be a specific part for replacement and as a result the one of the areas of difference that will be shown when the 3D vehicle scan file is compared to the vehicle standard file.

[0090] Figure 9 illustrates an example graphical representation of indications provided by the computer processing system on the interactive graphical user interface 14 display when the replacement part is being manipulated in space. As the computer system tracks the location of the replacement part in space in relation to the damaged vehicle 8 using the visualization system, the computer system will display feedback to the technician pertaining to how close the replacement part 6a, 6b, 6c is to the OEM required position for repair. Feedback from the computer system for when the replacement part is placed in or near its desired location can be measurements in the form of, for example, a distance on an x,y,z plot. The feedback can be in theform of, for example, audio feedback, haptic feedback, on an accessory system in the form of haptic feedback gloves or other wearable device, or graphical feedback in the form of change in color, texture, pattern, as herein described. As shown in panel A, when the replacement part 6a is at a far distance the replacement part 6a can be indicated as distant from the vehicle, and can be indicated using one or more color, pattern, or luminosity range, or a combination thereof. As shown in panel B, when the replacement part 6b is closer to its required position near the vehicle the display on the graphical user interface 14 can show a different indication, in this case a change in pattern from hashes to dots, when the vehicle part is within a closer range to its desired location. When the replacement part 6c is at its desired location as shown in panel C the display can show another indication that the part is correctly positioned and ready for attachment. These indications could be in the form of, but not limited to, a heatmap, color or pattern, and can be shown on a legend in the computer display on the graphical user interface 14. During the repair, customizable jigs can be used to fasten to the vehicle part by bolt or nut patterns that can fasten to the part by existing threads on the part, and can also be used to fasten to the vehicle part clamping mechanisms.

[0091] Figure 10 is an example visualization system having a wearable device 20 that can be used as an accessory device to visualize the computer screen or graphical user interface (GUI) 14 and can be interactive. The wearable device 20 can be placed into the field of view of the camera system and be tracked in virtual space. The processing system 16 can place the wearable device 20 into the virtual space such that the wearable device 20 can act as virtual camera, able to view macro spaces within the virtual space and able to zoom in to areas of the virtual space. The wearable device 20 can track the wearer’s location in virtual space allowing the wearer to see the virtual space relative to their location in it including, but not limited to, the distance and angle of reference to any objects in the virtual space. The wearable device 20 can also engage in haptic feedback based on information from the computer processing system 16 pertaining to relevant information when fitting the replacement parts and their position and orientation, both absolute and relative to other features or objects, in real and / or virtual space.

[0092] Figure 11 is an example of a universal jig 10 that can be used to assist a technician in holding a replacement part 6 in place while the user is fitting the replacement part 6 to a damaged vehicle 8. The physical part can be manipulated and moved while being attached to the jig system. In one embodiment the part can be moved by the user and will stop and stay in anexact location as the user ceases to apply pressure to the part. The jig system may have multiple accessory components such as, but not limited to, adjustable vice grips, screws, bolts, and C- clamp jaws to allow it to attach itself to the physical part and hold it in place. The jig system may also be on stands or hang from the ceiling or attached to a frame rack or to a post lift.

[0093] Figure 12 illustrates an image of virtual frame and frame rail positioned in virtual space. When a 3D scan is taken of the physical vehicle, an image of the virtual vehicle is overlayed onto the computer vehicle on the display screen to match the virtual vehicle and the physical vehicle. A 3D scan is first be taken of the actual vehicle without the part on it. This 3D scan is uploaded into the computer system. The visioning system in front of the actual vehicle can then take dynamically updating images using dynamic tracking of the actual part relative to the actual vehicle. The system then recreates a virtual image on the display screen of the actual vehicle such that the part can be aligned in virtual space to guide the positioning of the replacement on the actual vehicle. Here the frame rail is shown on the display screen in the virtual display aligned in virtual space next to the frame onto which it is intended to be attached. As the physical frame rail is moved in actual space closer to the actual frame where the frame rail is to be attached, the orientation and location of the frame rail relative to the frame can be represented on the display screen, which can be then used as a guide for part alignment. In one embodiment a color on the display screen will indicate the distance, alignment, and orientation of the frame rail relative to the frame, where greater than about 10cm will be red and between about 0.5-10cm but not within an acceptable alignment will be yellow. Once the part is accurately placed the display screen can show green for accurate alignment. It is understood that accurate placement and alignment must be within about 2mm to satisfy OEM standards. Instead of a graphical user interface, the distance and alignment of the replacement part can also be indicated to a technician using, for example, auditory cues, haptic feedback, and visual cues such as changes in lighting.

[0094] Figure 13 illustrates an image of a frame rail in virtual space accurately placed and affixed to the frame using the present system.

[0095] Figure 14 illustrates a side view virtual image of a frame rail in virtual space at a location distant from a frame onto which it is intended to be placed and affixed.

[0096] Figure 15 illustrates an image of a frame rail accurately placed and affixed to the frame using the present system. The present system can also recommend a particular jigging system arrangement and / or particular jig for placement of the particular part on the vehicle.

[0097] The use of the terms user and technician are understood to be interchangeable throughout this specification. Further, anywhere an aspect or embodiment describes a task that can be performed by a user, it is also understood that such a task or portion thereof could be automated such that it is assisted partially or wholly by mechanical devices.

[0098] All publications, patents and patent applications mentioned in this specification are indicative of the level of skill of those skilled in the art to which this invention pertains and are herein incorporated by reference. The reference to any prior art in this specification is not, and should not be taken as, an acknowledgement or any form of suggestion that such prior art forms part of the common general knowledge.

[0099] The invention being thus described, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the scope of the invention, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.

Claims

WE CLAIM1. A computer implemented method comprising: scanning a damaged vehicle with a three dimensional (3D) scanner to obtain a 3D datafile of the damaged vehicle; matching the 3D datafile of the damaged vehicle to a 3D datafile of an original equipment manufacturing (OEM) vehicle standard by matching the make and model of the damaged vehicle, the 3D datafile of the OEM vehicle standard associated with a plurality of vehicle parts for the OEM vehicle standard; creating an image overlay of the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard; identifying an area of damage on the damaged vehicle by comparing the 3D datafile for the OEM vehicle standard to the 3D datafile of the damaged vehicle and identifying a repair area where a measurement threshold between the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard is greater than within a desired tolerance; identifying at least one replacement part in the repair area; and during a repair: imaging a location of the replacement part on the damaged vehicle; and identifying when the replacement part is within a placement tolerance on the damaged vehicle such that the replacement part can be aligned on the damaged vehicle.

2. The method of claim 1, wherein the 3D datafile is a point cloud file, mesh file, standard tessellation language (STL) file, or any other datafile that can represent an object in 3D space within a desired scanning accuracy.

3. The method of claim 2, wherein the 3D datafile of the damaged vehicle has a scanning accuracy better than 5mm.

4. The method of any one of claims 1-3, wherein the desired tolerance is less than 5mm.

5. The method of any one of claims 1-4, further comprising displaying a location of the replacement part and damaged vehicle in the image overlay on a graphical user interface.

6. The method of claim 5, wherein the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard can be manipulated in three dimensions on the graphical user interface.

7. The method of any one of claims 1-6, further comprising providing an indication when the replacement part is aligned on the vehicle.

8. The method of claim 7, wherein the indication is represented on the graphical user interface as one or more of color, pattern, and highlighting.

9. The method of claim 7, wherein the indication comprises one or more of sound, haptic feedback, and displayed distance measurements between the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard.

10. The method of any one of claims 1-9, further comprising, during the repair, dynamically tracking the replacement part relative to the damaged vehicle with the 3D scanner.

11. The method of any one of claims 1-10, wherein the 3D scanner uses one or more of time- coded structured light, time of flight scanning, white light scanning, light projector imaging, laser vision, laser displacement scanning, active stereo random pattern projector (RPP) scanning, laser triangulation 3D scanning , laser 3D scanning, infrared scanning, structured light 3D scanning, photogrammetry, video photogrammetry, stereo photogrammetry, audio or ultrasound scanning, contact-based 3D scanning, and laser pulse-based 3D scanning.

12. The method of any one of claims 1-11, wherein matching the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard comprises using a matching algorithm.

13. The method of any one of claims 1-12, wherein creating an image overlay comprises aligning the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard and displaying on a display one or more of edge distribution, texture, pattern, absolute measurement of deviation, and shape.

14. A vehicle repair system comprising: a visualization system comprising one or more 3D scanners for capturing a 3D image of a damaged vehicle or part thereof; a processing system connected to the visualization system for receiving the 3D image and creating a 3D datafile of the damaged vehicle; and a vehicle and parts database comprising a plurality of 3D datafiles of original equipment manufacturing (OEM) vehicles and parts sorted by make and model; wherein during repair the visualization system dynamically updates a location of a replacement part relative to the damaged vehicle in a repair area and provides an indication of the relative location of the replacement part to the damaged vehicle.

15. The system of claim 14, further comprising a graphical user interface connected to the processing system for receiving and displaying the 3D datafile of the damaged vehicle and the 3D datafile of the OEM vehicle standard.

16. The system of claim 15, wherein the graphical user interface is displayed on a 2D display or 3D display.

17. The system of claim 15 or 16, further comprising an augmented reality display device for displaying at least part of the graphical user interface.

18. The system of any one of claims 14-17, further comprising a 3D printer capable of receiving the 3D datafile of the damaged vehicle and printing a suitable part for replacement.

19. The system of any one of claims 14-18, wherein the processing system can create a mapping of a virtual replacement by aligning, with the visualization system, the location of the replacement part with a location of the physical replacement part on the damaged vehicle.

20. The system of any one of claims 14-19, wherein the processing system can eliminate objects physically blocking the repair area and the graphical user interface can show the repair area and virtual replacement part without the eliminated object based on alignment and measurement of the replacement part and repair area.

21. The system of any one of claims 14-20, wherein the visualization system comprises one or more 3D scanner which use one or more of time-coded structured light, time of flight, laser vision, laser displacement, active stereo random pattern projector (RPP), laser triangulation 3D scanning , laser 3D scanning, structured light 3D scanning, photogrammetry, contact-based 3D scanning, and laser pulse-based 3D scanning.

Images