Vehicle Inspection System

Abstract

A computer-implemented method of vehicle inspection includes processing one or more images of a vehicle (V) with a defect detection algorithm to identify defects on the vehicle; and generating an identifier (18) on a display of an immersive display device (14) when the vehicle and the defect are within a field of view of the immersive display device, the identifier being displayed adjacent to the defect (16).

Description

[Technical Field]

[0001] The present invention relates to the field of vehicle inspection systems. [Background technology]

[0002] Various situations arise where humans inspect vehicles for defects such as scratches, scuffs, chips, dents, dings, warping, gaps, flatness, etc. This inspection may also include reviewing the specifications of a built vehicle against customer required specifications to identify deviations therebetween, such as during quality control stages during vehicle manufacturing, during the vehicle shipping process, and during rental vehicle pick-up and return.

[0003] If a defect or deviation needs to be identified and graded, or if it falls outside acceptable standards, it is repaired. This involves pinpointing the defect, measuring its size or impact, and then communicating clear instructions to the team responsible for resolving the problem.

[0004] It is known to provide camera-based imaging systems for acquiring images of a vehicle being inspected. In such systems, images of the vehicle are acquired and a defect detection algorithm can identify suspected defects based on the images. A report can be generated showing the vehicle and the suspected defects to be used as a guide for an inspector to check the vehicle.

[0005] However, defects are potentially small in size (e.g., less than 1 mm) and uniformly coated on large surfaces (typically 30 million mm 2Due to the difficulty of locating defects on painted surfaces (greater than 100 mm), locating a defect can be very time-consuming, even if its location on the vehicle is indicated. For example, on a production line, an inspector may identify the defect's location and mark it in the plant quality control system, after which a repair team must repair it by showing its approximate location on a butterfly diagram. With an average time at each station of 60 to 80 seconds, even a short delay in locating the defect can significantly impact the ability to repair the problem without stopping the line.

[0006] The present inventors have therefore recognised that known vehicle inspection systems can be improved. Summary of the Invention

[0007] According to a first aspect of the present invention, there is provided a computer-implemented method of vehicle inspection comprising: processing one or more images of the vehicle with a defect detection algorithm to identify defects on the vehicle; and generating an identifier on a display of an immersive display device when the vehicle and the defect are within a field of view of the immersive display device, the identifier being displayed adjacent to the defect.

[0008] Thus, the vehicle inspection method according to the first aspect of the present invention enhances a user's view of the vehicle with an identifier displayed on the screen of an immersive display device. The identifier is presented adjacent to the defect and can guide a user of the immersive display device to the location of the defect or deviation on the vehicle. This method can therefore increase the speed and accuracy of the vehicle inspection and repair process.

[0009] The identifier is shaped and positioned so as not to obscure the defect, and thus the identifier may be shaped to allow a user to see the defect on the screen when the identifier is present on the screen.

[0010] The identifier has a boundary shape surrounding the defect, the interior of the boundary shape being transparent within which the defect is visible to the user. The boundary shape can be sized such that its width and height or diameter are larger than the corresponding dimensions of the defect on the screen, so that the bounding box of the boundary shape closely matches the size of the defect without interfering with the user's visibility of the defect.

[0011] The identifier appears to be static relative to the defect, so that as the user moves relative to the vehicle, the identifier remains adjacent to the defect.

[0012] The defect detection algorithm may be configured to classify the defects by a defect type label, and the method may further include displaying the defect type label on the screen. Thus, the method may intuitively inform a user about the type of defect that the defect detection algorithm has identified as being present on the vehicle. The label may be displayed on the screen in a position that does not obscure the defect and may be positioned adjacent to an identifier so that a user can intuitively associate the label with the defect.

[0013] The method may further include providing the user with user-selectable feedback inputs to confirm whether the defect type labels are correct. The inputs may include, for example, "correct" and "incorrect" buttons, and the user may select a hand gesture method, such as, in the case of a phone or tablet, pressing a portion of the screen corresponding to one of the buttons, or, in the case of a headset, moving their hand in free space as detected by a hand tracking system.

[0014] The method may further include providing positive or negative reinforcement for training the defect detection algorithm in response to the user's selection of the feedback input. This may include confirming or rejecting a defect, selecting a correct label, or adding a defect not detected by the defect detection algorithm. Once a defect is identified and confirmed, any vehicle images in which the defect appears may be labeled by creating a bounding box in each frame and passed to a learning algorithm. Additionally, if the immersive display device incorporates a camera, additional enlarged images may be stored before or after repair for later evaluation or use to train the defect detection algorithm.

[0015] The method may include rating the defect by size or severity in response to the user's selection of the feedback input. A defect detection algorithm may estimate size based on a localization algorithm and its bounding box, or by comparing the defect to defects graded in previous ratings. This allows a user to quickly confirm, change, or reject the rating while simultaneously viewing the defect and a user interface.

[0016] Processing the image may include identifying a plurality of defects, and generating an identifier may include generating a corresponding identifier for each identified defect. In some embodiments, multiple identifiers may be displayed simultaneously for each defect displayed on the screen until a user selects an input to confirm the defect type is correct, or may be generated one at a time for a user to evaluate each defect in turn.

[0017] The immersive display device is an augmented reality (AR), mixed reality (MR), or virtual reality (VR) display device.

[0018] The method may include identifying the vehicle using the immersive display device, for example based on a QR code or vehicle identification number (VIN) on the vehicle, which may be used to display the correct identifier for the vehicle.

[0019] According to a second aspect of the present invention, there is provided a vehicle inspection system comprising: a controller that receives data identifying a defect on a vehicle and displays an identifier on a screen of an immersive display device when the vehicle and the defect are within a field of view of the immersive display device, the identifier being displayed adjacent to the defect.

[0020] According to a third aspect of the present invention, there is provided a vehicle imaging station for acquiring one or more images of a vehicle; a controller that processes the images using a defect detection algorithm to identify defects on the vehicle; an immersive display device that generates an identifier on a display of the immersive display device when the vehicle and the defect are within a field of view of the immersive display device, and displays the identifier on the screen adjacent to the defect; A vehicle inspection system is provided, comprising:

[0021] Any feature of the first aspect may be equally applicable to the second and third aspects.

[0022] According to a fourth aspect of the present invention, there is provided a production system comprising: two or more vehicle imaging stations for acquiring one or more images of a vehicle at different stages of a production process; a controller that processes the images using a defect detection algorithm to identify defects on the vehicle; a display device and interface for identifying and evaluating said defects; an immersive display that displays the state of the vehicle at different stages of production; A vehicle inspection system is provided, comprising:

[0023] According to a fourth aspect of the invention, the immersive display device may be configured to allow an operator to assess damage to a vehicle, assess whether the vehicle has been correctly repaired, or compare whether damage is present at different stages of the production process.

[0024] The method may include comparing defects on multiple vehicles at once, for example, displaying defects on a 3D model of the vehicle, or generating heat maps or videos showing trends over time across a batch of vehicles. By using an immersive display, operators can much more easily visualize and assess damage, allowing them to more quickly identify root causes and production issues.

[0025] When the vehicle and defect are within the field of view of the immersive display device, an identifier is generated on the display of the immersive display device, and the identifier is displayed adjacent to the defect on the screen.

[0026] In any of the above embodiments of the present invention, the term defect may be used to refer to damage, marks, or stains on the vehicle, or deviations from the vehicle's required specifications. [Brief explanation of the drawings]

[0027] Specific embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: do.

[0028] [Figure 1] FIG. 1 is a diagram of a vehicle inspection system according to an embodiment of the present invention. [Figure 2] 2 is a diagram of a display of the immersive display device of the vehicle inspection system of FIG. 1. [Figure 3] FIG. 3 is a flow chart illustrating a computer-implemented method for vehicle inspection according to one embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating the AI ​​training and deployment stages for a defect detection algorithm according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0029] FIG. 1 illustrates a vehicle inspection system generally designated 10 according to an embodiment of the present invention.

[0030] The system 10 includes an imaging station 12 for acquiring images of the vehicle, a controller C for processing the images to identify one or more defects on the vehicle, and an immersive display device 14. The system 10 may also include a database for storing information about identified defects, such as specific types of defects. The database may also include vehicle identification information, such as the VIN.

[0031] Imaging station 12 may take any suitable form that enables images of the vehicle to be used to identify defects on the vehicle. Imaging station 12 preferably includes one or more digital cameras and image acquisition software configured to acquire one or more digital images of one or more portions of vehicle V. The imaging station may include, for example, an imaging system as described with reference to FIG. 3 of WO 21 / 064351, including a blemish detection camera configured to observe light and dark imaged background surfaces reflected through the vehicle, and a dent detection camera configured to observe structured light images projected by a structured light source.

[0032] The controller C is communicatively connected to the imaging station 12 via a data network and is configured to receive images captured by the imaging station 12 and execute a defect detection algorithm to identify vehicle defects using the evaluation images. The defect detection algorithm is described in more detail with reference to FIG. 4.

[0033] Defect information, such as defect type, size and location, can be stored in a memory resource.

[0034] The controller C is also communicatively connected to the immersive display device 14 via a data network, preferably via a wireless network.

[0035] The controller C is configured to use the defect information to cause the immersive display device 14 to display an identifier that guides the user U to the location of the defect on the vehicle V.

[0036] This can be achieved, for example, by aligning the defect information to the correct location on a digital twin of the vehicle V. The digital twin can be in the form of a virtual 3D model that can be pre-loaded onto the immersive display device 14. In this method, the immersive display device 14 is initialized with the virtual 3D model so that the position, rotation, and scale of the virtual 3D model can be used to match the position, rotation, and scale of the virtual 3D model to the actual vehicle V. This matching can be achieved using one or more computer vision techniques, such as edge detection, feature matching, and SLAM (Simulation Localization And Mapping). Alternatively, this matching can be achieved using software such as Vuforia, Vi This can be achieved by using libraries designed for augmented reality (AR) that perform model tracking, such as sionLib and Wikitude.

[0037] Once the immersive display device 14 has been initialized with the virtual 3D model and a proper alignment with the vehicle V has been obtained, the user U can move around the vehicle V using model tracking techniques without losing track of the virtual and real models of the vehicle V.

[0038] Then, using the defect information provided by the imaging station 12 and the defect detection algorithm, the 3D position (x, y, z) of the defect from the virtual 3D model in the reference frame of the virtual 3D model can be used to generate an overlay of the defect in the correct position, which can be displayed in the correct position on the actual vehicle V. As understood herein, "correct position" refers to the location of the physical defect on the vehicle V.

[0039] The controller C may take any suitable form, such as a general purpose computing device, and may comprise a distributed computing system.

[0040] The data network may comprise a standardized connection for high speed data communication such as USB, IEEE802, or IEEE1394, including wired and / or wireless links.

[0041] The or each controller may be implemented as a special-purpose computer system having a computer processor and non-transitory computer-readable memory that stores computer instructions for performing the functions described herein. The controller C may include one or more digital signal processors (DSPs) for analyzing large amounts of digital image data.

[0042] 2, there is shown a display screen 14a of the immersive display device 14 looking at a vehicle V. A defect 16, here a scratch, is visible on the vehicle V. An identifier 18 is displayed on the display 14a, sized and positioned to immediately surround the defect without obscuring it, in order to direct the user U to the defect.

[0043] The controller C can display at least some of the defect information to the user U on the immersive display device 14 as a defect identifier 20. For example, if the defect detection algorithm identifies a scratch on a door panel when processing the acquired images, the screen of the immersive display device 14 can include a text box that the user can see and that does not obscure the defect on the vehicle V, informing the user U that the identifier is directing them to what the algorithm has estimated to be the scratch.

[0044] The controller C causes the immersive display device 14 to display user-selectable feedback inputs 22 to the user U, allowing the user to confirm whether there is a defect on the vehicle V that matches the displayed damage information. The user can provide an input gesture to confirm whether the damage information is correct or incorrect. This feedback can be communicated to the controller C to train the defect detection algorithm by providing positive or negative reinforcement. For example, if the damage information displayed on the screen informs the user U that the defect is a scratch, but the user U determines the defect is a dent, the user U can select the “N?” input to negatively reinforce the defect detection algorithm.

[0045] The controller C can communicate feedback from the user U to a database of stored defects along with the labels that the user U considers correct for a particular defect. The labels can correspond to the type of defect. Following the example above, the controller can communicate an "N?" input from user U to the database along with a label representing the defect as a dent.

[0046] The machine learning (ML) algorithm can then be retrained, e.g., offline, in a dedicated computing system with updated information from user U to automatically update the AI ​​model used to generate the detections. Once retrained, the new AI model can be used to generate more accurate results based on user U's feedback.

[0047] The controller C may be configured to change the size of the defect identifier 20 based on how close the user is to the vehicle to provide a mixed reality (MR) experience. Similarly, the orientation and / or shape of the identifier 20 may change based on the position and viewing angle of the user U relative to the defect 16.

[0048] The immersive display device 14 may comprise a smartphone, a tablet, smart lenses, AR glasses or headset, or MR glasses or headset. For example, a suitable immersive display device 14 is the Microsoft Hololens 2, etc.

[0049] The system 10 can be configured to allow the vehicle V to be identified using the immersive display device 14, for example, based on a QR code or VIN number on the vehicle. The vehicle identification information can be communicated to the controller C and used to call up the vehicle's identifier and damage information for display on the screen of the immersive device.

[0050] This may be achieved by communicatively connecting controller C via a network connection to a database having information about vehicle V. When user U scans a QR code or enters a VIN number, controller C may query the database for the identifier and damage information for the particular vehicle V. This may be assisted by a network connection that can connect to the database and query for the required information using, for example, SQL queries.

[0051] Still referring to FIG. 3, a computer-implemented method for vehicle inspection is shown generally at 100 .

[0052] In step 102, the method includes processing one or more images of the vehicle using a defect detection algorithm to identify defects on the vehicle.

[0053] In step 104, the method includes generating an identifier on a display of the immersive display device when the vehicle and defect are within a field of view of the immersive display device, the identifier being displayed adjacent to the defect.

[0054] The method may include an optional step 106 of presenting a user-selectable feedback input to the user to confirm whether the defect type label is correct. The input may include, for example, a "correct" button and an "incorrect" button, and the user may select a hand gesture method, such as pressing a corresponding portion of the screen in the case of a phone or tablet, or moving the hand in free space in the case of a headset and being detected by a hand tracking system.

[0055] The method can include an optional step 108 of providing positive or negative reinforcement to train the defect detection algorithm in response to a user selecting a feedback input, such as confirming or rejecting a defect, selecting a correct label, or the like. The training algorithm may include adding defects that were not detected by the defect detection algorithm, or adding defects that were not detected by the defect detection algorithm. Once defects are identified and confirmed, any vehicle images in which the defects are depicted can be labeled by creating bounding boxes in each frame and passed to the training algorithm. Also, if the immersive display device has a built-in camera, additional enlarged images can be stored before or after repair, evaluated at a later time, and used to train the defect detection algorithm.

[0056] The method can include an optional step 110 in which the user selects a function to rate (grade) the defect by size or severity in response to selecting a feedback input. The defect detection algorithm can estimate size based on a localization algorithm and its bounding box, or by comparing the defect to defects graded by previous evaluations. This allows the user to quickly confirm, change, or reject the defect rating while simultaneously viewing the defect and the user interface.

[0057] Processing the image 102 may optionally include identifying multiple defects, and generating identifiers 104 may optionally include generating multiple corresponding identifiers, one for each identified defect. The multiple identifiers may be displayed simultaneously for each defect displayed on the screen, and in some embodiments, the multiple identifiers may be displayed simultaneously until the user selects an input to confirm the defect type is correct, or the multiple identifiers may be generated one at a time, allowing the user to evaluate each defect.

[0058] The method may include an optional step 112 of identifying the vehicle using an immersive display device, for example, based on a QR code or vehicle identification number (VIN) on the vehicle, which may be used to display the correct identifier for the vehicle. This step 112 may be performed at any time between steps 102 and 104, for example, as shown in FIG. 3.

[0059] Referring now to FIG. 4, a system diagram illustrating exemplary AI training phase 130 and AI deployment phase 140 for a system according to an embodiment of the present invention is shown.

[0060] The training phase 130 comprises a data and pre-processing module 132, an AI algorithm module 134, a learning algorithm module 136, and the architecture / platform upon which the training phase 130 is executed.

[0061] In the data and preprocessing module 132, training images of damaged vehicles are provided to indicate what the system is attempting to identify and quantify. For example, images are provided with visible and labeled scratches and / or dents. For each type of damage, the extent can be labeled so that the AI ​​can infer both the type of damage and its extent. Preferably, the images are spaced so that 1 millimeter on the vehicle corresponds to approximately 4 or 5 pixels on the image of the vehicle, ideally 6 pixels / mm or more (25-40 pixels / mm). 2 In one exemplary implementation, this preferred resolution is achieved using 8-12 cameras capturing 61 megapixel images. Alternatively, the sensor resolution may be reduced while the number of cameras and frame rate are increased to capture the required level of detail. The training data also includes labeling information corresponding to regions of the vehicle where instances of damage are located. The labeled regions associated with the images may correspond to bounding boxes that define the regions of the image that contain damage. Thus, the labeling information may include, for a given image, the relative X- and Y-coordinate locations of the top left corner of the region, and the width and / or height of the region. The training data includes a region or regions (shown together) and a label corresponding to the class of damage (e.g., scratch, dent, chip, etc.) contained within the region. Each image in the training data can be associated with multiple labeled regions. In one exemplary implementation, the training data includes 500 manually annotated images of damaged vehicles, each image including one or more labeled regions associated with either a scratch class, a dent class, or a chip class.

[0062] The AI ​​algorithm module 134 may include known algorithms such as convolutional neural networks (CNNs), support vector machines (SVMs), and the like.

[0063] The learning algorithm module 136 applies the training data 132 to an AI algorithm 143. If the AI ​​algorithm is CNN-based, for example, the learning algorithm may include backpropagation with stochastic gradient descent. If the AI ​​algorithm is SVM-based, the learning algorithm may include the use of known methods such as quadratic programming.

[0064] The AI ​​learning platform may comprise any suitable conventional computing device, including, for example, one or more GPUs, and may be implemented as a distributed network of mobile devices.

[0065] The deployment phase 140 forms an important part of the vehicle imaging station 12 and comprises a new data module 142 , a model module 144 and a prediction module 146 .

[0066] The model module 144 includes the trained algorithm output from the learning algorithm module 136 and executed on the controller C, but may alternatively be executed by a data processor on a server, for example.

[0067] The model module 144 receives as input the damage assessment images from the new data module 142. The trained model is thus a program that can be executed to identify defects in a vehicle using the acquired images.

[0068] The model module 144 outputs a prediction 146 including one or more of the occurrence and type of damage, the extent of damage, and the location of damage.

[0069] In one example, the defect multi-tasking CNN uses a machine learning dataset to assess the vehicle condition and provide a probability of damage, a class of damage, and a size of damage. The multi-tasking CNN can operate locally on a data processor associated with the imaging station and / or in cloud-based computing, such as on a server. The multi-tasking CNN can continue to grow and learn using images acquired within the system. In one example, a trained neural network can be expanded using new data. Alternatively, the model can be updated by re-training the entire model, possibly using the trained model as a starting point; i.e., rather than starting with a completely random configuration of network weights, pre-trained weights are used as a starting point.

[0070] In one particular exemplary implementation, the AI ​​algorithm module 134 implements the YOLO algorithm trained on a dataset such as that described above in connection with the data and preprocessing module 132. The YOLO algorithm includes a CSPDarknet53 backbone and a YOLOv5 object detector. The architecture has a cross-stage partial block that splits and merges feature maps in the base layer. The hyperparameters of the YOLO algorithm are determined using a standard grid search approach, and the algorithm uses the Adam optimizer with ε=1e -7 , β1=0.9, β1=0.999.

[0071] Images can be uploaded to cloud storage, where they can be retrieved by clients from anywhere in the world. Once all images are uploaded, they are delivered to the AI ​​via a worker queue, and the service running the AI ​​processes the images in the order they were uploaded. The AI ​​operates directly on vehicle images. The machine learning model can consist of a convolutional neural network with a YOLO object detection architecture, and the ImageNet backbone is trained on tens of thousands of historical examples of vehicle defects so that it can "learn" what constitutes a defect and what does not. The AI ​​model can be deployed on powerful cloud servers with state-of-the-art GPU computing to process images and return detections as coordinate-based bounding boxes.

[0072] In other embodiments, the defect detection algorithm may have any suitable configuration that enables the controller C to generate identifiers on the display of the immersive display device to identify defects on the vehicle.

[0073] The above-described embodiments are illustrative rather than limiting of the present invention, and it will be understood by those skilled in the art that various alternative embodiments can be constructed without departing from the scope of the present invention as defined by the appended claims. In one example, an embodiment of the present invention is applied to a vehicle inspection system having two or more vehicle imaging stations configured to acquire one or more images of a vehicle at different stages of a production process, a controller configured to process the images using a defect detection algorithm to identify defects on the vehicle, a display and interface for identifying and grading the defects, and an immersive display configured to display the condition of the vehicle at different stages of production.

Claims

1. 1. A computer-implemented method for vehicle inspection, comprising: processing one or more images of the vehicle with a defect detection algorithm to identify defects on the vehicle; generating an identifier on a display of the immersive display device when the vehicle and the defect are within a field of view of the immersive display device, the identifier being displayed adjacent to the defect; A method comprising:

2. The method of claim 1 , wherein the identifier is shaped and positioned so as not to obscure the defect.

3. 3. The method of claim 1, wherein the identifier comprises a boundary shape surrounding the defect, the interior of the boundary shape being transparent so that the defect is visible to the user within the boundary shape.

4. 4. The method of claim 1, wherein the identifier is displayed statically relative to the defect.

5. the defect detection algorithm is configured to classify the defects by a defect type label; The method further includes displaying a label of the defect type on the screen.

5. The method according to claim 1, wherein the first and second electrodes are connected to a first electrode.

6. 6. The method of claim 5, wherein the label is displayed on the screen in a position that does not obscure the defect, and optionally the label is positioned adjacent to the identifier so that a user can intuitively associate the label with the defect.

7. 7. The method of claim 5 or 6, further comprising providing the user with a user-selectable feedback input to confirm whether the defect type label is correct.

8. 8. The method of claim 7, further comprising providing positive or negative reinforcement for training the defect detection algorithm in response to the user selecting the feedback input.

9. 9. The method of claim 5, further comprising providing a second feedback input in response to the user selecting the feedback input, the second feedback input providing the user with the ability to rate the defect in size or extent.

10. said step of processing the image includes identifying a plurality of defects; generating an identifier includes generating a corresponding identifier for each identified defect.

10. The method according to any one of claims 1 to 9.

11. 11. The method according to any one of claims 1 to 10, wherein the immersive display device is an augmented reality (AR) or mixed reality (MR) or virtual reality (VR) display device.

12. The method further includes identifying the vehicle using the immersive display device.

12. The method according to any one of claims 1 to 11,

13. 1. A vehicle inspection system comprising: a controller that receives data identifying a defect on a vehicle and displays an identifier on a screen of an immersive display device when the vehicle and the defect are within a field of view of the immersive display device, the identifier being displayed adjacent to the defect.

14. a vehicle imaging station for capturing one or more images of the vehicle; a controller that processes the images using a defect detection algorithm to identify defects on the vehicle; an immersive display device that generates an identifier on a display of the immersive display device when the vehicle and the defect are within a field of view of the immersive display device, and displays the identifier adjacent to the defect on the screen; A vehicle inspection system comprising:

15. two or more vehicle imaging stations that capture one or more images of the vehicle at different stages of the production process; a controller that processes the images using a defect detection algorithm to identify defects on the vehicle; a display device and interface for identifying and evaluating said defects; an immersive display that displays the state of the vehicle at different stages of production; A vehicle inspection system comprising:

Images