Semi-automatic labeling of datasets
Semi-automatic labeling using convolutional neural networks iteratively refines labels with user interaction, addressing inefficiencies in traditional labeling methods, enhancing data quality and processing speed for insurance claim grading and repair estimation.
Patent Information
- Application Number
- JP2022048334
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2015-10-02
- Filing Date
- 2022-03-24
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2036-10-03
AI Technical Summary
The process of labeling training data for supervised learning algorithms is labor-intensive, costly, and inefficient, especially when dealing with large datasets and inconsistent data quality, which hampers the effectiveness of machine learning applications such as auto insurance claim grading and repair estimation.
A semi-automatic labeling method using machine learning models, particularly convolutional neural networks, to iteratively refine labels by user review, allowing passive or active user interaction, and employing techniques like dimension reduction and visualization to enhance efficiency.
This approach significantly reduces the time and cost of labeling large datasets while improving data quality, enabling accurate classification and regression of unlabeled images with varying conditions, facilitating efficient processing of insurance claims.
Smart Images

Figure 0007577085000001 
Figure 0007577085000002 
Figure 0007577085000003
Abstract
Description
[Technical field]
[0001] The present invention relates to classification (or regression) of data in a dataset. In particular, the present invention relates to assigning tags to data in one or more datasets to facilitate the application of machine learning techniques to the one or more datasets. The present invention also relates to a method for computer-assisted quality control during data classification (or regression), as well as a method for semi-automated tagging of data in one or more datasets. [Background technology]
[0002] In the application of supervised learning algorithms for classification (or regression) or regression, the training data must first be correctly labeled, i.e., the dependent variable must be correctly assigned to each data point of the training data. Supervised learning algorithms are regression or classification techniques where the values for the dependent variables are assumed to be known and accurate. The dependent variable is the variable being learned, which is discrete in the case of classification and continuous in the case of regression, and is also known as the tag or label in classification. The values of the dependent variables for the training data may have been obtained by manual annotation, where a knowledgeable human expresses his or her opinion as to what the ground truth data value of the dependent variable is, or by the ground truth data value itself, obtained as a record of the results of the ground truth data by other means. For example, in a geological application, the training set may be a set of 3D seismic scans, where the data points are the voxels in the scans, and the dependent variable is an indication of the resource endowment at the point in space represented by the voxel, which value may have been discovered by drilling or probing. In legal applications, the training set may be a set of past case proceedings, where the data points are collections of documents representing case proceedings, and the ground truth data values of the dependent variable are the actual financial outcomes of the proceedings against the defendants. The fully labeled data is then used to train one or more supervised learning algorithms.
[0003] In many instances, it is necessary for knowledgeable humans to generate training data by adding tags to individual data points. Creating this training data (i.e., correctly classifying the data) is very labor-intensive, costly, and inconvenient, especially when large amounts of training data are to be used and when the quality of the pre-creation of the data is not consistently high. Traditional interactive labeling is computationally expensive and fails to produce good results.
[0004] In traditional image analysis for auto insurance claims rating and repair estimates, images are taken in a controlled environment under standardized conditions (lighting, angle, zoom, background, etc.). To provide images from a controlled environment, special equipment is required at dedicated locations, and the automobiles to be analyzed are transported to those dedicated locations, which is very costly and inconvenient. Summary of the Invention
[0005] Aspects and / or implementations may provide methods and / or systems for labeling data in one or more datasets that may enable labeling of the one or more datasets with improved efficiency.
[0006] Further, aspects and / or implementations may provide an improved system for image analysis for automobile insurance claim rating and repair estimates that may alleviate at least some of the problems described above. In particular, the system may accommodate images from commodity hardware in uncontrolled environments.
[0007] According to one aspect, a method is provided for modeling an unlabeled or partially labeled target dataset with a machine learning model for classification (or regression), the method including processing the target dataset with the machine learning model, creating a subgroup of the target dataset for presentation to a user for labeling or label verification, receiving label verification or user re-labeling or user labeling of the subgroup, and re-processing the updated target dataset with the machine learning model.
[0008] User labeling or label verification combined with modeling of an unclassified or partially classified target dataset with a machine learning model may enable efficient labeling of unlabeled components of the target dataset. By using a machine learning model for modeling, images with various imaging conditions (illumination, angle, zoom, background, occlusion, etc.) can be effectively processed. The machine learning algorithm may be, for example, a convolutional neural network, a support vector machine, a random forest, or a neural network. Optionally, the machine learning model is well suited to perform classification or regression on high-dimensional images (e.g., 10,000 pixels or more).
[0009] Optionally, the method may include determining targeted subgroups of the target dataset for targeted presentation to a user for labeling and label verification of the targeted subgroups, which may enable the user to passively respond to queries suggested to the user, thereby reducing dependency on user initiative, skills, and knowledge to improve model and dataset quality.
[0010] Optionally, the creating may include determining a plurality of representative data instances and creating a cluster plot of only the representative data instances and presenting the cluster plot. This may reduce computational load and allow for rapid display of high-dimensional datasets and thus rapid creation of cluster plots for visualization. Optionally, the plurality of representative data instances may be determined in a feature space. Optionally, the plurality of representative data instances may be determined in an input space. Optionally, the plurality of representative data instances may be determined by sampling. Optionally, the creating may include dimensionality reduction of the plurality of representative data instances to two or three dimensions. Optionally, the dimensionality reduction may be by t-distributed stochastic neighbour embedding (t-SNE).
[0011] Optionally, the creating may include creating the images in a grid and presenting the grid, which may allow for particularly efficient identification of non-standard images.
[0012] Optionally, the creating may include identifying data instances similar to the one or more selected data instances by a Bayesian set method and presenting the similar data instances. The Bayesian set method may allow for particularly efficient processing, thereby reducing the time required to perform the processing.
[0013] According to another aspect, there is provided a method of generating a computational model for estimating vehicle damage repair using a convolutional neural network, the method including receiving a plurality of unlabeled vehicle images, processing the vehicle images with a convolutional neural network, creating a subgroup of vehicle images for presentation to a user for labeling or label verification, receiving a label verification or user re-labeling or user labeling of the subgroup, and re-processing the plurality of vehicle images with the convolutional neural network.
[0014] User labeling or label verification combined with modeling a target dataset containing unlabeled images with a convolutional neural network may enable efficient classification (or regression) of unlabeled images of the target dataset. By using a convolutional neural network for modeling, images with various imaging conditions (illumination, angle, zoom, background, occlusion, etc.) can be effectively processed. Another machine learning algorithm may replace the convolutional neural network.
[0015] Optionally, the method may include determining a targeted subgroup of vehicle images for targeted presentation to a user for labeling and label verification of the targeted subgroup. This may allow the user to passively respond to queries suggested to the user, thereby reducing dependency on user initiative, skills, and knowledge to improve model and dataset quality. Optionally, the creating may include one or more of the steps of creating such data.
[0016] Optionally, the method may further include receiving a plurality of non-vehicle images along with a plurality of unlabeled vehicle images, processing the non-vehicle images along with the vehicle images through a convolutional neural network, preparing the non-vehicle images for presentation to a user for validation, receiving validation of the non-vehicle images, and removing the non-vehicle images to generate a plurality of unlabeled vehicle images, which may enable improvement of the dataset including irrelevant images.
[0017] A subgroup of vehicle images may all show a particular vehicle part. This may allow tagging of images by vehicle part. An image may have more than one vehicle part tag associated with it. A subgroup of vehicle images may all show a particular vehicle part in a damaged state. This may allow labeling of images by damage status. A subgroup of vehicle images may all show a particular vehicle part in a damaged state that is repairable. A subgroup of vehicle images may all show a particular vehicle part in a damaged state that is suitable for replacement. This may allow labeling of images with an indication of whether repair or replacement is most appropriate.
[0018] According to another aspect, there is provided a computational model for estimating vehicle damage repairs generated by the method as described above, which may enable particularly good generation of a model capable of modeling vehicle damage and appropriate repair / replacement responses.
[0019] The computational model may be adapted to calculate a repair cost estimate by identifying one or more damaged portions from the image, identifying whether the damaged portions are repairable or suitable for replacement, and calculating a repair cost estimate for the vehicle damage, which may enable faster processing of insurance claims for the vehicle damage.
[0020] Optionally, to enhance usability, the computational model may be adapted to calculate a certainty of the repair cost estimate. Optionally, to enhance usability, the computational model may be adapted to determine a scrapping recommendation. Optionally, to enhance the quality of the repair cost estimate, the computational model may be adapted to calculate its output conditional on a plurality of images of the damaged vehicle to estimate the vehicle damage repair. Optionally, to enhance the quality of the repair cost estimate, the computational model may be adapted to receive a plurality of images of the damaged vehicle to estimate the vehicle damage repair. Optionally, to enhance usability, the computational model may be adapted to calculate an estimate of interior damage. Optionally, to enhance usability, the computational model may be adapted to request one or more further images from the user.
[0021] According to another aspect, there is provided software adapted to generate the computational model as above. According to another aspect, there is provided a processor adapted to generate the computational model as above.
[0022] Aspects and / or embodiments may extend to a method of modeling data substantially as described herein and / or illustrated with reference to the accompanying drawings.
[0023] Aspects and / or embodiments may also extend to a method of generating a computational model for estimating vehicle damage repairs as described herein and / or illustrated with reference to the accompanying drawings.
[0024] Aspects and / or embodiments may also extend to computational models substantially as described herein and / or illustrated with reference to the accompanying drawings.
[0025] Aspects and / or embodiments may also extend to software for modelling data substantially as described herein and / or illustrated with reference to the accompanying drawings.
[0026] Aspects and / or embodiments may also extend to a system for modeling data substantially as described herein and / or illustrated with reference to the accompanying drawings.
[0027] Aspects and / or embodiments may also extend to methods and / or apparatus substantially as described herein and / or illustrated with reference to the accompanying drawings.
[0028] Aspects and / or embodiments may also provide computer programs and computer program products for performing any of the methods described herein and / or embodying any of the apparatus features described herein, as well as computer readable media having stored thereon a program for performing any of the methods described herein and / or embodying any of the apparatus features described herein.
[0029] Aspects and / or embodiments may also provide signals embodying computer programs performing any of the methods described herein and / or embodying any of the apparatus features described herein, methods of transmitting such signals, and computer products having an operating system supporting computer programs performing any of the methods described herein and / or embodying any of the apparatus features described herein.
[0030] Any apparatus features described herein may also be provided as method features, and vice versa. As used herein, means-plus-function features may alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated memory.
[0031] Any feature in one embodiment may be applied to other embodiments in any suitable combination. In particular, method embodiments may be applied to apparatus embodiments and vice versa. Furthermore, any, some, and / or all features in one embodiment may be applied to any, some, and / or all features in any other embodiment in any suitable combination.
[0032] It is to be understood that specific combinations of the various features described and defined in any embodiment may be incorporated and / or provided and / or used independently.
[0033] Furthermore, features implemented in hardware may generally be implemented in software and vice versa, and any references to software and hardware features within this specification should be interpreted accordingly. [Brief description of the drawings]
[0034] These and other aspects of the present invention will become apparent from the following example embodiments described with reference to the following drawings, which have like reference numerals and in which: [Figure 1] FIG. 1 is a schematic diagram of a method for labeling data. [Diagram 2] FIG. 2 is a schematic diagram of the steps of the method of FIG. 1. [Diagram 3] FIG. 1 is a schematic diagram of a system for labeling data. [Figure 4a] FIG. 1 is a diagram of a graphic user interface with cluster plots. [Figure 4b] FIG. 1 is a diagram of a graphic user interface with cluster plots. [Diagram 5] FIG. 1 is a diagram of a graphic user interface using a grid of images. [Figure 6a] FIG. 1 is a diagram of a graphic user interface for targeted monitoring. [Figure 6b]FIG. 1 is a diagram of a graphic user interface for targeted monitoring. [Figure 7] 1 is a schematic diagram of a system for vehicle damage estimation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0035] For roughly a decade, vehicle body shops and claims adjusters in many countries have been taking photographs of damaged vehicles as evidence to support repair estimates submitted to insurers or solicitors. With roughly 19 million auto insurance claims filed each year in the United States alone, and roughly 10 images per claim, there is a large amount of image data on damaged vehicles.
[0036] Machine learning is an attractive tool to leverage existing vehicle damage images, and deep learning (and convolutional neural networks in particular) has made great strides towards automatic recognition and understanding of high-dimensional sensory data. One of the fundamental ideas behind these techniques is that algorithms can determine how to best represent the data by learning to extract the most useful features. If the extracted features are sufficient (sufficiently differentiable), any basic machine learning algorithm can be applied to the extracted features to obtain the best results. Convolutional neural networks (also referred to as convnets or CNNs) are particularly well suited for categorizing image data, and graphic processor unit (GPU) implementations of convolutional neural networks trained by supervised learning have demonstrated high image classification (or regression) performance on "pristine" images (employed under non-standardized conditions, across car models, with diversity in e.g. lighting, angle, zoom, background, occlusion, and design, including errors and irrelevant images, and with diversity in terms of quality and reliability).
[0037] To utilize a large amount of vehicle damage images to train convolutional neural networks, the data needs to be as error-free as possible, and in particular the images need to be correctly labeled. Industrial datasets pose novel problems for deep learning, such as dealing with noisy / missing / inconsistently or partially labeled data, which may also contain irrelevant data.
[0038] For machine learning to perform good classification (or regression), it is necessary to have good data quality for training and to train a good enough model on that data. Traditionally, users are required to first prepare data for training by going through the data and (re)labeling it until they are satisfied with the quality. The model is then trained on the clean data.
[0039] Labeling (and more generally cleaning) a training dataset by a user assigning labels to images is a very time-consuming and costly procedure that is prohibitive in commercial applications.
[0040] If the creation of the training dataset and the training of the model are interleaved, a greatly improved efficiency can be achieved. This is not an intuitive approach, since the algorithm starts learning with a dataset that is known to be insufficient. However, if it takes advantage of the ability of machine learning algorithms to identify heterogeneous and potentially erroneous datasets, the algorithm can be very efficient. Each iteration of model training informs the best approach for the subsequent relabeling iteration (and vice versa). The end result of this iterative process is a dataset of sufficient quality and a model that provides sufficient differentiation features for this dataset.
[0041] The data can be in the form of images (where each image represents an individual data set), or the data can be any high-dimensional data, such as text (where each word, for example, represents an individual data set) or sound.
[0042] To enable the use of existing image data to train convolutional neural networks, we now describe semi-automatic labeling.
[0043] Semi-automated labeling semi-automates the labeling of a dataset. A model is trained on data known to contain errors. The model attempts to model and classify (or regress) the data. The classification, also referred to as labeling or tagging, of selected data points (individual images or groups of images) is reviewed and corrected or approved by a user (also referred to as an authority or supervisor). The labels are refined iteratively and then the model is refined based on the labeled data. The user can actively review the model output and search for images for review and labeling, or the user can passively respond to queries from the model regarding the labeling of specific images.
[0044] Figure 1 is a schematic diagram of a method for semi-automatic labeling. Figure 2 is a schematic diagram of steps of the method of semi-automatic labeling of Figure 1. Figure 3 is a schematic diagram of a system 100 for semi-automatic labeling. The processor 104 provides information to a user 110 via an input / output 108 regarding how the dataset 102 is to be modeled using a computational model 106. The user 110 provides guidance to the processor 104 via the input / output 108 for modeling the dataset 102 using the computational model 106.
[0045] The operation sequence for semi-automatic labeling with active user review is as follows: 1. Pre-train the model on the most similar data possible (in terms of quantity and labels). 2. Model the target data with the pre-trained model. 3. Create modeled target data for user review.
[0046] A model is used to extract features of the target dataset (called a feature set).
[0047] b. Perform dimensionality reduction on the feature set.
[0048] c. Assign no labels to feature points / assign labels to some feature points / assign labels to all feature points.
[0049] d. Applying visualization techniques to the labeled feature set. 4. Present the user with an efficient interface for viewing and editing the tagged feature sets.
[0050] A user efficiently browses a set of labeled features to find areas to examine.
[0051] b. The user verifies or modifies the labels displayed on the interface. 5. Repeat the cycle from step 2 with the verified / corrected labeling until sufficient data and model quality is achieved. 6. Use some / all of the labeled dataset or feature sets to fine-tune the state-of-the-art feature extraction model until sufficient data and model quality is achieved.
[0052] In an example of a semi-automatic labeling procedure as designed above, approximately 30,000 images can be labeled into a scheme with 18 classes with 90% accuracy in one hour by a single user.
[0053] In the case of passive user response to a query (also called targeted monitoring), steps 3 and 4 of the above sequence are as follows. 3. Prepare full modeled data for user review.
[0054] A model is used to extract features of the target dataset (called a feature set).
[0055] b. Perform dimensionality reduction on the feature set.
[0056] c. Assign no labels to feature points / assign labels to some feature points / assign labels to all feature points.
[0057] d. Applying visualization techniques to the labeled feature set.
[0058] e. Estimate the next best user query. 4. Present a query to the user to review the labeled feature set.
[0059] a. Efficiently presenting queries to users.
[0060] b. The user verifies or modifies the labels displayed on the interface.
[0061] Passive and active user reviews can also be combined by providing both in parallel with each other.
[0062] Step 3c of "assigning labels to some / all feature points" may be performed for classification by clustering techniques such as partitioning the feature space into class regions. Step 3c may also be performed for regression by discrete techniques such as defining discrete random values across the feature space.
[0063] As part of step 6 (fine tuning), the following additional steps may be performed:
[0064] a. Run the model on unseen data and rank the images by the probability of classification (or regression) (this is possible because it's binary).
[0065] b. Presenting high and low probability images to the user, particularly for identifying errors in the information.
[0066] In a variant, semantic clustering in a cluster plot (e.g., data is shown separated by image content, e.g., all car bumper images are shown together) is augmented with probability ranks (e.g., with colors representing probability) to allow for more effective fine-tuning.
[0067] There are several additional considerations to take into account in implementing the sequence designed above, as follows: Initialize the process by making the best use of any existing labels. In the worst case, the labels are useless and unsupervised initialization is performed. Otherwise, a supervised model can be trained on any available labels. Optimize the visualization of extracted features to help the user understand what the model is doing. Since real features exist in a high-dimensional space (i.e., >1000 dimensions), they need to be reduced to 2 or 3 dimensions while preserving as much information as possible. Doing this visualization in real time provides great benefits. Relabeling a portion of the data to maximize the benefit to the next training iteration. One approach is for the model to give the user a ranked list of images / image clusters that were deemed "most confusing" during model training. Optimizing the retraining of the model to take into account new user input. In the simplest case, the user specifies the extent to which they think the model should be retrained. This affects how representative the retraining is and how long it takes. The model needs to be expressive enough to take advantage of new information given to it, but not so much that it overfits the new data. Evaluate the actual performance of the model for each iteration. Usually, some portion of the data is not used for training, so the performance of the model can be evaluated on that portion. However, not using a small amount of recently relabeled data for training can significantly slow down the relabeling cycle. A good balance must be found between the two.
[0068] Some techniques that can be used to perform the semi-automatic labeling described above are as follows. Pre-trained convolutional neural networks Extract features by parallelizing across GPUs Principal Component Analysis (PCA) for dimensionality reduction. This is especially suitable for t-distributed stochastic neighbour embedding (tSNE), for Bayesian sets PCA may not be a good fit. If tSNE is fast enough, dimensionality reduction may not even be necessary. Feature set exploration for seeding centroids using k-means clustering algorithm · t-distributed stochastic neighbour embedding (tSNE) for k-means centroids A Graphic User Interface (GUI) with a cluster plot of tSNE, where clusters are represented by circles with centroids as centers, number of images represented by diameters, and most prevalent class color as color. -GUI grid of up to 100 images to validate / edit labels Bayesian Sets Applied to Convolutional Neural Networks Softmax refinement of the model -Siamese tweaks to the model -Tweaking triplet loss of the model The pre-trained convolutional neural network may, for example, be trained on images from the ImageNet ensemble.
[0069] FIG. 4a is an illustration of a graphic user interface with a cluster plot that provides semantic clustering (e.g., all car bumper images are in the same area in the cluster plot). The cluster plot shows circles that indicate the distribution of a data set in a feature space. The plot is presented to a user who may then select one or more of the circles for further review. Labeled / unlabeled status may be indicated in the plot, for example, by the color of the circle. Selected / unselected for review may be indicated in the plot, for example, by the color of the circle. FIG. 4b is an illustration of a graphic user interface with a cluster plot, where the color of the circle indicates the label associated with the data. When the user hovers the mouse pointer over a circle, the user may be presented with image data. User selection of a group of circles may be accomplished by having the user draw a border around the group of interest in the cluster plot.
[0070] FIG. 5 is an illustration of a graphic user interface using a grid of images. Images selected in the cluster plot are shown in the grid for user review. The grid may, for example, have a row of eight images across and six columns of images below each other. In the illustrated example, the grid shows 7×5 images. The human visual cortex is particularly efficient at organizing and identifying disparate images in a grid format. By displaying images in a grid format, a large number of images may be presented to the user for review by the user in a short period of time. For example, with 48 images per screen, 21 screens may allow a user to review over 1000 images. Images in the grid may be selected or deselected for labeling with a particular label. Images may be selected or deselected for further review, such as similarity searching.
[0071] Similarity searches may be performed to find images that are similar to a particular image or group of images of interest. This may allow a user to find an individual image of particular interest (e.g., an image of a scratched windshield within a cluster of windshield images), find additional images that are similar, and provide labels for the images collectively.
[0072] Figures 6a and 6b are diagrams of a graphic user interface for targeted surveillance, where a number of images (7 images in the illustrated example) that appear to be clustered are provided to the user, and a field is provided for user input of labels for the images. Figure 6a shows an empty field for user input, while Figure 6b shows the field with a label entered by the user, as well as an image marked with a colored frame, the color indicating the label associated with that image.
[0073] We now describe in more detail how to perform dimensionality reduction on the feature set (step 3.c above). In the example, the feature set is a 4096-dimensional vector (and more generally, an N-dimensional vector) with values ranging from approximately -2 to 2 (and more generally, a typical range). Dimensionality reduction to two or three dimensions (as may be intuitively understood by humans) may require enormous computational resources and may take an enormous amount of time. To shorten this computationally labor-intensive step, the dataset is clustered in the feature space and a single representative data instance (also referred to as a centroid, e.g., a k-means cluster centroid) from each cluster is selected for further processing. Dimensionality reduction is then performed only on the representative data, thereby reducing the computational load to such an extent that very rapid visualization of very large datasets is possible. Data points from the dataset are not individually shown to the user in the cluster plot, but the diameter of the circle in the cluster plot shown to the user indicates the number of data points that are close to the associated representative data instance in the feature space and therefore inferred to have the same or similar label value. By selecting a circle in the cluster plot, the user is presented with all of the images represented by that circle. This allows the user to check all of the images represented by the representative image. The scaling of the circle can be optimized and / or adjusted by the user for clarity of display.
[0074] We now describe in more detail how to perform similarity searches. Images are represented in feature space by high-dimensional vectors (e.g., 4096-dimensional vectors) with a range of values (e.g., approximately -2 to 2). Performing similarity searches on a large number of such vectors can be computationally labor-intensive and time-consuming. Bayesian sets can provide a very quick and easy means of identifying entities similar to an image or group of images of particular interest. To apply the Bayesian set method, the data (here the high-dimensional vectors) is required to be binary rather than having a range of values. To apply the Bayesian set method, the feature set vectors are converted to binary vectors; i.e., values close to zero are changed to zero and values far from zero are changed to one. For similarity searches with the Bayesian set method, this can yield good results. The application of Bayesian sets to convolutional neural networks (or more generally, machine learning models that are suitable for images and use sparse representations) is particularly beneficial because convolutional neural networks typically produce feature sets with sparse representations (many zeros in the vector) that are consequently easier to cast into binary vectors with sparse representations in the context of semi-automated labeling.
[0075] We now describe in more detail the semi-automatic labeling applied to vehicle damage estimation. For a given instance of vehicle damage, the result is a prediction of required repairs and a corresponding repair cost estimate based on pristine images of the damaged vehicle. This may enable an insurer, for example, to make a decision on how to proceed with a response to the vehicle damage. The result may include a rating recommendation such as "scrap vehicle", "major repairs needed", or "minor repairs needed".
[0076] 7 is a schematic diagram of a system 700 for vehicle damage estimation. A user 710 takes an image 712 of a damaged vehicle 716 using a camera 714 and transmits the image 712 to the system 700 via a mobile device 708 (e.g., a tablet or smartphone). The processor 704 evaluates the image 712 using a computational model 706 and generates a vehicle damage estimate, which is provided to the user 710 via the mobile device 708. A report may be provided to other parties, such as an insurance carrier or a vehicle repair shop. The image 712 may be taken directly by the mobile device 708. The image 712 may be added to the dataset 702 and the model 706 may be updated with the image 712.
[0077] To generate a repair estimate, the procedure is divided into the following steps for optimal processing:
[0078] 1. Recognize the set of damaged parts via deep learning (preferably convolutional neural network). For example, images provided by the vehicle owner do not provide part labels, so a fairly rich model for image data is required. The vehicle owner may be required to provide an image in which the entire vehicle is visible. Real-time interactive feedback to the user may be implemented to ensure that the most suitable and appropriate image is provided. For example, feeding images through one or more "quality assurance" classifiers and returning results in real time ensures that the user takes all the images required for an accurate repair estimate.
[0079] 2. Predict "repair" / "replace" label for each damaged part via a convolutional neural network. The repair / replace distinction is typically very noisy and may result in mislabeling. To solve this, part labels per image are identified. Repair / replace labels are then more reliable since they are per part, not per image. Cross-referencing may help to obtain repair / replace labels for individual images where the corresponding part is present. To remove the need for zoomed-in images, a related product of images where the entire vehicle is present may be created. Real-time interactive feedback to the user may be implemented to obtain specific zoomed-in images for parts when there is low certainty of not obtaining them. Step 2 may be combined with the previous step 1 by predicting "not visible" / "no damage" / "repair" / "replace" label for each part.
[0080] 2.5. Convolutional Neural Networks and Predictive Analytics Predict “No Damage” / “Repair” / “Replace” Labels for Relevant Interior Parts. Accurately predicting interior damage is difficult and even expert human assessors may struggle. To enable better results, telematics data may be provided from the vehicle to determine which internal electronic components are faulty / working and to add to the predictive analytics regression (e.g., accelerometer data).
[0081] 3. Obtain the working time for performing each working operation, for example by prediction or by taking an average. This step may also involve a convolutional neural network. It may be preferable to predict the severity of the damage rather than the working time itself. Working time data may be obtained from a third party. If an average time is used, adjustment of the average time may be made depending on one or more easily observable parameters, such as the model type of the vehicle, the set of all damaged parts, the severity of the damage, etc.
[0082] 4. Obtain part prices & labor rates for replacing each part. Prices and rates may be obtained by lookup or by taking average values. To look up prices and rates, an API call may be made to, for example, an insurance provider, third party, or associated repair shop database. Average values may be obtained by lookup, and if an average price or rate is used, adjustments to that average price or rate may be made depending on one or more observable or available parameters such as model type, set of all damaged parts, severity of damage, fault / no fault, etc.
[0083] 5. Calculate repair estimates by adding and multiplying prices, wages, and hours. The uncertainty of the repair estimates can also be modeled to obtain a posterior distribution of the repair estimates. For example, a 95% confidence interval of the total repair cost, or the probability that the vehicle will be scrapped, may be provided. If there is insufficient confidence in the repair estimate, the claim may be delegated to a human.
[0084] With this procedure, a repair estimate can be generated upon first damage notification by the policyholder, for example from an image taken with a smartphone. This may allow for claims processing almost immediately after damage to the vehicle occurs. It may also allow for quick selection of, for example, by a mobile app: If the damaged vehicle is a total loss, a new vehicle If major repairs are required, a loaner car If major repairs are required, a repair shop with suitable capabilities and prices · Replacement parts from your preferred supplier for fast part sourcing when major repairs are required On-site repairs if minor damage occurs (e.g. repairing scratches on windshield) Images can be provided for repair estimates at a time later than the first damage notification, e.g. after official services such as police or emergency personnel have left, or at a body shop or other specialized facility. An output posterior distribution of repair estimates can be generated, e.g. to provide a better perspective of the 95% confidence interval of the repair estimate, or the probability of scrapping. The repair estimate process can be dual machine / human generated, e.g. by handing the estimate over to a human operator in cases where the estimate obtained by the model has low confidence or in sensitive cases. Images can be taken by parties other than the policyholder (e.g. passengers in the damaged vehicle, other parties involved in the accident, police officers, ambulance / emergency personnel, damage adjusters / valuers, insurance agent, broker, solicitor, repair shop employee). Images provided for repair estimates can be from a camera or other photographic device. Other relevant information, such as excess values and / or expected premium increases, can be provided to the policyholder to discourage claims.
[0085] By implementing repair estimating as described above, several benefits can now be achieved by both the insurer and the policyholder. For example, the insurer can: · Reduce administrative costs of administering claims. · Reduce claims rates by providing an accurate or at least a good approximation of appropriate premium increases. · Lower claim costs through faster processing and reduced chances of costly injury claims. · Reduce no-fault claims (in certain countries) by sending policyholders directly into a managed repair chain. Reduce operating time. · Increase customer retention. · Incentivize potential customers to switch insurers.
[0086] Policyholders get superior customer service and can take advantage of suppliers who bid against their regular customers, certain parts suppliers can benefit from preferred supplier status, and vehicle repair and body shops can avoid spending time preparing quotes.
[0087] In the above steps, a convolutional neural network is used. A multi-instance learning (MIL) convolutional neural network that can accommodate multi-image queries may work significantly better than a convolutional neural network for single image queries. Multiple images may help remove image noise, especially from angles, lighting, occlusion, lack of context, insufficient resolution, etc. In the case of classification, this differs from traditional image classification, where a class is output conditional on a single image. In the context of collision repair estimates, it may often be impossible to capture all the information required to output the repair estimate components in a single image. In the example, the fact that the rear bumper needs repair may only be recognized by taking a close-up image of the damage, which loses the context information required to confirm that it is part of the rear bumper that is being photographed. By training a machine learning model that uses information in the multiple images in the example, the machine learning model can output that the rear bumper needs repair. In a convolutional neural network architecture that can accommodate multi-image queries, a layer is provided in the convolutional neural network that pools across images. Max pooling, average pooling, median pooling, or learning pooling may be applied. For greater simplicity, a single image convolutional neural network may be used.
[0088] We now describe in more detail the procedure for generating a model capable of achieving steps 1 and 2 of generating a repair estimate as described above: recognizing a set of damaged parts and predicting a "repair" / "replace" label. This is essentially achieved by solving the labeling problem by semi-automatic labeling as described above. The procedure is applied to a dataset containing unlabeled vehicle images for each vehicle part to be recognized / diagnosed.
[0089] A. Remove irrelevant images. By removing irrelevant data, the data becomes more easily presentable.
[0090] 1. Extract features for the target dataset using a pre-trained model (as described in more detail above).
[0091] 2. Present the user with how the data is modeled (tSNE GUI plot as above), allowing the user to easily identify irrelevant clusters since they are semantically distinct.
[0092] 3. Receive user selection (or acceptance) of irrelevant clusters and remove the corresponding images from the dataset.
[0093] 4. Repeat until no further irrelevant images are removed.
[0094] B. Create classifiers for "part invisible", "part undamaged", and "part damaged".
[0095] 1. Extract features for the target dataset using the model and the target data as generated in step A above.
[0096] 2. Present the user with how the data is modeled (a tSNE GUI plot as above), allowing the user to identify highly biased clusters and label them if appropriate.
[0097] If an area of feature space has not been explored, the user is presented with how a subset of the data that the user has not yet examined would be modeled. The user may explore such information, or active learning algorithms may be used to identify and provide areas of review to the user.
[0098] - For unbiased clusters, provide images to the user for browsing and labeling using similarity search.
[0099] Similarity searching can result in rapid identification of images that have common labels.
[0100] The user has prior knowledge of the class hierarchy with subclasses (and possibly also densities) to ensure that the model correctly represents the real-world vehicle damage possibilities (e.g., if a certain type of repairable left front fender damage can occur in reality, the model needs to be able to identify such cases).
[0101] If the identified features do not adequately disentangle the class hierarchy, increased user supervision may be required.
[0102] If the user does not have an established class hierarchy available, the user can build subclasses on an ad-hoc basis by browsing and learning from the dataset.
[0103] A distribution is generated for each cluster and each page. If prominent cases are reached, the user can ponder on them for longer and explore them by similarity search.
[0104] 3. Receive user labeling (or label verification) and update the dataset.
[0105] 4. Train the model: If the partial classification (or regression) is not satisfactory, repeat the cycle from step 2 with the validated / corrected labeling until sufficient data and model quality is achieved.
[0106] 5. Once features are no longer discriminative (e.g., cluster content is found to be less variable and label editing becomes a matter of more subtle visual patterns), fine-tuning is performed. Rather than working through cycles in sequence, fine-tuning can also be sandwiched between or combined with previous cycles.
[0107] 6. Extract features from the target dataset.
[0108] 7. Present to the user how the data is modeled. Images may be presented ranked by the classification (or regression) output so that the user can browse through the classification (or regression) output to understand which subclasses the model correctly identified and which subclasses were not correctly recognized. The user can focus on the next step of learning depending on which subclasses were not correctly recognized by a similarity search. Suggested next learning steps may be provided to the user by active learning techniques that can automate the browsing and identification of the incorrectly recognized subclasses.
[0109] 8. Receive guidance from the user and update the dataset accordingly.
[0110] 9. Train the model. If the model accuracy is not satisfactory, repeat the cycle from step 6 with the verified / corrected labeling until sufficient data and model quality are achieved.
[0111] C. Create a “repair part” and “replacement part” classifier (the target dataset may contain partially mislabeled images).
[0112] 1. Extract repair / replacement metadata from a csv / txt file that associates specific damaged part images with the appropriate action.
[0113] 2. Assign repairs / replacements to parts labeled "damaged parts."
[0114] 3. Train the model with the updated target dataset and extract features from the dataset.
[0115] 4. Present the user with how the data is modeled (a tSNE GUI plot as above), allowing the user to identify highly biased clusters and label them if appropriate.
[0116] - In the case of unbiased clusters, provide the images to the user for viewing and labeling using similarity search, as described in more detail in step B.4 above.
[0117] 5. Receive user labeling (or label verification) and update the dataset.
[0118] 6. Train the model: If the subclassification (or regression) is not satisfactory, repeat the cycle from step 4 with the validated / corrected labeling until the model accuracy is satisfactory.
[0119] D. Combine the labeled data from steps B and C to train a single 4-class classifier ('unseen parts', 'undamaged parts', 'repair parts', 'replacement parts').
[0120] E. Measure the accuracy of the trained model. For this, a random test dataset is required. The preferred technique for obtaining a test dataset is to take a random sample from the full dataset and then have a user view all images in the test dataset and correctly assign all labels. Some assistance may be available from semi-automated labeling, but the correct labeling of all images in the test dataset must be verified by the user.
[0121] The application of interior damage prediction will now be described in more detail. Interior damage prediction can be performed with predictive analytics, such as regression models, for example. Images of damaged vehicles do not allow direct observation of the interior parts.
[0122] A. Predict repair estimates: Regress on repair costs.
[0123] 1. Determine a measure of the predictive ability of the image: Regress the total cost of repair with gradually decreasing regressors. In practice, the way in which regressors that are expensive to measure can be estimated and removed is as follows.
[0124] - Record and review the status of only a few parts. It may be possible to generate accurate estimates of the total cost. The number of parts that can be omitted from the regression model is analyzed.
[0125] In some cases, images of interior parts of the vehicle are recorded and examined (for example by opening the hood) and even certain parts are removed to view certain interior parts. In some cases, it is sufficient to record and examine images of only the exterior of the vehicle. The number of interior parts that can be omitted from the regression model is analyzed.
[0126] - Consider the degree of damage of a part to decide on a labor action (repair, replace, do nothing). This can be fed with the output of a repair / replace classifier (trained on semi-automatically labeled data as above).
[0127] - Consider component pricing: for example, consider exact original equipment part prices, current / historical average prices, Thatcham prices.
[0128] - Consider whether it is a fault / no-fault claim.
[0129] - Assess total labour costs: for example by reference to exact labour rates, average labour rates or fault / no-fault labour rates and also by reference to exact labour hours, average labour hours or Thatcham labour hours for each labour movement.
[0130] - Consider other metadata such as car type, mileage, etc.
[0131] - Evaluate the sensitivity of the prediction (x% classification error => y% cost prediction error).
[0132] - Consider whether a typical expected error (e.g., 6%) can be predicted by metadata fields such as type of damage, company providing the estimate, etc.
[0133] - Consider a rule-based sequence of available labor from a lookup.
[0134] 2. Assess the predictive power of the images.
[0135] - Take the top regression model from above and replace certain ground truth data values with the convolutional neural network results: replace the "repair" / "replace" labels for visible parts with the equivalent predictions from the convolutional neural network model. In this way, the classification output flows into a regression. The regression parameters may be fine-tuned to match the convolutional neural network output. The number of parts considered is reduced as the number of parts that can be omitted from the regression model is analyzed.
[0136] - A convolutional neural network is trained to perform the regression directly on the image. The total cost is regressed on the image and all other observables. The error in the predicted repair cost is reflected.
[0137] B. Predicting Total Loss: Regressing Scrapped Vehicles The step performed for step A above (regressing repair costs) is adapted to regress a binary indicator of whether to scrap a damaged vehicle instead of paying the repair costs to repair it.
[0138] In the above process, the sequence of steps can vary. More information is available in the images of the damaged parts than in binary repair / replace decisions. Therefore, by regressing the repair costs onto the images, we can increase accuracy compared to image-free models.
[0139] The implementation of the repair estimate may include further features such as: Features to prevent and detect image fraud and other fraud Features that determine who is at fault; and / or Features to take and analyze images of the other vehicle and / or property involved in the collision for processing by the insurance provider It will be understood that the invention has been described above purely by way of example and modifications of detail can be made within the scope of the invention.
[0140] Each feature disclosed in this specification, and (where appropriate) the claims and drawings, may be provided independently or in any suitable combination.
[0141] Any reference numerals appearing in the claims are by way of illustration only and shall have no limitation on the scope of the claims.
Claims
1. 1. A processor-implemented method for estimating repairs for vehicle damage using a computational model, comprising: Using the computational model to evaluate images of a damaged vehicle and generate a vehicle damage estimate; The step of generating a vehicle damage estimate includes the steps of identifying a damaged portion; determining whether the damaged part is repairable or suitable for replacement using an image classifier that outputs a repair or replacement label for the damaged part; and predicting a repair or replacement label for the damaged part; The repair label indicates that the damaged part should be repaired and the replacement label indicates that the damaged part should be replaced.
2. The method of claim 1 , wherein the computational model includes a convolutional neural network for predicting a repair or replacement label for the damaged portion.
3. The method of claim 1 or claim 2, wherein the computational model includes a convolutional neural network for recognizing the damaged portion.
4. The method of claim 1 , wherein generating the vehicle damage estimate comprises generating a repair cost estimate.
5. The method of claim 1 , wherein generating the vehicle damage estimate comprises recognizing the set of damaged parts and predicting a repair or replacement label for each damaged part.
6. The method of claim 1 , wherein generating the vehicle damage estimate comprises predicting a total repair cost and / or indicating whether to scrap the damaged vehicle.
7. The method of claim 1 , wherein the computational model is capable of supporting multi-image queries.
8. 8. The method of claim 1, wherein generating the vehicle damage estimate comprises predicting labor times for repairs by evaluating images of the damaged vehicle using a convolutional neural network for the damaged portions labeled with the repairs.
9. 9. The method of claim 1, wherein generating the vehicle damage estimate comprises, for the damaged part labeled for replacement, obtaining parts and labor costs for replacing the damaged part.
10. 1. A system for vehicle damage estimation including a processor, the processor includes a computational model adapted to evaluate images of a damaged vehicle and generate a vehicle damage estimate; Generating the vehicle damage estimate includes identifying a damaged portion; determining whether the damaged part is repairable or suitable for replacement using an image classifier that outputs a repair or replacement label for the damaged part; and predicting a repair or replacement label for the damaged part; The system wherein the repair label indicates that the damaged part should be repaired and the replacement label indicates that the damaged part should be replaced.
11. The system of claim 10 , wherein the computational model includes a convolutional neural network for predicting a repair or replacement label for the damaged portion.
12. The system of claim 10 or claim 11, wherein the computational model is capable of supporting multi-image queries.
13. 13. The system of claim 10, wherein generating the vehicle damage estimate includes predicting labor times for repairs by evaluating images of the damaged vehicle using a convolutional neural network for the damaged portions labeled with the repairs.
14. 14. The system of claim 10, wherein generating the vehicle damage estimate includes obtaining parts and labor costs for replacement for the damaged parts labeled for replacement.
15. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Vehicle repair / replacement information management system, and vehicle abnormality cause information management system
CN102150174A
Freight wagon blocking key missing fault identification method based on support vector machine
CN103295027A
Vehicle loss assessment method and device and terminal device
CN104268783A
Key part fault detection method and device
CN104408477A
Vehicle repair cost estimation system and recording medium storing repair cost estimating program
JP1999073456A
