Method and system for locating vehicle abnormality based on image

Abstract

The present invention provides a method and apparatus for anomaly detection based on a neural network, which can be trained on undamaged normal vehicle images and can detect unknown / never seen vehicle damage of random types and ranges from images acquired in various contexts. The provided method and apparatus are implemented by functional units that are trained to perform anomaly detection using a training data set under a GCAN model, the training data set containing images of undamaged vehicles, complete vehicle frame images, and enhanced vehicle frame images of vehicles.

Description

Technical Field

[0001] The present invention generally relates to image processing technology. More specifically, the present invention relates to a technology for locating abnormal areas in vehicle images using a Generative Contextualized Adversarial Network (GCAN). Background Art

[0002] When an insurance claim is filed for an automobile as a result of a traffic accident, in addition to the information provided by the claimant for insurance claim processing, it is often necessary for an inspector or claims adjuster to obtain photographs and perform an on-site assessment of the extent of the damage to the vehicle before the insurance company can provide accurate compensation to the insured.

[0003] The industry has adopted many methods to remotely assess the damage of vehicles. For example, U.S. Patent Application Publication No. 2018 / 0260793A1 discloses a method for automatically estimating the repair cost of a vehicle based on the use of a classical convolutional neural network (CNN), which requires multiple historical images in its training. U.S. Patent Application Publication No. 2018 / 0293552A1 discloses a variety of methods for generating vehicle maintenance plans based on damage areas and damage types; U.S. Patent Application Publication No. 2019 / 0244337A1 discloses an unsupervised image-based anomaly detection system using a multi-scale context-dependent deep auto-encoding Gaussian mixture model.

[0004] Nevertheless, the problem with the above neural network-based methods is that they require a large number of annotated images of damaged vehicles or vehicle components as training datasets for machine learning. Since images of vehicle damage are relatively rare and the damage variations are too large (e.g., there is no "typical" damage), training datasets based on images of damaged vehicles are always insufficient to achieve detection capabilities to detect random types and ranges of unknown / never seen vehicle damage, such as shape deformation of the vehicle frame, from images acquired in various contexts. Therefore, a better approach is needed in this area. Summary of the invention

[0005] An object of the present invention is to provide a method and apparatus for anomaly detection based on a neural network that can be trained using only images of normal, undamaged vehicles, but is capable of detecting random types and ranges of vehicle damage from images acquired in a variety of contexts. The provided method and apparatus are implemented by functional units that are trained to perform anomaly detection using a training data set under a GCAN model, the training data set containing images of normal, undamaged vehicles.

[0006] According to various embodiments, the functional unit includes at least: an image processor configured to segment one or more input vehicle images; a texture encoder configured to generate a texture feature map from the segmented vehicle images; a frame extractor configured to extract a vehicle frame image from the segmented vehicle images; an integral vehicle frame (IVF) reconstructor configured to reconstruct an IVF image from the vehicle frame image extracted by the frame extractor; an image reconstructor configured to reconstruct a complete vehicle image from the IVF image reconstructed by the IVF reconstructor and the texture feature map generated by the texture encoder; an anomaly locator configured to compare a processed vehicle image with its corresponding reconstructed complete vehicle image to detect abnormal areas; a frame encoder discriminator configured to generate a score of likelihood between two IVF feature maps; and an image discriminator configured to generate a score of likelihood between an input vehicle image and a reconstructed vehicle image.

[0007] According to various embodiments, the functional unit is a generative convolutional neural network (generator) trained under a generative contextualized adversarial network (GCAN) model using a training dataset comprising multiple images of an undamaged vehicle, IVF images, and enhanced vehicle frame images of the vehicle.

[0008] In one embodiment, the training of the generator includes extracting an IVF image from an input vehicle image by a frame extractor, wherein the input vehicle image is a real or enhanced image of a normal vehicle; extracting a texture feature map from the input vehicle image by a texture encoder; generating a damage-enhanced vehicle frame image by performing damage enhancement on the IVF image by an image processor; generating a first IVF feature map from the damage-enhanced vehicle frame image by a first frame encoder; reconstructing the IVF image from the first IVF feature map by a frame decoder; reconstructing a complete vehicle image from the reconstructed IVF image and the texture feature map by an image reconstructor; extracting a second reconstructed IVF image from the reconstructed complete vehicle image by a frame extractor; generating a second IVF feature map from the second reconstructed IVF image by a second frame encoder; generating a score of the likelihood between the first IVF feature map and the second IVF feature map by a frame encoder discriminator; and optimizing the parameterization of the frame encoder and the frame extractor of the IVF reconstructor using the score of the likelihood between the first IVF feature map and the second IVF feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the present invention are described in more detail below with reference to the accompanying drawings, in which:

[0010] Figure 1 is a block diagram showing functional units of an apparatus for image-based anomaly detection for vehicle damage assessment according to one embodiment of the present invention;

[0011] Figure 2 Depicts a process of training a GCAN-based generator for image-based anomaly detection for vehicle damage assessment according to one embodiment of the present invention;

[0012] Figure 3 Depicting a process of reasoning by an apparatus according to an embodiment of the present invention for image-based anomaly detection for vehicle damage assessment;

[0013] Figure 4a An exemplary input vehicle image is shown; and

[0014] Figure 4b Anomalies detected in an input vehicle image by a trained GCAN-based generator according to one embodiment of the present invention are shown. DETAILED DESCRIPTION

[0015] In the following description, as a preferred example, a method and apparatus for anomaly detection for vehicle damage assessment using a generative contextualized adversarial network (GCAN) model are proposed. It will be clear to those skilled in the art that modifications including additions and / or substitutions may be made without departing from the scope and spirit of the invention. Specific details may be omitted so as not to obscure the invention; however, the present disclosure is written to enable those skilled in the art to practice the teachings herein without excessive experimentation.

[0016] Figure 1 A block diagram showing functional units of an apparatus for image-based anomaly detection for vehicle damage assessment according to one embodiment of the present invention is depicted. The apparatus may include: an image processor 101 configured to perform various image processing techniques (such as image segmentation and enhancement) on an input image of a vehicle; a texture encoder 102 configured to generate a texture feature map from the vehicle image processed by the image processor 101; a frame extractor 103 configured to extract a vehicle frame image from the vehicle image processed by the image processor 101; an IVF reconstructor 104 configured to reconstruct an IVF image from the vehicle frame image extracted by the frame extractor 103; an image reconstructor 105 configured to reconstruct a complete vehicle image from the reconstructed IVF image generated by the IVF reconstructor 104 and the texture feature map generated by the texture encoder 102; and an anomaly locator 106 configured to compare the processed vehicle image and its corresponding reconstructed complete vehicle image to detect an abnormal area. The IVF reconstructor 104 may include a frame encoder 1041 configured to map the extracted vehicle frame image to generate an IVF feature map, and a frame decoder 1402 configured to remap the IVF feature map back to the IVF image.

[0017] Preferably, the frame encoder 1041 may include a stack of convolutional layers configured to extract multi-scale contextual information represented in the IVF feature map. The frame decoder 1042 may include a stack of deconvolutional layers to reconstruct the complete vehicle frame from the IVF feature map. Skip connections may be used to connect the convolutional layers of the frame encoder 1041 to the deconvolutional layers of the frame decoder 1042, so that the fine details and reconstructed positions of the edges in the vehicle frame at the pixel level can be better preserved.

[0018] In one embodiment, the texture encoder 102, the frame extractor 103, the IVF reconstructor 104, the image reconstructor 105, and the anomaly locator 106 are generative convolutional neural networks (generators) trained under a generative contextualized adversarial network (GCAN) model using a training data set containing multiple images of an undamaged vehicle, IVF images, and enhanced vehicle frame images of the vehicle. The trained generator is then used to perform anomaly detection.

[0019] In the training of the generator, the apparatus further includes: a frame encoder discriminator 107 configured to generate a score of likelihood between two IVF feature maps; and an image discriminator 108 configured to generate a score of likelihood between an input vehicle image and a corresponding reconstructed vehicle image. In one embodiment, the frame encoder discriminator 107 and the image discriminator 108 may be discriminative convolutional neural networks. The frame encoder discriminator 107 and the image discriminator 108 may also be trained via a GCAN model.

[0020] Figure 2 Depicted is a process of training a GCAN-based generator for image-based anomaly detection for vehicle damage assessment according to one embodiment of the present invention. To improve the representativeness of the training dataset, more vehicle image samples are created using image enhancement techniques. For example, a vehicle image can be flipped (mirrored) to create a new image, and then its features are calculated and added to the training dataset. The training process includes:

[0021] S201: Extracting an IVF image D201 from an input vehicle image D200 by the frame extractor 103, wherein the input vehicle image D200 may be a real or enhanced image (without damage) of a normal vehicle;

[0022] S202: The texture encoder 102 extracts a texture feature map D202 from the input vehicle image D200; wherein the texture feature map D202 includes spatial texture feature information of the input vehicle image D200;

[0023] S203: the image processor 101 performs damage enhancement on the IVF image D201 to generate a damage enhanced vehicle frame image D203; wherein in one embodiment, the damage enhancement includes modifying one or more randomly selected regions in the vehicle frame image D201, and wherein the modification includes one or more of random translation, random rotation, random scaling, random line addition, noise addition, partial occlusion, and region replacement by replacing the selected region with a dissimilar region of another vehicle frame image;

[0024] S204: generating a first IVF feature map D204 from the damage-enhanced vehicle frame image D203 by the (first) frame encoder 1041;

[0025] S205: The frame decoder 1042 reconstructs an IVF image D205 from the first IVF feature map D204;

[0026] S206: The image reconstructor 105 reconstructs a complete vehicle image D206 from the reconstructed IVF image D205 and the texture feature map D202;

[0027] S207: extracting a second reconstructed IVF image D207 from the reconstructed complete vehicle image D206 by the framework extractor 103;

[0028] S208: The (second) frame encoder 1041a generates a second IVF feature map D208 from the second reconstructed IVF image D207;

[0029] S209: Generate a likelihood score L201 between the first IVF feature graph D204 and the second IVF feature graph D208 by the frame encoder discriminator 107; and use the likelihood score L201 to optimize the parameterization of the frame encoder 1041 of the IVF reconstructor 104 and the frame extractor 103;

[0030] S210: The image reconstructor 105 reconstructs the vehicle image D210 from the IVF image D201 and the texture feature map D202;

[0031] S211: The image discriminator 108 generates a likelihood score L202 between the input vehicle image D200 and the reconstructed vehicle image D210; and the likelihood score L202 is used to optimize the parameterization of the image reconstructor 105 and the texture encoder 102;

[0032] Alternatively, step S211 may be: generating, by the image discriminator 108 , a score L202 of likelihood between the input vehicle image D200 and the reconstructed complete vehicle image D206 ; and using the score L202 of likelihood to optimize parameterization of the image reconstructor 105 and the texture encoder 102 ;

[0033] S212: The image reconstructor 105 reconstructs a damaged vehicle image D212 from the damage-enhanced vehicle frame image D203 and the texture feature map D202; and

[0034] S213 : Detecting abnormal regions by comparing the reconstructed damaged vehicle image D212 and the reconstructed complete vehicle image D206 by the abnormality locator 106 to generate an abnormal region detection score L203 ; and optimizing parameterization of the abnormality locator 106 using the abnormal region detection score L203 .

[0035] In one embodiment, the (first) frame encoder 1041 and the (second) frame encoder 1041a are the same frame encoder 1041 .

[0036] In order to optimize the parameterization of the generator, the derivative (or gradient value) of the likelihood score with respect to the weight value of the generator is calculated and used to update the weight value. For likelihood scores associated with more than one generator, the chain rule of differential calculus is applied to the calculation, where the gradient value of the likelihood score with respect to the weight value of the generator at the last network layer is calculated, and then used to calculate the gradient value with respect to the weight value of the generator at the penultimate network layer. This calculation is then repeated until the gradient value of each weight value of the relevant generator is obtained. These gradient values ​​are then subtracted from the corresponding weight values ​​to obtain updated weight values ​​to improve the performance of the generator.

[0037] For example, the gradient value of the likelihood score L201 with respect to the weight value of the frame encoder 1041 of the IVF reconstructor 104 is calculated, and then used to calculate the gradient value with respect to the weight value of the frame extractor 103. The gradient value of the likelihood score L202 with respect to the weight value of the image reconstructor 105 is calculated, and then used to calculate the gradient value with respect to the weight value of the texture encoder 102.

[0038] Figure 3 The process of reasoning performed by an apparatus according to an embodiment of the present invention for image-based anomaly detection for vehicle damage assessment is described. The reasoning process may include the following steps:

[0039] S301: The image processor 101 performs image segmentation on one or more images D300 of a vehicle image, and generates one or more segmented vehicle images D301;

[0040] S302: The texture encoder 102 generates a texture feature map D302 for a damaged vehicle from the segmented vehicle image D301;

[0041] S303: extracting a damaged vehicle frame image D303 from the segmented vehicle image D301 by the frame extractor 103;

[0042] S304: The IVF reconstructor 104 reconstructs the IVF image D304 from the damaged vehicle frame image D303;

[0043] S305 : The image reconstructor 105 reconstructs a complete vehicle image D305 from the reconstructed IVF image D304 and the texture feature map D302 .

[0044] S306: Compare the segmented vehicle image D301 with the corresponding reconstructed complete vehicle image D305 pixel by pixel, and generate an abnormal area detection score for each pixel; if the abnormal area detection score of the compared pixel of the reconstructed complete vehicle image D305 is greater than the threshold, the compared pixel is classified as an abnormal area, and the classified abnormal area is marked on the segmented vehicle image D301 by the anomaly locator 106 to generate an abnormally marked vehicle image D306.

[0045] The present invention was experimented. Figure 4a An exemplary input vehicle image is shown; and Figure 4b Anomalies detected in an input vehicle image by the trained GCAN-based generator according to the aforementioned embodiment of the present invention are shown, where the detected anomalies are indicated and surrounded by black outlines.

[0046] In various embodiments, the abnormal regions may be annotated as a binary mask on the corresponding reconstructed complete vehicle image. In some embodiments, the mask is constructed by setting pixels to white.

[0047] In various embodiments, the threshold may have a predefined default value and may be adjusted to have a user-specified value based on various implementation scenarios.

[0048] Implementations according to various embodiments of the present invention are used for anomaly detection in certain types of vehicles. For example, in an implementation for anomaly detection of sedans, the training data set in the training of the GCAN-based generator contains images of normal, undamaged sedans of different manufacturers and models. To give another example, in an implementation for anomaly detection of watercraft, the training data set in the training of the GCAN-based generator contains images of normal, undamaged boats of different categories and manufacturers. Similarly, by including a training data set of images of a specific type of vehicle, implementations for anomaly detection in many different types of vehicles are possible.

[0049] The functional units of the apparatus and methods according to the embodiments disclosed herein can be implemented using computing devices, computer processors or electronic circuit systems, including but not limited to application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and other programmable logic devices configured or programmed according to the teachings of the present disclosure. Based on the teachings of the present disclosure, a person skilled in the art of software or electronics can easily prepare computer instructions or software codes that run in a computing device, a computer processor or a programmable logic device.

[0050] All or part of the methods according to the embodiments may be performed in one or more computing devices, including a server computer, a personal computer, a laptop computer, a mobile computing device such as a smart phone, and a tablet computer.

[0051] Embodiments include computer storage media having stored therein computer instructions or software codes that can be used to program a computer or microprocessor to perform any of the processes of the present invention. Storage media may include, but are not limited to, floppy disks, optical disks, Blu-ray disks, DVDs, CD-ROMs and magneto-optical disks, ROMs, RAMs, flash memory storage devices, or any type of medium or device suitable for storing instructions, codes, and / or data.

[0052] Each functional unit according to various embodiments may also be implemented in a distributed computing environment and / or a cloud computing environment, where all or part of a machine instruction is executed in a distributed manner by one or more processing devices interconnected by a communication network (such as an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media).

[0053] The foregoing description of the present invention has been provided for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art.

[0054] The embodiment was chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. An apparatus for image-based anomaly detection for vehicle damage assessment, comprising: an image processor configured to segment one or more input vehicle images; a texture encoder configured to generate a texture feature map from the segmented vehicle image; a frame extractor configured to extract one or more vehicle frame images from the segmented vehicle images; a full vehicle frame IVF reconstructor configured to reconstruct one or more IVF images from the extracted vehicle frame images; an image reconstructor configured to reconstruct one or more complete vehicle images from the reconstructed IVF image and the texture feature map; as well as an anomaly locator configured to compare the segmented vehicle image with a corresponding reconstructed complete vehicle image of the segmented vehicle image to detect one or more abnormal regions, The texture encoder, the frame extractor, the IVF reconstructor, the image reconstructor and the anomaly locator are GCAN generators trained under a generative contextualized adversarial network (GCAN) model using a training dataset, wherein the training dataset includes multiple images of undamaged vehicles, IVF images and enhanced vehicle frame images of the vehicle.

2. The device according to claim 1, wherein the IVF reconstructor comprises: A frame encoder configured to map the extracted vehicle frame image to generate an IVF feature map; and a frame decoder configured to remap the IVF feature map back to the reconstructed IVF image.

3. The device according to claim 2, wherein the frame decoder comprises a stack of one or more convolutional layers, wherein the stack of one or more convolutional layers is configured to extract multi-scale contextual information represented in the IVF feature map; wherein the frame encoder comprises a stack of one or more deconvolution layers to reconstruct a complete vehicle frame from the IVF feature map; and The convolutional layers of the frame encoder and the deconvolutional layers of the frame decoder are connected via one or more skip connections so as to preserve fine details and reconstructed positions of edges in the complete vehicle frame at a pixel level.

4. The apparatus according to claim 2, further comprising: A frame encoder discriminator configured to generate a likelihood score between two IVF feature maps; as well as An image discriminator is configured to generate a score of likelihood between an input vehicle image and a corresponding reconstructed vehicle image.

5. A method for image-based anomaly detection for vehicle damage assessment, comprising: performing image segmentation on the image of the vehicle image to generate a segmented vehicle image; generating a texture feature map for a damaged vehicle from the segmented vehicle image; extracting a damaged vehicle frame image from the segmented vehicle image; reconstructing an IVF image from the damaged vehicle frame image; reconstructing a complete vehicle image from the reconstructed IVF image; comparing the segmented vehicle image and the reconstructed complete vehicle image pixel by pixel; generating an abnormal region detection score for each pixel, and classifying the compared pixels as an abnormal region if the abnormal region detection score of each compared pixel in the reconstructed complete vehicle image is greater than a threshold; as well as marking the classified abnormal regions on the reconstructed complete vehicle image to generate an abnormality-annotated vehicle image, The method is performed by a GCAN generator trained under a generative contextualized adversarial network (GCAN) model using a training dataset, wherein the training dataset includes multiple images of an undamaged vehicle, an IVF image, and an enhanced vehicle frame image of the vehicle.

6. A method for training a device for image-based anomaly detection for vehicle damage assessment, the method comprising: extracting an IVF image from an input vehicle image by a frame extractor, wherein the input vehicle image is a real or enhanced image of a normal vehicle; extracting a texture feature map from the input vehicle image by a texture encoder; generating a damage-enhanced vehicle frame image by performing damage enhancement on the IVF image by an image processor; generating, by a frame encoder, a first IVF feature map from the damage-enhanced vehicle frame image; reconstructing an IVF image from the first IVF feature map by a frame decoder; Reconstructing a complete vehicle image from the reconstructed IVF image and the texture feature map by an image reconstructor; extracting, by the frame extractor, a second reconstructed IVF image from the reconstructed complete vehicle image; generating, by the frame encoder, a second IVF feature map from the second reconstructed IVF image; Generating, by a frame encoder discriminator, a score of the likelihood between the first IVF feature graph and the second IVF feature graph; as well as The score of the likelihood between the first IVF feature map and the second IVF feature map is used to optimize parameterization of the frame encoder and the frame extractor.

7. The method of claim 6, wherein the lesion enhancement comprises modifying one or more randomly selected regions in the IVF image by one or more of: random translation, random rotation, random scaling, random line addition, noise addition, partial occlusion, and region replacement by replacing one or more of the selected regions with one or more dissimilar regions of another IVF image.

8. The method according to claim 7, further comprising: reconstructing a vehicle image from the vehicle frame image and the texture feature map by the image reconstructor; generating, by an image discriminator, a score of likelihood between the reconstructed vehicle image and the input vehicle image; as well as The score of the likelihood between the reconstructed vehicle image and the input vehicle image is used to optimize parameterizations of the image reconstructor and the texture encoder.

9. The method according to claim 7, further comprising: reconstructing a damaged vehicle image from the damage-enhanced vehicle frame image and the texture feature map by the image reconstructor; detecting an abnormal region by comparing, by an anomaly locator, the reconstructed damaged vehicle image and the reconstructed intact vehicle image to generate an abnormal region detection score; and The anomaly region detection score is used to optimize parameterization of the anomaly locator.

Images