Hierarchical multi-class exposure defect classification in images

Abstract

Embodiments of the present disclosure relate to hierarchical multi-class exposure defect classification in images. Embodiments of the present invention provide systems, methods, and computer storage media for detecting and classifying exposure defects in images using a neural network trained via a limited amount of labeled training images. An image can be applied to a first neural network to determine whether the image includes an exposure defect. A detected defect image can be applied to a second neural network to determine an exposure defect classification of the image. The exposure defect classification can include severe underexposure, moderate underexposure, light underexposure, light overexposure, moderate overexposure, severe overexposure, and the like. The image can be presented to a user with the exposure defect classification.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure generally relate to neural networks, and more specifically to detecting and classifying exposure defects in images using neural networks. BACKGROUND

[0002] As the number of channels (e.g., cell phones, portable cameras, laptops, tablets) on which digital content (images, videos, animations) is created and consumed has exploded, it has become extremely important to efficiently detect and organize defective content before consumption. The increasing use of portable and convenient devices with sensors to capture and store digital media (e.g., cameras) has allowed users to collect large amounts of media. Additionally, various applications (such as, but not limited to Stock, and the like) provide large image repositories for users to consume. In such applications, images can be uploaded by users of the application.

[0003] Digital images captured by both novice and professional photographers can often suffer from exposure defects, such as overexposure and underexposure. As the collection of digital media grows larger, it has become increasingly important to detect such exposure defects in order to maintain some quality for the media collection. However, exposure defect detection often requires a balance between the need for accurate detection and the need for efficiency. Detecting exposure defects in digital media becomes particularly cumbersome when large amounts of media are stored by users or in application repositories. Current exposure defect detection solutions are insufficient to provide such a balance. Often, with existing solutions, users must manually detect media with exposure defects. In some cases, the cost and tedium of manual detection can lead to user frustration or users accepting substandard content, limiting the usability of digital media repositories. SUMMARY

[0004] Embodiments of the present invention relate to an exposure defect classification system to efficiently and accurately detect and classify exposure defects in images using a neural network trained via a limited amount of labeled training data. In this regard, an image can be applied to a trained first neural network to determine whether the image includes an exposure defect. Then, if the image is determined to include an exposure defect, the image can be applied to a second neural network. The second neural network can be trained to classify the image as including an overexposure defect or an underexposure defect. In some embodiments, the second neural network can output an exposure score indicative of an underexposure level and / or an overexposure level in the input image. The levels can correspond to exposure defect classification types such as, but not limited to, moderate overexposure, severe overexposure, mild overexposure, moderate underexposure, severe underexposure, and / or mild underexposure. Then, the images with exposure defects can be presented and / or stored based on the classification and / or score.

[0005] As such, a user can be efficiently, effectively, and automatically provided with relevant information regarding images that have exposure defects. This enables the user to easily delineate relevant information regarding the underlying images without having to visually search through an entire image gallery to find images that can suffer from exposure defects, thereby allowing the user to efficiently remove and / or repair defective images.

[0006] This summary of the invention is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features of the claimed subject matter, nor is it meant to be used in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0007] The present invention is described in detail below with reference to the attached drawing figures, wherein:

[0008] Figure 1 is a block diagram of an exemplary environment suitable for implementing embodiments of the present invention according to embodiments of the present invention;

[0009] Figure 2 is a data flow diagram illustrating an example process for exposure defect classification of images using a neural network according to embodiments of the present invention;

[0010] Figure 3 illustrates an example process for training a binary model according to embodiments of the present invention;

[0011] Figure 4 illustrates an example output of a defect detection system according to embodiments of the present invention;

[0012] Figure 5is a flowchart illustrating a method for detecting and classifying exposure defects in images, in accordance with an embodiment of the present application; and

[0013] Figure 6 is a block diagram of an exemplary computing device suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0014] SUMMARY

[0015] The proliferation in the number of channels on which digital media can be created and consumed has significantly increased over the years, resulting in a large amount of digital content. For example, digital media, such as images, can be captured by various electronic devices having sensors, including digital cameras, drones, camcorders, and the like. The captured digital media, such as images, videos, and the like, can be stored in repositories, such as digital photo albums, Stock, and the like.

[0016] However, such captured digital media often suffers from exposure defects, such as overexposure and underexposure. Specifically, an image suffers from overexposure when more light than needed falls on the sensor that captures the digital image, such as when the captured object is too bright. Overexposure results in loss of details in the digital image. As such, an overexposed image can often include bright or white region(s) that encounter too much light. Further, underexposure occurs when less light than needed falls on the sensor that captures the digital image, such that the object appears very dark in the digital image. Underexposure can result in loss of details corresponding to the object. As a result, an underexposed image can often include dark or black region(s) that encounter little light.

[0017] When exposure defects occur in association with digital media, it can be valuable for a user to detect digital media having such exposure defects to enable a more optimal and efficient user experience, especially as the amount of digital content increases. As one example, a large amount of digital media content can be accessible via a digital photo album (e.g. Stock). A user can desire to access and / or utilize images in the digital photo album that have good exposure, such that the amount of details in the images is optimal for a viewer. Accordingly, an individual or entity can selectively remove or flag digital media having poor exposure to increase or improve the quality of digital media accessible via the digital photo album.

[0018] Currently, to detect exposure defects in digital media, one approach requires a user to manually detect media having exposure defects. For example, this conventional approach requires a user to manually determine which images or frames (e.g., video frames, animation frames) include exposure defects, and to flag or delete those images or frames. As the collection grows, this manual detection can become increasingly cumbersome. Further, this approach requires a user to arbitrarily detect exposure defects in images or frames. This arbitrary manual detection can often be subjective and result in incorrect classification of the media.

[0019] Another conventional approach includes using image editing software to detect exposure defects in each image or frame using a histogram. In this regard, the image editing software can generate a histogram representing the tonal values of a digital image or frame in a graph. Specifically, the histogram represents the amount of tones of a particular brightness in the image or frame, which ranges from black to white as the brightness increases. For example, an overexposed image or frame can include a graph having high white tonal values. Similarly, an underexposed image can be represented in a graph having high black tonal values. A user of the image editing software can perform a histogram analysis on each digital image or frame to determine whether there is an exposure defect in any of the images / frames. However, this individual analysis for each image or frame can be time consuming. Further, the tonal values in the histogram can also be affected by sensor settings, the scene being photographed, the user's vision, etc. As such, the histogram can not always provide an accurate analysis of the exposure of the image, and often results in incorrectly classifying the image or frame as good, overexposed, and / or underexposed.

[0020] Accordingly, embodiments described herein relate to supporting efficient and accurate detection and classification of exposure defects for digital images. In particular, embodiments described herein generally detect and classify exposure defects without relying on tonal histograms or extensive manual classification and / or flagging. To this end, embodiments relate to using a trained neural network to automatically detect and classify exposure defects in digital media (e.g., images, video frames, animation frames). Generally, the automatically classified digital media is accurately classified as including well-exposed media, overexposed media, underexposed media, etc. This classification can then be used, for example, to improve the quality of the images, curate the images, organize the presentation of the images, etc.

[0021] In implementations, the neural network(s) first determine whether an image includes an exposure defect, and if the image includes an exposure defect, the same or different neural network(s) determine an exposure defect classification based on the level of the exposure defect in the image. At high levels, in embodiments, the neural network(s) are used to detect exposure defects and provide their classification in the image. The neural network(s) can be trained and used to predict exposure levels (e.g., overexposed, underexposed, well-exposed) for each digital image, which can be used to classify images stored in a device, in the cloud, and / or in a digital media repository. The present system can classify images into different exposure levels, such as severely overexposed, lightly overexposed, moderately overexposed, severely underexposed, lightly underexposed, moderately underexposed, well-exposed (e.g., well-exposed, no exposure defects), etc. In contrast to conventional solutions that require a user to perform histogram analysis or subjective classification of each image, the present system automatically uses the neural network(s) trained on a limited amount of manually labeled training data and / or noisy data to predict exposure defects represented in the images. The present system allows for a more robust and efficient way of predicting exposure defects within digital images, which can then be used to perform various functions, such as organizing digital images by exposure level, deleting defective images, flagging defective images, changing the exposure of defective images, etc.

[0022] In some examples, for any received media (e.g., images, videos, animations), a first neural network can be used to determine whether the media includes an exposure defect. The neural network can be trained to classify one or more frames (e.g., images, video frames) of the media as well- or poorly-exposed frames or images. Then, the frames or images determined to include an exposure defect (e.g., poorly-exposed frames) can be applied to a second neural network to determine an exposure level for each image. The exposure level can include an exposure score and / or an exposure defect classification for each frame or image determined to include an exposure defect. For example, the score for each frame or image can be between 0 and 1, where 0 indicates a severely underexposed level and 1 indicates a severely overexposed level. In such examples, the exposure defect classification can include severely overexposed, lightly overexposed, moderately overexposed, severely underexposed, lightly underexposed, moderately underexposed, etc. In this way, each frame or image can be classified as well-exposed (e.g., well-exposed) if the corresponding frame is determined to not include an exposure defect and classified as one of the above exposure defect classifications.

[0023] The frames or images can then be presented to a user based on the classifications. The classifications can be used to perform various functions, such as organizing the digital frames or images by exposure level, deleting defective images, flagging defective images, changing the exposure of defective images, etc. In one example, a user can be presented with defective images or frames and / or corresponding exposure levels. The user can be presented with options regarding removing / deleting one or more of the defective frames or images, fixing the exposure in one or more of the defective frames or images, etc. In other examples, the defective frames or images can be presented with different labels or flags based on the exposure level. In some other examples, a user can be presented with a robust graphical representation of the number of frames or images detected as including exposure defects in a set of images. For example, a ratio of well-exposed frames or images to defective frames or images can be presented. A ratio of well-exposed classifications to other classifications can also be presented.

[0024] In some examples, the first neural network can be trained using a weakly supervised learning algorithm such that a small set of labeled training images can be used to generate more training images from a noisy dataset. Initially, the neural network can be initially trained using the small set of labeled images to detect whether an image includes an exposure defect. The initially trained neural network can then be retrained using the noisy dataset of images by determining an inferred score for each image in the noisy dataset using the trained neural network. The inferred scores of the noisy dataset can then be used to determine a threshold score for classifying each image as a well-exposed or defective image. The threshold score can be determined based on a distribution ratio of well-exposed images to defective images in the labeled training images such that the threshold score provides the same distribution ratio of well-exposed images to defective images in the noisy dataset. Images below the threshold score can be classified as well-exposed images, while images above the threshold score can be classified as defective images. In some examples, one or more misclassified images can be removed from the noisy dataset. Misclassified images can be images that were misclassified as well-exposed images in the defective image classification. The new noisy dataset with the classifications can then be used to retrain the neural network. The retrained neural network can then be used as the first neural network at inference time to detect whether an image or frame includes an exposure defect. In this way, the first neural network can be trained using a larger dataset without manually labeling each image in the noisy dataset.

[0025] In some examples, the first and second neural networks can share a plurality of top layers, but include one or more bottom layers (e.g., fully connected layers) trained for binary defect detection and defect detection levels, respectively. Each network can be trained with different learning rates (e.g., weights) for each layer. In some examples, the fully connected layer(s) of the second neural network can be trained using training images labeled with five classifications or classification scores from 0 to 1, including 0, 0.25, 0.5, 0.75, and 1. The classifications can respectively indicate exposure levels such as severe underexposure, mild underexposure, moderate overexposure / underexposure, mild overexposure, and severe overexposure. In this way, the first neural network can be trained to detect the presence of exposure defects in an image, and the second neural network can be trained to detect the level of exposure defects in the image.

[0026] Aspects of the technology disclosed herein provide a number of advantages over previous solutions. For example, one previous method involves manually labeling media. However, this method does not account for the subjective bias of the labeler to intelligently and consistently determine images that suffer from exposure defects in an automated and accurate manner. As the collection becomes larger and the misclassification of images that suffer is greater, such manual detection or determination becomes increasingly cumbersome, failing to provide accurately labeled images. Further, another previous method requires a user to perform a histogram analysis on each image, making the process time consuming. This previous method is also constrained by the assumption that a histogram always provides an accurate analysis of the exposure of an image and can often result in incorrectly classifying an image as a good, overexposed, and / or underexposed image.

[0027] To avoid such constraints on the present system, for example, embodiments of the technology described herein systematically develop an algorithm to use one or more neural networks to detect and classify exposure defects in images to generate an automated, consistent, and accurate classification of images without the need for manually labeling large amounts of training data (e.g., training images). Unlike conventional systems, embodiments of the present technology utilize a small dataset of labeled training images to generate a large dataset of classifications or labels from a noisy dataset of images to train a neural network to detect the presence of exposure defects in images. Further, embodiments of the present technology utilize the results of the trained neural network to detect the level of defects in images by applying the defective images to another neural network. Embodiments of the present technology can provide accurate and efficient determination and classification of underlying exposure defects for digital media by using neural networks that are specifically trained using limited training data to automatically detect exposure defects and / or their corresponding levels in digital media.

[0028] Digital media exposure defect detection environment

[0029] Reference is now made toFigure 1 A block diagram of an exemplary environment 100 suitable for implementing embodiments of the present application is shown. Generally, the environment 100 is suitable for supporting exposure defect detection in digital media, and among other things, supports exposure defect classification of digital media using a neural network(s) trained to detect and classify exposure defects in digital media.

[0030] The environment 100 includes a network 120, a client device 110, a data store 140, and a defect detection system 130. In Figure 1 the illustrated embodiment, the client device 110 includes a digital media gallery 115. Generally, the digital media gallery 115 includes a collection of media. As described herein, in some embodiments, the digital media gallery 115 can present an exposure-based organization of the collection of media. The client device 110 can be any kind of computing device capable of collecting and / or presenting media, such as via an exposure-based media organization. For example, in embodiments, the client device 110 can be a computing device, such as the computing device 600, described below with reference to Figure 6 In embodiments, the client device 110 can be a personal computer (PC), a laptop computer, a workstation, a mobile computing device, a PDA, a cellular telephone, etc. The components of the environment 100 can communicate with one another via the network 120, which can include, without being limited to, one or more local area networks (LANs) and / or wide area networks (WANs). Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.

[0031] Generally, in response to obtaining a collection of media in a digital media gallery, detection and determination of an exposure classification for each media can be performed using an algorithm to predict an exposure defect type with a neural network(s) trained using a limited amount of labeled data. By employing such an algorithm of the present disclosure to detect and classify exposure defects in media, the collection of media can be classified in a contextual and intelligent manner in order to provide a user with relevant information about the exposure of the underlying media.

[0032] Digital media library 115 can be incorporated into or integrated into an application, or added as an add-on or plugin to an application. The application can generally be any application capable of supporting digital media organization, and can be a standalone application, a mobile application, a web application, etc. In some implementations, the application includes a web application that can run in a web browser and can be at least partially hosted on a server side. Additionally or alternatively, the application may include a dedicated application. In some cases, the application can be integrated into an operating system (e.g., as a service). Although digital media library 115 is generally discussed in association with an application herein, in some cases, digital media library 115 or a portion thereof may be additionally or alternatively integrated into an operating system (e.g., as a service) or a server (e.g., a remote server). In some examples, digital media library 115 may be stored in a cloud computing environment.

[0033] The digital media library 115 of client device 110 may include a user interface or application that allows users to store or submit media. Any suitable user interface and input components may be used, such as those relative to... Figure 6 The I / O component 620 describes the user interface and input components. Various quantities and / or types of media can be received for exposure defect detection. For example, the digital media library 115 can accept images, videos, animations, slideshows, etc. In some embodiments, the digital media library 115 can receive a collection of media for exposure defect detection. Generally, the digital media library 115 communicates with the defect detection system 130 via network 120 and can send media or media collections from client device 110 to the defect detection system 130 for processing. Although the media collection is described as being sent to the defect detection system 130, this is not necessary, as any or all components of the defect detection system 130 can be located on client device 110, data storage 140, or elsewhere. Furthermore, the media collection can be stored in association with the defect detection system 130 and accessible via a digital media library application operating on the client device.

[0034] Data repository 140 can store a variety of media and corresponding exposure defect classifications. In some examples, data repository 140 can include a repository of classification types (e.g., good exposure, severe overexposure, mild overexposure, moderate overexposure, severe underexposure, mild underexposure, moderate underexposure, etc.) and their corresponding characteristics (e.g., scores, score ranges) with respect to the classification types. In some examples, data repository 140 can also include training data sets used to train one or more neural networks of defect detection system 130, such as but not limited to binary model 134 and classification model 136. The training data sets can include labeled data sets, unlabeled data sets, noise data sets, retraining data sets, etc. Each classification type can be associated with a corresponding score and / or score range. For example, a severe underexposure classification can be associated with an exposure score of 0 (e.g., a threshold score) and / or an exposure score range of 0-0.25, a moderate underexposure classification can be associated with an exposure score of 0.25 and / or an exposure score range of 0.25-0.5, a mild underexposure classification can be associated with an exposure score of 0.5 and / or an exposure range of 0.45-0.5, etc. In one example, the classification types can be associated with an exposure score between 0 and 1, such that a score closer to 0 indicates severe underexposure, while a score closer to 1 indicates severe overexposure.

[0035] Defect detection system 130 is generally configured to detect and classify exposure defects. Generally, and at a high level, defect detection system 130 obtains (e.g., receives or acquires) a medium or a set of media and determines an exposure defect classification for each medium that optimally represents the exposure levels encountered in the medium. Defect detection system 130 can receive an image(s) or frame(s) (e.g., video frames, animation frames) from digital media gallery 115 of client device 110. In some examples, defect detection system 130 can receive the image(s) from data repository 140. In some examples, defect detection system 130 or a portion thereof can be part of digital media gallery 115. In other examples, defect detection system 130 can be located on a remote server, such that digital media gallery 115 can communicate with defect detection system 130 via network 120. In some examples, defect detection system 130 can be configured to receive a plurality of images or frames from digital media gallery 115 and determine a plurality of exposure defect classifications for the plurality of images or frames. Figure 1 In the embodiment illustrated in FIG. 1, defect detection system 130 includes image receiver 132, binary model 134, classification model 136, and presenter 138. Although the examples presented herein generally relate to images, this is for example purposes only. For example, the present system can similarly function to detect exposure defects in other forms of digital media, such as but not limited to videos, animations, etc.

[0036] The defect detection system 130 is generally configured to detect and classify images based on detected exposure levels in each image. Specifically, the defect detection system 130 can determine whether an image suffers from an exposure defect and / or a type (e.g., level) of exposure defect in the image. The defect detection system 130 scores the image using a neural network(s) to indicate an exposure level in the image. The defect detection system 130 can use two neural networks to first determine whether an image suffers from an exposure defect and, in response to determining that an image includes an exposure defect, then determine an exposure defect level in the image and classify the image into an exposure defect type. The exposure defect level can be used to determine a classification of the image to present to the client device 110.

[0037] The image receiver 132 can receive, acquire, access, or obtain images or sets of images (“image(s)”) from, for example, the digital media gallery 115 of the client device 110. In some embodiments, the received image(s) can correspond to images, frames representative of a video (e.g., a snapshot of a video), and / or sensor data representative of a representation of a sensor’s sensing field (e.g., a depth map or point cloud for a LIDAR sensor, a value graph for an ultrasonic sensor, etc.). The images can be in any of a variety of formats, such as, for example and without limitation, compressed images such as in Joint Photographic Experts Group (JPEG) or luminance / chrominance (YUV) formats, compressed images like frames derived from a compressed video format such as H.264 / Advanced Video Coding (AVC) or H.265 / High Efficiency Video Coding (HEVC), raw images such as originating from a Red-White-Blue (RCCB), Red-White (RCCC), or other type of imaging sensor, and / or other formats. Additionally, in some examples, the image(s) can be in a raw or captured format (e.g., without any pre-processing, while in other examples, the image(s) can undergo pre-processing (e.g., noise balancing, demosaicing, scaling, cropping, enhancement, white balancing, tone curve adjustment, etc., such as using a sensor data pre-processor (not shown)). For example, the image(s) can be resized (e.g., smaller, larger) to a predetermined size. In other examples, the image(s) can be enhanced by rotating or flipping the image by varying the degrees (e.g., randomly selected, selected based on a random analysis). In other examples, the image(s) can undergo center cropping. As used herein, the image(s) can refer to unprocessed images, pre-processed images, or a combination thereof. The image(s) can include initial images (e.g., as captured by one or more image sensors), downsampled images, upsampled images, cropped or region of interest (ROI) images, otherwise enhanced images, and / or combinations thereof.

[0038] After receiving or obtaining the image(s), each image can then be applied to the binary model 134. The image(s) can be applied to the binary model as unprocessed image(s) and / or preprocessed image(s). For example, the image(s) can be resized images that are resized (e.g., smaller, larger) to a predetermined size. In other examples, the image(s) can be augmented images generated by rotating or flipping the image by varying the degrees (e.g., randomly selected, selected based on a random analysis). In other examples, the image(s) can be center crops of the initial image, which can be of a predetermined size. In another example, the image(s) can be the initial image resized to a smaller size, and then a center crop of the resized image can be used as the image(s). The image(s) can be preprocessed to the size of the training images used to train the binary model 134 and / or the classification model 136. For example, if the binary model 134 is trained using images that are resized or re-sized to a predetermined size (e.g., 224 length by 224 width), the image(s) can also be the initial image resized to the same or similar size as the training images. It should be noted that the preprocessing techniques discussed herein are for example purposes only, and any other preprocessing techniques can be used to generate the preprocessed image(s). In some examples, each pixel of the image(s) can be associated with a value(s) corresponding to an amount of brightness, light, and / or contrast encountered by that pixel.

[0039] The binary model 134 is generally configured to detect whether the image(s) include or suffer from exposure defects. The binary model 134 can include a machine learning model trained to score the images based on a probability or confidence that the image includes an exposure defect. The binary model 134 can then classify the images based on the confidence score output by the machine learning model. In some examples, the machine learning model of the binary model 134 can be trained to classify the images as well-exposed images and defective images as output. Each of the image(s) can be classified as a well-exposed or defective image using the binary model 136 based on a corresponding confidence score associated with the image. A well-exposed image classification can indicate that the image includes a good or proper exposure and does not include an exposure defect. A defective image classification can indicate that the corresponding image includes an overexposure or overexposed image defect. Specifically, an image can be classified as a defective image when more or less light than needed falls on the sensor that captures the digital image, such as when the captured object is too bright. In some examples, the binary model 136 can be trained to detect bright (e.g., white) areas and dark (e.g., black) areas in the input image to determine whether the image includes an exposure defect (e.g., underexposure, overexposure). A pixel-by-pixel analysis can be done by the binary model 134 to detect the amount of exposure (e.g., light, brightness, contrast) in the image.

[0040] To classify the images as well-exposed or defective images, a confidence score can be determined by the binary model 134 for the input image(s). The confidence score can indicate a probability that the corresponding image includes an exposure defect (e.g., underexposure, overexposure). For example, the confidence score can be in a range of 0 to 1, such that an increase in the score corresponds to an increased probability that the image includes an exposure defect. In such an example, a confidence score closer to 0 can indicate that the image is well-exposed or includes a good exposure, while a confidence score closer to 1 can indicate that the image is defective, or vice versa.

[0041] The binary model 134 can use a predetermined confidence threshold to classify the images as well-exposed images or defective images. The confidence score can be correlated to a confidence that the image(s) are defective, such that the higher the confidence score of an image, the more likely the image is defective, or vice versa. For example, the confidence score of an image can be between 0 and 1, where images with a confidence score below a predetermined confidence threshold of 0.5 are determined to be well-exposed images and images with a confidence score equal to or above the predetermined confidence threshold of 0.5 are considered to be defective images, or vice versa.

[0042] The binary model 134 can determine a classification of the image(s) based on a corresponding confidence score generated by the machine learning model using a confidence score threshold and / or a confidence score range. The binary model 134 can apply the image(s) as input to the trained machine learning model, which outputs a confidence score and / or a corresponding classification for each of the image(s) based on a pixel-by-pixel analysis of the image(s). The confidence score can be based on a feature analysis of the image, including a pixel-by-pixel analysis, that detects an amount of light, brightness, and / or contrast associated with each pixel of the image. An image that includes a set of pixels associated with a dark region or a bright region, as indicated by the pixels in the region, can be associated with a higher confidence score than an image that has an appreciable amount of light or brightness in all regions, or vice versa. The confidence score can then be used by the machine learning model or a separate component of the binary model to classify the image as an exposure good image or a defective image, as described above.

[0043] In alternative embodiments, a feature-based algorithm can be used as the binary model 134 to classify the image(s) and / or generate a corresponding confidence score for the image(s). A feature-based algorithm refers to an algorithm that is trained to detect exposure features in an image. Exposure features can include light, brightness, and / or contrast in a region of the image, which can then be associated with a confidence score that indicates a probability that the image includes an exposure defect. The confidence score can then be used by the feature-based algorithm or a separate component to classify the image as an exposure good image and a defective image based on a confidence threshold and / or range as described above.

[0044] According to the description provided with respect to Figure 3 The binary model 134 can be trained using a training dataset and / or a noise dataset, according to the description provided. The binary model 134 can be trained to detect or predict whether the image(s) are exposure good images or defective images. In some examples, the binary model 134 can provide a classification and / or a confidence score corresponding to the image(s) to the presenter 138 for presentation to the client device 110. In other examples, the binary model 134 can be further configured to communicate or send the image(s) corresponding to the predicted defective images to the classification model 136.

[0045] Although examples are described herein with respect to using neural networks and specifically convolutional neural networks (CNNs) or deep neural networks (DNNs) as machine learning model(s) (e.g., binary model 134, classification model 136), this is not intended to be limiting. For example, and without limitation, machine learning model(s) of the present disclosure can include any type of machine learning model(s), such as machine learning model(s) that use linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k- nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptron, long / short term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machine, etc.), computer vision algorithms, and / or other types of machine learning models.

[0046] Classification model 136 is generally configured to classify defect images based on exposure levels in the image(s). Classification model 136 can include a machine learning model that is trained to score images based on exposure defect levels detected in the image(s) using a regression algorithm. Classification model 136 can then classify images based on exposure defect levels in the image(s). In some examples, the machine learning model of classification model 136 can be trained to classify images. Each image in the image(s) can be classified as an exposure defect classification, such as, but not limited to, severe underexposure, moderate underexposure, mild underexposure, mild overexposure, severe overexposure, severe overexposure, etc., using classification model 136. In some examples, each image in the image(s) can be classified as an underexposed or overexposed classification using classification model 136. Generally, an exposure defect classification refers to a type or classification of exposure based on the degree or level of exposure.

[0047] To classify images with respect to exposure defects, an exposure score can also be determined by classification model 136 for the defect image(s). An exposure score can be indicative of an exposure level in a corresponding image. An exposure score can represent both an overexposure defect or an underexposure defect in an image. For example, an exposure score can be in a range of 0 to 1, such that an increase in the score corresponds to an increase in the exposure level. In such an example, an exposure score closer to 0 can indicate that an image is severely underexposed, while an exposure score closer to 1 can indicate that an image is severely overexposed, or vice versa. In some examples, if a corresponding exposure score of an image is below 0.5, the image can be classified as an underexposed image, and if the corresponding exposure score of an image is above 0.5, the image can be classified as an overexposed image, or vice versa.

[0048] The classification model 136 can determine an exposure defect classification for the image(s) based on the corresponding exposure score generated by the machine learning model using an exposure score threshold and / or exposure score range. The classification model 136 can apply the defect image(s) to the trained machine learning model, which outputs an exposure score and / or a corresponding exposure defect classification for each of the defect image(s). The exposure score can be related to the exposure level of the image such that the higher the exposure score for an image, the more exposure is included in the image. The exposure score can be based on a feature analysis of the image, including a pixel-by-pixel analysis, that detects an amount of light, brightness, and / or contrast associated with each pixel of the image. An image that includes a set of pixels associated with bright or white regions (as indicated by the pixels in the regions) can be associated with a higher exposure score than an image that has dark or black regions, or vice versa. For example, the exposure score for an image can be on a scale between 0 and 1, where an image with an exposure score lower than 0.25 is classified as including a severe underexposure, an image with an exposure of 2.5 to 0.4 is classified as including a moderate underexposure, an image with an exposure score of 0.4 to 0.5 is classified as including a light underexposure, an image with an exposure score of 0.5 to 0.6 is classified as including a light overexposure, an image with an exposure score of 0.6 to 0.75 is classified as including a moderate overexposure, and an image with an exposure score higher than 0.75 is classified as including a severe overexposure.

[0049] In some non-limiting examples, separate machine learning models can be trained to score images for underexposure and overexposure defects. In such examples, one machine learning model can be trained to score defect images between 0 and 1 for underexposure, where an exposure score of 0 indicates a severe underexposure and a score of 1 indicates a light underexposure, or vice versa. Similarly, another machine learning model can be trained to score defect images between 0 and 1 for overexposure, where an exposure score of 0 indicates a severe overexposure and a score of 1 indicates a light overexposure, or vice versa. In some other examples, a single machine learning model can be trained to score images for both underexposure and overexposure differently. It should be noted that the above exposure score ranges for exposure defect classification are provided for example purposes only, and other ranges and thresholds can similarly be used to classify images into exposure defect classifications.

[0050] A machine learning model or feature-based algorithm can be used as classification model 136 to classify images(s) into exposure defect categories and / or generate corresponding exposure scores for images(s). In some examples, the machine learning model or feature-based algorithm can output exposure scores for images(s), and another component of the defect detection system 130 can use the exposure scores to generate corresponding classifications for images(s) based on the exposure scores.

[0051] According to relative to Figure 2 As described, classification model 136 can be trained using a relatively small labeled training dataset. The training dataset used to train classification model 136 can be a set of images labeled with corresponding exposure scores and / or exposure defect classifications. Classification model 136 can be trained to detect or predict the type of exposure defect in (multiple) images.

[0052] In some examples, classification model 136 may provide renderer 138 with exposure defect classifications and / or exposure scores corresponding to the images(s) for presentation to client device 110. In some examples, classification model 136 may provide exposure scores to renderer 138, and renderer 138 may then classify the image as an exposure defect category based on the classification scores. Renderer 138 may use the exposure defect classifications and / or exposure scores to then present the image in some manner as described below. In some examples, binary model 134 and classification model 136 may be combined into a single model or neural network to detect defective images and classify them as well-exposed images and exposure defect categories. In some examples, a single model may be trained to score images between 0 and 1, where an exposure score of 0 indicates an image is underexposed (e.g., severely underexposed), an exposure score of 0.5 indicates an image is well-exposed, and an exposure score of 1 indicates an image is overexposed (e.g., severely overexposed), or vice versa. It should be noted that other scoring criteria may also be used to score images. The binary model 134, the classification model 136, and / or a combination of individual models can be stored in the data repository 140 and can be accessed when an image is received for classification.

[0053] Presenter 138 is generally configured to receive or obtain exposure defect classifications and / or exposure scores from classification model 138 to induce presentation to a user of client device 110. Multiple images may be presented to the user via the user interface of client device 110 along with their corresponding exposure defect classifications. In some examples, exposure defect classifications may be used by presenter 138 to perform various functions, such as organizing multiple images in digital media library 115 by exposure level or exposure defect classification. In this regard, multiple images may be presented to the user of client device 110 in combination with their corresponding exposure defect classifications and / or exposure defect scores. In some other examples, presenter 138 may be configured to remove defective images and present only good images in digital media library 115. In this case, presenter 138 may receive the classifications and / or corresponding confidence scores of multiple images from binary model 134, and then the multiple defective images may be removed based on the multiple confidence scores or classifications. If available, images (multiple) can be presented to the user of client device 110 by combining one or more of the corresponding confidence score, classification, exposure defect score and / or exposure defect classification.

[0054] In some other examples, the defective images(s) may be labeled based on corresponding exposure defect categories to indicate to the user that the images are defective and / or include the corresponding exposure defect category. In yet another example, the renderer 138 may be configured to automatically change or repair the exposure of(s) of(s) images, at least in part, based on corresponding exposure scores and / or exposure levels. Further, the renderer 138 may be configured to provide the user with optional options via the user interface of the client device 110 regarding removing / deleting one or more defective images(s), repairing the exposure of one or more defective images(s), etc. In some other examples, the renderer 138 may be configured to present to the user a robust graphical representation of the number of(s) images(s) detected as including exposure defects in a set of images. For example, the ratio of well-exposed frames or images to defective frames or images may be presented. The ratio of well-exposed categories to other categories may also be presented. In this way, information about exposure defects in(s) images can be presented to the user in an efficient, accurate, and user-friendly manner.

[0055] Turn Figure 2 , Figure 2 An exemplary data flow diagram is illustrated, illustrating an example process 200 for classifying exposure defects in an image using a binary model 210 and a classification model 230 according to an embodiment of the present invention. Binary model 210 (such as...) Figure 1binary model 134) can be trained to receive an image as input and output a prediction classifying the image as either a well-exposed image 222 or a defective image 224. The binary model 210 can be a neural network (e.g., DNN, CNN) that includes multiple convolutional layers and fully connected layers. In some examples, the binary model 210 can include three parts, including a residual neural network layer 212, a custom layer 214, and a fully connected layer 216. The residual neural network layer 212 can be any known convolutional layer of a residual neural network and can include multiple layers (e.g., 34, 50, 60, etc.). The custom layer 214 can include one or more layers to connect the residual network layer 212 to the fully connected layer 216. However, any known custom layer can be used.

[0056] The fully connected layer 216 of the binary model 210 can be trained to classify an image as either a well-exposed image 222 or a defective image 224. To do so, the binary model 210 can be trained to predict a confidence score for each image and classify the image based on whether the corresponding confidence score is below or above a predetermined confidence threshold 220. In some examples, the predetermined confidence threshold 220 can be 0.5 and the binary network 210 can be trained to predict a confidence score for each image that is between 0 and 1. Thus, then, images having a confidence score below the predetermined confidence threshold can be classified as well-exposed images and images having a confidence score equal to or above the predetermined confidence threshold can be classified as defective images.

[0057] The image(s) classified as defective images 224 by the binary model 210 can then be applied to a classification model 230, such as but not limited to Figure 1 the classification model 136 of FIG. 1 to determine an exposure score 240 for the image(s) and / or classify the defective images into exposure defect categories. The classification model 230 can be trained to output an exposure score 240 for each defective image based on inputting the defective image 224. The classification model 230 can be a neural network (e.g., DNN, CNN) that includes multiple convolutional layers and fully connected layers. In some examples, the classification model 230 can include three parts, including a residual neural network layer 212, a custom layer 214, and a fully connected layer 236. The classification model 230 can include the same residual neural network layer 212 and custom layer 214 as the binary model 210. However, the fully connected layer 236 of the classification model 230 can be trained to detect or determine exposure levels in the defective images.

[0058] The classification model 230 can be trained to predict an exposure score 240 for each image that indicates an exposure level of the image. In some examples, the fully connected layer 236 of the classification model 230 can be trained using a regression algorithm with defect images labeled into five categories (e.g., category 242, category 244, category 246, category 248, and category 250) between 0 and 1. In this way, the classification model 230 can be trained to classify an input defect image into the five categories, each category or score indicating an exposure level from severely underexposed to severely overexposed as the score increases from 0 to 1. The defect image 224 can then be classified based on the proximity of the corresponding exposure score to one of the five categories. In some examples, the classification model 230 can also be trained to classify images into exposure defect categories. In other examples, the exposure score output by the classification model 230 can then be used to classify the images separately from the model. Category 242 can indicate severely underexposed, category 244 can indicate moderately underexposed, category 246 can include lightly underexposed or lightly overexposed, category 248 can indicate moderately overexposed, and category 250 can indicate severely overexposed. For example, a defect image in the defect image 224 with an exposure score of 0.1 can be classified as including severely underexposed because the exposure score 0.1 is proximate to category 242 (i.e., the closest category). Similarly, a defect image with an exposure score of 0.8 can be classified as including moderately overexposed because it is proximate to category 248. In this way, images with exposure defects can be automatically detected and classified in an accurate and efficient manner by utilizing a trained neural network (e.g., binary model, classification model).

[0059] In some examples, the binary model 210 and the classification model 230 can each be trained at different learning rates for different layers of the network. In such examples, the learning rate increases from the top layers (e.g., residual neural network layers 212) to the bottom layers (e.g., fully connected layers 216, fully connected layers 236, layers closer to the error function). For example, the first twenty layers of the model can be trained at a lower learning rate than the next twenty layers, which can be trained at a lower learning rate than the final layers of the network.

[0060] Referring now to Figure 3 , Figure 3 FIG. 1 illustrates an example process 100 for training a binary model 104 according to embodiments of the present disclosure. The binary model 104 can be a neural network trained using a weakly supervised learning algorithm. The binary model 104 can be a neural network, such as but not limited to a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM) network, a deep neural network (DNN), a generative adversarial network (GAN), or any combination thereof. Figure 1The clean dataset 302 can be used as ground truth data to initially train the binary model 304. The clean dataset 302 can include a relatively small dataset of images, each labeled as either an exposure good image (e.g., good image) or a defect image (e.g., bad image). The clean dataset 302 can be relatively smaller (e.g., include a smaller number of images) than a noisy dataset (e.g., unlabeled dataset of images) used to fully train a neural network, such as the noisy dataset 306. A distribution ratio of exposure good images to defect images in the clean dataset 302 can be obtained or determined. The noisy dataset 306 can then be applied to the initially trained binary model 304 to classify the images in the noisy dataset 306 as either exposure good images and defect images. The noisy dataset 306 can include a larger dataset of unlabeled images. The initially trained binary model 304 can predict a confidence score (e.g., inference score) for each image in the noisy dataset 306. The inference scores of the noisy dataset 304 can then be used to determine a threshold score for classifying each image in the noisy dataset 304 as either an exposure good or a defect image. The threshold score can be determined based on the distribution ratio in the clean dataset 302 such that the threshold score provides the same distribution ratio of exposure good images to defect images in the noisy dataset 306 as the clean dataset 302. Images in the noisy dataset 306 below the threshold score can be classified as exposure good images and images above the threshold score can be classified as defect images. The classified images in the noisy dataset 306 can be determined as the final dataset 308 to use as ground truth data for retraining the binary model 304 as an exposure defect image.

[0061] In some examples, one or more misclassified images can be removed from the noisy dataset 306 to generate the final dataset 308. The misclassified images can be images that were misclassified by the initially trained binary model 304 in the noisy dataset 304 as exposure good images. The final dataset 308 with classifications can then be used to retrain the binary model 304. The retrained binary model 304 can then be used as the binary model 134 to detect whether the image includes an exposure defect in real-time or near real-time. Figure 1 In this way, the binary model 304 network can be trained using a larger dataset without the need to manually label each image in the noisy dataset 306.

[0062] In some examples, the final dataset 308 can be further iteratively pruned to remove misclassified images. In such examples, the final dataset 308 can be divided into three subsets: a training set, a validation set, and a holdout set. A classifier or model can first be trained using the training set and the validation set to detect well-exposed and valid images in the final dataset. Then, the holdout set can be applied to the trained classifier. Then, a small number (e.g., 5, 10) of images from the holdout set that are most likely to be misclassified by the classifier can be removed or deleted from the holdout set. Then, the training set, the validation set, and the new holdout set can be combined, and the process of removing misclassified images can be iteratively repeated a predetermined number of times to remove misclassified images from the final dataset 308. The pruned final dataset 308 can then be used to retrain the binary model 304. In this way, the binary model 304 can eventually be trained using a more accurately labeled set of training images.

[0063] Referring to Figure 4 , Figure 4 is an illustration of example output 400 of a defect detection system according to an embodiment of the present invention. For example, Figure 1 The defect detection system 130 of FIG. 4A can receive image 412, image 414, and / or image 416 as input for exposure defect detection and classification. For each image in the image category 410, the output 400 can include a confidence score category 420, a binary model classification category 430, an exposure score category 440, and an exposure defect classification category 450. For example, the output of the defect detection system for image 412 can include a confidence score 422 (i.e., a confidence score of 0.00262324) and a binary model classification 432 (e.g., well-exposed) output by a binary model (e.g., binary model 134). Since image 412 is determined to be a well-exposed image, image 412 can not be associated with an exposure score and an exposure defect classification.

[0064] Similarly, the output of the defect detection system for image 414 can include a confidence score 424 (i.e., a confidence score of 0.999731) and a binary model classification 434 (e.g., defect) output by a binary model (e.g., binary model 134). As can be appreciated, the binary model classification 434 is based on the confidence score 424 being above a confidence threshold (e.g., 0.5). Further, since image 414 is predicted to be a defect image, the output corresponding to image 414 can also include an exposure score 444 (i.e., an exposure score of 0.916261) and an exposure defect classification 454 (i.e., severe overexposure) output by a classification model (e.g., classification model 136). Image 414 can be classified as including severe overexposure based on the exposure score of image 414 being close to one.

[0065] The output of the defect detection system for the image 416 can include a confidence score 426 (i.e., confidence score 0.7985233) and a binary model classification 436 (e.g., defect) output by a binary model (e.g., binary model 134). As can be appreciated, the binary model classification 436 is based on the confidence score 426 being above a confidence threshold (e.g., 0.5). Further, since the image 416 is predicted to be a defective image, the output corresponding to the image 416 can also include an exposure score 446 (i.e., exposure score 0.375255) and an exposure defect classification 456 (i.e., moderate underexposure) output by a classification model (e.g., classification model 136). The image 416 can be classified as including moderate underexposure based on the exposure score of the image 416 exceeding approximately 0.25. It should be appreciated that the image and associated scores and classifications are for example purposes only, and different scores can be associated with different classifications based on different thresholds and ranges used to train the respective binary and classification models.

[0066] Exemplary flowchart

[0067] Referring now to Figure 5 , a flow diagram is provided that illustrates a method for detecting and classifying exposure defects in images according to the present disclosure. Each block of the method 500 includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing stored instructions in memory. The method can also be implemented as computer-usable instructions stored on a computer storage media. The method can be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few.

[0068] First, at block 502, an image is accessed. The image can be received by an image receiver, such as the image receiver 132 of the Figure 1 The image can be accessed from a digital media gallery, such as the digital media gallery 115 of the client device 110 of the Figure 1 At block 504, a first neural network is used to determine whether the image includes an exposure defect. A binary model, such as the binary model 134 of the Figure 1 The binary model can be used to determine whether the received image includes an exposure defect by classifying the image as a defective image or an exposure good image.

[0069] At block 504, in response to determining that the image does not include an exposure defect, the method ends. In some examples, in response to determining that the image does not include an exposure defect, the image can be classified as an exposure good image.

[0070] At block 506, in response to determining that the image includes an exposure defect, a second neural network is used to determine an exposure defect classification associated with the image. The exposure defect classification includes one of overexposure and underexposure. For example, a classification model, such as the classification model 136 of Figure 1 FIG. 1, can be used to classify a defective image as including or suffering from overexposure or underexposure. In some examples, the classification model can also determine an exposure score that indicates a level of exposure defect detected in the image. The exposure score can then be used by the classification model to determine the exposure defect classification associated with the image.

[0071] At block 508, the image with an indication of the exposure defect classification is provided. The image with an indication of the exposure defect classification can be provided or presented via a user interface of a client device, such as the client device 110 of Figure 1 FIG. 1.

[0072] Exemplary operating environment

[0073] Having described an overview of embodiments of the application, an example operating environment in which embodiments of the application can be implemented is described below to provide a general context for various aspects of the application. Reference will now be made in detail to Figure 6 an example operating environment for implementing embodiments of the application is shown and generally designated as computing device 600. Computing device 600 is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the application. Neither should computing device 600 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated.

[0074] The application can be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, whether or not such business method patent applications are ultimately concluded to be patent eligible subject matter under 35 U.S.C. § 101. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, etc. The application can also be practiced in distributed computing environments where tasks are performed by remote-processing devices that are linked through a communications network.

[0075] With reference to Figure 6The computing device 600 includes a bus 610 that directly or indirectly couples the following devices: memory 612, one or more processors 614, one or more presentation components 616, input / output (I / O) ports 618, input / output components 620, and an illustrative power supply 622. Bus 610 represents what can be one or more busses (such as an address bus, data bus, or a combination thereof). Although the various Figure 6 components of the computing device 600 are shown by lines in the figures, in actuality, the various components are not so clearly delineated, and the lines are more of a metaphor to show the proximity of the components to each other than an actual division. For example, a presentation component, such as a display device, can be considered an I / O component. Also, a processor has memory. The inventors recognize that this is the nature of the art, and reiterate that Figure 6 the figures are merely illustrative of exemplary computing devices that can be used with one or more embodiments of the present invention. Distinction is not made between a "workstation," "server," "laptop computer," "handheld device," or "other Figure 6 device," as all of these can be used in the same manner.

[0076] The computing device 600 typically includes a variety of computer-readable media. Computer-readable media can be any media that can be accessed by the computing device 600 and includes both volatile and nonvolatile media and removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computing device 600. Computer storage media is non-transitory and does not include modulated data signals. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer-readable media.

[0077] Memory 612 includes computer-storage media in the form of volatile and / or nonvolatile memory. The memory can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. Computing device 600 includes one or more processors that read data from various entities such as memory 612 or I / O components 620. Presentation component(s) 616 present data indications to a user or other devices. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc.

[0078] I / O ports 618 allow computing device 600 to be logically coupled to other devices including I / O components 620, some of which can be built in. Illustrative components, modules, and functions include a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, touchpad, touch screen, etc. The I / O components 620 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to an appropriate network element for further processing. A NUI can implement any combination of speech recognition, gesture recognition, facial recognition, biometric recognition, navigation of the air gestures and on- and near-screen user interface, air gestures, head and eye tracking, and touch recognition associated with a display of the computing device 600, as described in more detail below. The computing device 600 can be equipped with depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations of these, for gesture detection and recognition. Additionally, the computing device 600 can be equipped with accelerometers or gyroscopes that enable detection of motion. The output of the accelerometers or gyroscopes can be provided to the display of the computing device 600 to render immersive augmented reality or virtual reality.

[0079] Embodiments described herein support exposure defect detection and classification in digital media. Components described herein refer to integrated components of a defect detection system. Integrated components refer to hardware architecture and software framework that support functionality using a defect detection system. Hardware architecture refers to physical components and their interrelationships, and software framework refers to software that provides functionality that can be implemented with hardware that is embodied on a device.

[0080] End-to-end software-based defect detection systems can operate within defect detection system components to operate computer hardware to implement defect detection system functionality. At a low level, a hardware processor executes instructions selected from a machine language (also referred to as machine code or native) instruction set of a given processor. The processor recognizes the native instructions and performs corresponding low-level functions related to, for example, logic, control, and memory operations. Low-level software written in machine code can provide more complex functionality for higher-level software. As used herein, computer-executable instructions include any software, including low-level software written in machine code, higher-level software such as application software, and any combination thereof. In this regard, defect detection system components can manage resources and provide services for defect management system functionality. Any other variations and combinations thereof are contemplated with embodiments of the present invention.

[0081] Various components have been identified in the present disclosure, it should be understood that any number of components and arrangements can be employed to achieve the desired functionality within the scope of the present disclosure. For example, for conceptual clarity, the components of the embodiments depicted in the figures have been shown in a line arrangement. Other arrangements can also be implemented. For example, although some components are depicted as single components, many of the elements described herein can be implemented as discrete or distributed components or combined with other components and implemented in any suitable combination and location. Some elements can be entirely omitted. Furthermore, as described below, various functions described herein as being performed by one or more entities can be carried out by hardware, firmware, and / or software. For example, various functions can be carried out by a processor executing instructions stored in memory. As such, other arrangements and elements (e.g., machines, interfaces, functions, commands, and groupings of functions, etc.) can be used in addition to or instead of the ones shown.

[0082] The subject matter of the present invention is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, including different steps or combinations of steps in different groupings, to include variations in the steps disclosed herein, or to include different steps not disclosed herein. Moreover, although the terms "step" and / or "block" can be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

[0083] The present invention has been described in relation to particular embodiments which are intended in all respects to be illustrative rather than restrictive. Alternative embodiments will become apparent to those who are skilled in the art to which the present invention pertains without departing from its scope.

[0084] From the foregoing, it will be seen that this application is one which is well adapted to achieve all the ends and objects herein-above set forth, together with other advantages which are obvious and inherent to the system and method. It will be understood that certain features and subcombinations are of utility and can be employed without reference to other features and subcombinations. This is contemplated by and is within the scope of the claims.

Claims

1. A method for training a neural network, the method comprising: Receive a first set of images, each of which is labeled as either a well-exposed image or a defective image; Train the neural network using the first set of images; Receive a second set of images, the second set of images including unlabeled images; Determine a distribution ratio, which represents the distribution of the well-exposed image and the defective image within the first group of images; The second set of images is applied to the neural network to determine an inference score for each image in the second set of images, the inference score indicating the exposure level in the corresponding image; Based at least in part on the distribution ratio and the inferred score, each image in the second group of images is classified as a well-exposed image or a defective image; as well as The neural network is retrained using the second set of images and the classification.

2. The method according to claim 1, further comprising: At least one image in the second group of images is determined to be a misclassified image, based at least in part on the distribution ratio and the inference score; And after removing the misclassified images, the neural network is retrained using the second set of images.

3. The method according to claim 1, further comprising: A threshold score is determined at least in part based on the inferred score or the distribution ratio to classify the second group of images as either the well-exposed images or the defective images, the threshold score being determined such that the second distribution ratio corresponding to the second group of images is the same as the distribution ratio corresponding to the first group of images.

4. The method of claim 1, wherein the neural network is a deep neural network having multiple layers trained using multiple learning rates, such that a first set of layers of the multiple top layers of the neural network is trained at a lower learning rate than a second set of bottom layers of the neural network.

5. The method according to claim 1, further comprising: The second neural network is trained, at least in part, based on the trained neural network, to classify the image based on the level of exposure defects using a regression algorithm.

6. The method of claim 5, wherein the exposure defect level includes one or more of severe overexposure, mild overexposure, moderate overexposure, severe underexposure, mild underexposure, and moderate underexposure.

7. A computer storage medium storing computer-usable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations, the operations including: Receive a first set of images, each of which is labeled as either a well-exposed image or a defective image; Train the neural network using the first set of images; Receive a second set of images, the second set of images including unlabeled images; Determine a distribution ratio, which represents the distribution of the well-exposed image and the defective image within the first group of images; The second set of images is applied to the neural network to determine an inference score for each image in the second set of images, the inference score indicating the exposure level in the corresponding image; Based at least in part on the distribution ratio and the inferred score, each image in the second group of images is classified as a well-exposed image or a defective image; as well as The neural network is retrained using the second set of images and the classification.

8. The medium according to claim 7, wherein the operation further comprises: At least one image in the second group of images is determined to be a misclassified image, based at least in part on the distribution ratio and the inference score; And after removing the misclassified images, the neural network is retrained using the second set of images.

9. The medium according to claim 7, wherein the operation further comprises: A threshold score is determined at least in part based on the inferred score or the distribution ratio to classify the second group of images as either the well-exposed images or the defective images, the threshold score being determined such that the second distribution ratio corresponding to the second group of images is the same as the distribution ratio corresponding to the first group of images.

10. The medium of claim 7, wherein the neural network is a deep neural network having multiple layers trained using multiple learning rates such that a first set of layers of the multiple top layers of the neural network is trained at a lower learning rate than a second set of bottom layers of the neural network.

11. The medium according to claim 7, wherein the operation further comprises: The second neural network is trained, at least in part, based on the trained neural network, to classify the image based on the level of exposure defects using a regression algorithm.

12. The medium of claim 11, wherein the exposure defect level includes one or more of severe overexposure, mild overexposure, moderate overexposure, severe underexposure, mild underexposure, and moderate underexposure.

13. A neural network training system, comprising: One or more hardware processors and a memory, the memory being configured to provide computer program instructions to the one or more hardware processors; The neural network training environment is configured to use the one or more hardware processors to: Receive a first set of images, each of which is labeled as either a well-exposed image or a defective image; Train the neural network using the first set of images; Receive a second set of images, the second set of images including unlabeled images; Determine a distribution ratio, which represents the distribution of the well-exposed image and the defective image within the first group of images; The second set of images is applied to the neural network to determine an inference score for each image in the second set of images, the inference score indicating the exposure level in the corresponding image; Based at least in part on the distribution ratio and the inferred score, each image in the second group of images is classified as a well-exposed image or a defective image; as well as The neural network is retrained using the second set of images and the classification.

14. The system of claim 13, wherein the neural network training environment is configured to use the one or more hardware processors to further: The neural network is determined to be at least one misclassified image in the second set of images based at least in part on the distribution ratio and the inference score; and after removing the misclassified images, the neural network is retrained using the second set of images.

15. The system of claim 13, wherein the neural network training environment is configured to use the one or more hardware processors to further: A threshold score is determined at least in part based on the inferred score or the distribution ratio to classify the second group of images as either the well-exposed images or the defective images, the threshold score being determined such that the second distribution ratio corresponding to the second group of images is the same as the distribution ratio corresponding to the first group of images.

16. The system of claim 13, wherein the neural network is a deep neural network having multiple layers trained using multiple learning rates such that a first set of layers of the multiple top layers of the neural network is trained at a lower learning rate than a second set of bottom layers of the neural network.

17. The system of claim 13, wherein the neural network training environment is configured to use the one or more hardware processors to further: The second neural network is trained, at least in part, based on the trained neural network, to classify the image based on the level of exposure defects using a regression algorithm.

18. The system of claim 17, wherein the exposure defect level includes one or more of severe overexposure, mild overexposure, moderate overexposure, severe underexposure, mild underexposure, and moderate underexposure.

Images