Automatic Artifact Detection

Abstract

Techniques are provided for generating a trained classifier, including applying one or more of glitched or non-glitched images to a classifier, receiving a classification output from the classifier, adjusting weights of the classifier to improve the classification accuracy of the classifier, applying noise to a generator, receiving an output image from the generator, applying the output image to the classifier to obtain a classification, and adjusting weights of either the classifier or the generator based on the classification to improve the ability of the generator to reduce the classification accuracy of the classifier.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Application No. 63 / 035,625, filed on Jun. 5, 2020, entitled "AUTOMATED ARTIFACT DETECTION", and U.S. Patent Application No. 17 / 030,250, filed on Sep. 23, 2020, entitled "AUTOMATED ARTIFACT DETECTION", the entire contents of which are incorporated herein by reference.

Background Art

[0002] Videos, such as sequences of frames generated by video games, may contain glitches. There is no known automated system capable of detecting whether such a video contains glitches.

[0003] A more detailed understanding can be obtained from the following description given by way of example together with the accompanying drawings.

Brief Description of the Drawings

[0004] [Figure 1A] It is a block diagram of an exemplary computing device in which one or more features of the present disclosure can be implemented. [Figure 1B] It is a diagram showing a training device according to an example. [Figure 1C] It is a diagram showing an evaluation device including an evaluation system for generating a classification of input data based on a training network according to an example. [Diagram 2] It is a diagram showing a network for training a generator and a discriminator to generate glitchy and non - glitchy images and classify images as having or not having glitches according to an example. [Figure 3A] It is a flowchart of a method for generating a classifier for classifying an image as either containing or not containing glitches. [Figure 3B] A flowchart of a method for classifying an image as glitchy or non - glitchy according to an example.

Embodiments for Carrying Out the Invention

[0005] Techniques are provided for generating a trained discriminator. The techniques include applying one or more glitchy or non - glitchy images to the discriminator, receiving a classification output from the discriminator, adjusting the weights of the discriminator to improve its classification accuracy, applying noise to a generator, receiving an output image from the generator, applying the output image to the discriminator to obtain a classification, and adjusting the weights of either the discriminator or the generator to improve the generator's ability to reduce the discriminator's classification accuracy based on the classification.

[0006] 1A is a block diagram of an example computing device 100 in which one or more features of the present disclosure may be implemented. Device 100 may be, for example, but not limited to, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, a tablet computer, or any other computing device. Device 100 includes one or more processors 102, memory 104, storage 106, one or more input devices 108, and one or more output devices 110. Device 100 also includes one or more input drivers 112 and one or more output drivers 114. Any input driver 112 may be embodied as hardware, a combination of hardware and software, or software, and is responsible for controlling (e.g., controlling the operation of, receiving input from, and providing data to) input driver 112. Similarly, any output driver 114 may be embodied as hardware, a combination of hardware and software, or software and serves to control (e.g., control the operation of, receive input from, and provide data to) output driver 114, output device 110. It should be understood that device 100 may include additional components not shown in FIG. 1A.

[0007] In various alternatives, the one or more processors 102 may include a central processing unit (CPU), a graphics processing unit (GPU), a CPU and a GPU located on the same die, or one or more processor cores, each of which may be a CPU or a GPU. In various alternatives, the memory 104 may be located on the same die as one or more of the one or more processors 102 or may be located separately from the one or more processors 102. The memory 104 may include volatile or non-volatile memory (e.g., random access memory (RAM), dynamic RAM, cache).

[0008] Storage devices 106 include fixed or removable storage devices (e.g., but not limited to, hard disk drives, solid state drives, optical disks, flash drives). Input devices 108 include, but are not limited to, keyboards, keypads, touchscreens, touchpads, detectors, microphones, accelerometers, gyroscopes, biometric scanners, or network connections (e.g., wireless local area network cards for transmitting and / or receiving wireless IEEE 802 signals). Output devices 110 include, but are not limited to, displays, speakers, printers, haptic feedback devices, one or more optics, antennas, or network connections (e.g., wireless local area network cards for transmitting and / or receiving wireless IEEE 802 signals).

[0009] The input driver 112 and the output driver 114 comprise one or more hardware, software, and / or firmware components that interface with and drive the input devices 108 and the output devices 110, respectively. The input driver 112 communicates with one or more processors 102 and the input devices 108, allowing the one or more processors 102 to receive input from the input devices 108. The output driver 114 communicates with one or more processors 102 and the output devices 110, allowing the one or more processors 102 to send output to the output devices 110.

[0010] In various implementations, device 100 includes one or both of evaluation system 120 and training system 122. Evaluation system 120 is capable of detecting graphical anomalies (glitches) in images, such as images generated by a video game. Training system 122 trains one or more machine learning components (sometimes called "classifiers") of the network of evaluation system 120 to recognize glitches.

[0011] Some implementations of device 100 include a computer system configured to train one or more of the machine learning components. Some such implementations include a computer system, such as a server or other computer system associated with a server or server farm, that generates one or more trained classifiers. In some implementations, the computer system that generates the one or more trained classifiers uses the trained classifiers to evaluate whether an input image contains or does not contain a glitch. Other implementations of device 100 include a computer system (such as a client) configured to store a training network (generated by a different computer system) and evaluate input data (e.g., images) through the training network to determine whether the input data contains a glitch. Thus, device 100 generally represents the architecture of one or more computer systems that generate a training network and use the training network to determine whether one or more images contain a glitch.

[0012] FIG. 1B illustrates a training device 150 according to an example. In some implementations, the training device 150 is implemented as the device 100 of FIG. 1A, and the training system 152 is software, hardware, or a combination thereof for performing the operations described herein. The training system 152 accepts a training dataset 154 and trains a training network 156. The training dataset 154 includes a plurality of images that support training of the training network 156. The training system 152 trains the training network 156 based on the training dataset 154 to recognize whether a subsequent input image contains a glitch or does not contain a glitch. In some examples, the training network 156 includes a generative adversarial network, which is described in further detail herein.

[0013] FIG. 1C is a diagram of an evaluation device 160 including an evaluation system 162 for generating a classification 168 of input data 164 based on a training network 156 (which in some implementations is the training network 166 generated by the training system 152). The classification 168 includes an indication of whether the image of the input data 164 includes a glitch or does not include a glitch. In this specification, the term "defect" may replace the term "glitch". In some examples, the evaluation system 162 alternatively or additionally generates an output indicating whether any type of glitch is present in the input data 164.

[0014] As described above, the training system 152 implements a generative adversarial network. The generative adversarial network uses components of a generator and a discriminator that act in reverse, combined with backpropagation, to improve the performance of both such components. The training system 152 improves the ability of the generator to generate images that are convincingly considered to be either glitched or normal. Also, the training system 152 improves the ability of the discriminator to determine whether the output image from the generator is glitched or not.

[0015] Since the discriminator can determine whether an image is glitched or not, the evaluation system 162 utilizes the discriminator generated by the training system 152 when determining whether the input data 164 for classification is considered to include a glitch.

[0016] 2 illustrates a network 200 for training a generator 202 and a classifier 204 to generate glitched and non-glitched images and classify images as having or not having glitches, according to one example. Network 200 includes a generator 202 and a classifier 204 that together form a generative adversarial network (GAN). Generator 202 generates a generated image 203 based on input noise (not shown—the input noise is generated in any technically feasible manner, such as via a random number generator), and the classifier attempts to classify generated image 203 as normal or containing defects (glitches). Orchestrator 206 coordinates the training of network 200.

[0017] The orchestrator 206 trains the classifier using an input image set 201 that includes a set of images that either have glitches or do not have glitches. The particular images included in the input image set 201 depend on the training scheme used. Various training schemes are described elsewhere herein.

[0018] The orchestrator 206 trains the network 200 by providing noise to the generator 202 that outputs the generated image 203. The orchestrator 206 causes the discriminator 204 to output a classification 205 indicating whether the generated image 203 contains glitches or not. The orchestrator 206 accurately classifies the input image set 201 (i.e., as containing glitches if the input image set 201 contains only glitchy images, or as not containing glitches if the input image set 201 does not contain glitchy images) and trains the discriminator 204 to accurately classify the generated image 203. The orchestrator 206 trains the generator 202 to "fool" the discriminator 204 by maximizing the error rate of the discriminator 204. Thus, the orchestrator 206 continuously improves the ability of the generator 202 to generate "realistic-looking" images and the ability of the discriminator 204 to accurately classify images as containing glitches or not. In some implementations, the discriminator 204 is a convolutional neural network. In some implementations, the generator 202 is a deconvolutional neural network.

[0019] As described above, the initial input image set 201 contains images that either contain glitches or do not. Some exemplary glitches include shader artifacts, shape artifacts, discoloration artifacts, Morse code patterns, dotted line artifacts, parallel lines, triangulation, line pixelation, screen stuttering, screen tearing, square patch artifacts, blurring artifacts, and random patch artifacts.

[0020] Shader artifacts include visible artifacts associated with improper shading. A "shader program" is a program that runs on a graphics processing unit and performs functions such as transforming vertex coordinates (vertex shader program) and coloring pixels (pixel shader program). Shader artifacts occur when one or more polygons are improperly shaded. Examples of such improper shading appear visually in an image as polygons of different colors that blend into each other or fade in certain directions.

[0021] Shape artifacts are artifacts in which random polygonal, monochromatic shapes appear in an image. Discoloration artifacts are artifacts in which bright spots of a different color than expected exist in an image. Morse code patterns appear when memory cells on a graphics card become stuck, resulting in stuck values being displayed instead of the true image. In various examples, GPUs operating at speeds faster than the GPU was designed for or at temperatures higher than the GPU was designed for can result in Morse code patterns.

[0022] Dotted line artifacts typically involve dotted lines with random slopes and positions, or radial lines emanating from a single point. Parallel line artifacts involve lines that are parallel, have a uniform color, and are not part of the true image. Triangulation artifacts appear as a grid of triangles across the image, where a smoother, more natural image is actually correct. Line pixelation is characterized as stripes (such as horizontal stripes) with random colors within an image. Screen stuttering occurs when adjacent columns and rows of an image are swapped with each other. Screen tearing occurs as two consecutive frames in a video rendered within the same image; one part of the image is a scene at one point in time, and another part of the image is a scene at a different point in time.

[0023] A square patch artifact is a square patch of uniform or nearly uniform color that appears irregularly within an image. A blur artifact is a blur in a portion of an image that should appear in focus. A random patch artifact is a randomly shaped patch of uniform or nearly uniform color that appears irregularly within an image.

[0024] The phrase "deemed to include glitches" may be replaced in this specification with "includes glitches" and means that the identifier 204 labels an image as including glitches. Similarly, phrases such as "deemed not to include glitches" may be replaced with "does not include glitches" and mean that the identifier 204 labels the generated image 203 as not including glitches.

[0025] Referring to FIGS. 1B and 2 together, in some implementations, the training network 166 that the evaluation system 162 uses to classify input data 164 for classification is the identifier 204 generated by the network 200 of FIG. 2. In one example, a server or other system includes or is the training system 152. The server executes training to generate the identifier 204 and sends the identifier 204 to the evaluation system 162. The evaluation system uses the identifier 204 as the training network 166 to evaluate the input data and generate a classification 168. Specifically, the evaluation system inputs an image into the trained identifier 204 and generates an output classification indicating whether the image includes glitches.

[0026] FIG. 3A is a flowchart of a method 300 for generating a classifier to classify an image as either including or not including glitches. Although described with respect to the systems of FIGS. 1A - 2, any system configured to perform the steps of method 300 in any technically feasible order is within the scope of this disclosure.

[0027] Method 300 begins at step 302, where the orchestrator 206 provides noise to the generator 202 to generate an output image. As described elsewhere in this specification, the generator 202 is a neural network configured to generate an image from noise. In one example, the generator 202 is a deconvolutional neural network.

[0028] In step 304, the orchestrator 206 provides the output image to the discriminator 204 to classify the output image. The discriminator 204 is capable of classifying the image as either containing a glitch or not containing a glitch. As described in more detail below, the type of images in the training set 201 determines how images from the training set 201 and images generated by the generator 202 map to either "glitchy" or "non-glitchy" images. Generally, the term "fake" is used to denote an image generated by the generator 202, and the term "real" is used to denote an image provided as the training set 201. Again, the specific way in which an image that may be considered "glitchy" or "non-glitchy" maps to a "fake" or "real" image depends on the specific configuration of the discriminator 204 and the generator 202. Some such methods are described in more detail below.

[0029] In step 306, the orchestrator 206 performs backpropagation to update the weights of one or both of the generator 202 or the discriminator 204. In some implementations, a "pass" refers to one instance of providing noise to generate an output image (302) and providing an image to generate a classification (304), and during each "pass," the weights of either the discriminator 204 or the generator 202 are adjusted, but the weights of the other of the discriminator 204 or the generator 202 are not adjusted. In other words, during a single pass, the weights of either the discriminator 204 or the generator 202 are held constant. Backpropagation for the discriminator 204 involves adjusting the weights to minimize the error with respect to the actual classification of the image, which is based on whether the image is generated by the generator (fake) or the input image 201. In other words, backpropagation towards the classifier 204 attempts to maximize the accuracy with which the classifier 204 can identify whether an image was generated by the generator 202 or provided as the input image 201. Backpropagation towards the generator 202 involves maximizing the classification error of the classifier 204. In other words, in the case of the generator 202, backpropagation attempts to increase the chances that the classifier 204 will incorrectly label a "real" image as "fake" or a "fake" image as "real."

[0030] In step 308, the orchestrator 206 provides the training set images 201 to the classifier 204 for classification. As with the images from the generator 202, the classifier 204 processes the images through its neural network to generate a classification that classifies the image as either "real" or "fake." In step 310, the orchestrator 206 performs backpropagation as described above to update the weights of the generator 202 or the classifier 204.

[0031] In some examples, it is to be understood that there are three different types of paths. One is that a genuine image is provided to the discriminator 204, the discriminator 204 attempts to correctly classify the image, and then backpropagation is applied to the discriminator 204. One is that the generator 202 generates an image to be identified by the discriminator 204, and backpropagation is applied to the generator 202. One is that the generator 202 generates an image to be identified by the discriminator 204, and backpropagation is applied to the discriminator 204. Any of these types of paths can be executed in any desired order. In one example, these three types of paths are performed alternately such that different types are executed one after another. In another example, the three different types are grouped together or batched, and these batches are processed through the generator 202 and the discriminator 204.

[0032] In various examples, the generator 202 and the discriminator 204 can be implemented in any of several different ways. In one example, the training set 201 includes genuine images that all have no glitches. The discriminator 204 classifies an image received from the generator 202 or a "genuine" image received as part of the training set 201 as either being genuine (and thus not glitchy) or being fake (and thus glitchy). In this example, a "genuine" image is mapped to an "image without glitches" and a "fake" image is mapped to an "image with glitches". In another example, the training set 201 includes genuine images, all of which have glitches. In this example, a "genuine" image is mapped to an "image with glitches" and a "fake" image is mapped to an "image without glitches".

[0033] In another example, the discriminator 204 is trained to recognize images of multiple types of genuine and fake items. In one example, the discriminator 204 is trained to recognize glitchy and non-glitchy images of genuine and fake items. In this example, the images of "genuine" and "fake" items are not directly mapped to being glitchy or non-glitchy. In this example, the generator 202 generates either a glitchy image or an image of a genuine item. In some examples, the decision of whether to attempt to generate a glitchy image or an image of a genuine item is based on a randomly selected input. The input set 201 includes both images labeled as being glitchy and images labeled as not being glitchy. In this scenario, the discriminator 204 attempts to accurately classify the image as being genuine (i.e., received from the input set 201) and glitchy, genuine and non-glitchy, fake (i.e., generated by the generator 202) and glitchy, or fake and non-glitchy. The orchestrator 206 performs backpropagation to increase the generator 202's ability to deceive the discriminator 204 by misclassifying a fake image as genuine, and performs backpropagation to increase the discriminator 204's ability to appropriately classify the input image as being in any of the four categories listed above.

[0034] In yet another example, the system includes multiple combinations of the discriminator 204 and the generator 202, and each combination is tuned to a specific type of glitch. The input image 201 to each such combination includes images having the specific glitch assigned to that combination. The discriminator 204 is trained to appropriately classify images of genuine and fake items of the assigned glitch type, and the generator 202 is trained to "deceive" the discriminator 204 into misclassifying the images.

[0035] The training set 201 is generated in any technically feasible manner. In one example, one or more people obtain screenshots from an actual product. Some of these screenshots contain glitches, and some of these screenshots do not contain glitches.

[0036] FIG. 3B is a flowchart of a method 350 for classifying an image as containing or not containing glitches, according to one example. Although described with respect to the systems of FIGS. 1A - 2, those skilled in the art will understand that any system configured to perform the steps of method 3B is within the scope of the present disclosure.

[0037] Method 350 begins at step 350, where the trained discriminator 204 receives an input image to be classified from an input image source. The input image source is any entity capable of generating an image. In some examples, the input image source is software such as a video game that renders a scene and generates an output image for display. In other examples, the input image source is an image from other software or hardware such as a computer application or a video source.

[0038] At step 354, the trained discriminator 204 sequentially feeds the input image through the neural network layers of the discriminator. More specifically, as described elsewhere in this specification, each neural network layer receives either an input from a previous layer or an input to the network (e.g., the input image itself), processes that input through the layer, and provides an output to either a subsequent layer or the output of the network itself (e.g., as a classification). In one example, the discriminator 204 is implemented as a convolutional neural network that includes one or more convolutional layers that perform a convolution operation. In various examples, the discriminator 204 includes other types of operations performed in various layers, either in conjunction with or instead of the convolution operation.

[0039] In step 356, the trained classifier 204 generates an output classification based on the results from propagation through all layers. As mentioned above, the specific type of classification depends on how the classifier 204 was trained. Examples are provided elsewhere herein. In one example, the classifier 204 may label an image as either containing a glitch or not containing a glitch. In some examples, the classifier 204 may indicate which type of glitch is present in the image. In some such examples, the classifier 204 is actually multiple individual trained classifiers, as described above.

[0040] The identifier 204, generator 202, and / or orchestrator 206 may be embodied as software running on a programmable processor (which is a hardware processor including appropriate circuitry), as fixed-function hardware circuitry configured to perform the functions described herein, or as a combination of software and hardware.

[0041] Any of the various elements of FIGS. 1A-2, such as training system 152, evaluation system 162, network 200, generator 202, discriminator 204, and orchestrator 206, may, in various embodiments, be implemented as software running on a processor, as hardware (e.g., circuitry) hardwired to perform the various operations described herein, or a combination thereof.

[0042] It should be understood that many variations are possible based on the disclosure herein, and although features and elements are described above in particular combinations, each feature or element can be used alone without other features and elements, or in various combinations with or without other features and elements.

[0043] The provided method can be implemented on a general-purpose computer, processor, or processor core. Suitable processors include, by way of example, general-purpose processors, dedicated processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application specific integrated circuits (ASICs), field programmable gate array (FPGA) circuits, any other type of integrated circuit (IC), and / or state machines. Such processors can be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediate data such as netlists, which can be stored on a computer-readable medium. The result of such processing can be a mask work, which can then be used in subsequent semiconductor manufacturing processes to manufacture a processor implementing the features of the present disclosure.

[0044] The methods or flowcharts provided herein can be implemented in a computer program, software, or firmware incorporated into a non-transitory computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-transitory computer-readable storage media include read only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (e.g., internal hard disks and removable disks), magneto-optical media, and optical media (e.g., CD-ROM disks and digital versatile disks (DVDs)).

Claims

1. 1. A method for generating a trained classifier, comprising: applying a glitched image and a non-glitched image to a classifier, wherein the glitched image includes at least one of shader artifacts, shape artifacts, discoloration artifacts, morse code patterns, dotted line artifacts, parallel lines, triangulation, line pixelization, screen stuttering, screen tearing, square patch artifacts, blurring artifacts, and random patch artifacts; receiving a classification output from the classifier; adjusting the weights of the classifier to improve the classification accuracy of the classifier; injecting noise into the generator; receiving an output image from the generator; applying the output image to the classifier to obtain a classification; and adjusting weights of either the classifier or the generator based on the classification to improve the ability of the generator to reduce the classification accuracy of the classifier. method.

2. transmitting the trained classifier from the computer system that generates the trained classifier to a different computer system that is configured to activate the trained classifier to determine whether an image contains a glitch.

10. The method of claim 1.

3. further comprising, in the different computer system, classifying one or more images as having a glitch or not having a glitch. The method of claim 2.

4. The glitched image or the non-glitched image is selected from a training set.

10. The method of claim 1.

5. The training set includes all glitched images. The method of claim 4.

6. The training set includes all non-glitched images. The method of claim 4.

7. The training set includes glitched and non-glitched images, and the classifier is configured to provide one of four classifications including real and glitched, real and non-glitched, fake and glitched, and fake and non-glitched. The method of claim 4.

8. The training set includes images labeled with glitch type. The method of claim 4.

9. providing the training set to a plurality of classifier-generator pairs, each configured for a different type of glitch.

9. The method of claim 8.

10. 1. A system for generating a trained classifier, comprising: The orchestrator and A generator; a classifier; The orchestrator: applying a glitched image and a non-glitched image to a classifier, wherein the glitched image includes at least one of shader artifacts, shape artifacts, discoloration artifacts, morse code patterns, dotted line artifacts, parallel lines, triangulation, line pixelization, screen stuttering, screen tearing, square patch artifacts, blurring artifacts, and random patch artifacts; receiving a classification output from the classifier; adjusting the weights of the classifier to improve the classification accuracy of the classifier; applying noise to the generator; receiving an output image from the generator; applying the output image to the classifier to obtain a classification; Based on the classification, to improve the ability of the generator to reduce the classification accuracy of the discriminator, adjusting the weights of either the discriminator or the generator; configured to perform; system.

11. The orchestrator is further configured to transmit the trained discriminator from a computer system that generates the trained discriminator to a different computer system configured to activate the trained discriminator to determine whether an image contains a glitch. The system of claim 10.

12. The different computer system is configured to classify one or more images as containing a glitch or not containing a glitch using data for the trained discriminator. The system of claim 11.

13. The glitchy image or the non-glitchy image is selected from a training set. The system of claim 10.

14. The training set includes all glitchy images. The system of claim 13.

15. The training set includes all non-glitchy images. The system of claim 13.

16. The training set includes glitchy images and non-glitchy images, and the discriminator is configured to provide any one of four classifications including genuine and glitchy, genuine and non-glitchy, fake and glitchy, and fake and non-glitchy. The system of claim 13.

17. The training set includes images labeled with glitch types. The system of claim 13.

18. The orchestrator is further configured to provide the training set to a plurality of discriminator-generator pairs, each configured for a different type of glitch. The system of claim 17.

19. A computer-readable storage medium storing instructions, wherein the instructions, when executed by a processor, applying a glitched image and a non-glitched image to a classifier, wherein the glitched image includes at least one of shader artifacts, shape artifacts, discoloration artifacts, morse code patterns, dotted line artifacts, parallel lines, triangulation, line pixelization, screen stuttering, screen tearing, square patch artifacts, blurring artifacts, and random patch artifacts; receiving a classification output from the classifier; adjusting the weights of the classifier to improve the classification accuracy of the classifier; injecting noise into the generator; receiving an output image from the generator; applying the output image to the classifier to obtain a classification; adjusting weights of either the classifier or the generator based on the classification to improve the ability of the generator to reduce the classification accuracy of the classifier; causing the processor to generate a trained classifier by A computer-readable storage medium.

20. The instructions cause the processor to transmit the trained classifier from a computer system that generates the trained classifier to a different computer system that is configured to activate the trained classifier to determine whether an image contains a glitch.

20. The computer-readable storage medium of claim 19.

Images