Detection of ai generated images

Abstract

Embodiments of the present disclosure relate to detection of AI generated images. A method, apparatus, and non-transitory computer readable medium for obtaining an input image comprising a plurality of pixels. The machine learning model generates annotation information indicating whether each of the plurality of pixels is synthetically generated. A combined image is generated based on the annotation information. In some cases, the combined image shows a synthetically generated region of the input image.

Description

Background Art

[0001] The following generally relates to image processing, and more specifically, to image detection. Image processing refers to the use of a computer to edit an image using an algorithm or processing network. In some cases, image processing software can be used for various image processing tasks, such as image editing, image restoration, image generation, and image detection. Image detection includes the use of a machine learning model to identify and locate objects within an image. In some cases, a machine learning model is trained to classify image regions that contain artifacts or non-artifacts. Summary of the invention

[0002] Various aspects of the present disclosure provide methods, non-transitory computer-readable media, devices, and systems for image detection. According to one aspect of the present disclosure, a machine learning model is trained to generate annotation information indicating whether a pixel in a set of pixels in an image is synthetically generated. In some examples, the machine learning model receives training data, the training data comprising a set of real images and a set of generated images corresponding to the set of real images. In some embodiments, the machine learning model combines a portion of a real image from the training data with a portion of a generated image to obtain a combined image. The machine learning model is also trained based on the combined image.

[0003] A method, apparatus, non-transitory computer-readable medium, and system for image processing are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include: obtaining an input image including a plurality of pixels; generating annotation information using a machine learning model, the annotation information indicating whether each pixel in the plurality of pixels is synthetically generated; and generating a combined image based on the annotation information, wherein the combined image shows a synthetically generated region of the input image.

[0004] A method, apparatus, non-transitory computer-readable medium, and system for image processing are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include: obtaining training data, the training data including a plurality of real images and a plurality of synthetic images generated at least in part by a plurality of different generative machine learning models; and training the machine learning model based on the training data to detect whether the image is a synthetic image.

[0005] An apparatus, system, and method for image processing are described. One or more aspects of the apparatus, system, and method include: at least one processor; at least one memory including instructions executable by the at least one processor; and a machine learning model including parameters stored in the at least one memory, wherein the machine learning model is trained to generate annotation information indicating whether each of a plurality of pixels of an input image is synthetically generated based on training data from a plurality of generative machine learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 An example of an image processing system according to aspects of the present disclosure is shown.

[0007] Figure 2 An example of a method for detecting a composite image according to aspects of the present disclosure is shown.

[0008] Figure 3 An example of detecting a synthesized region of an inpainted image according to aspects of the present disclosure is shown.

[0009] Figure 4 An example of a method for detecting a synthetic region of an image at a pixel level according to aspects of the present disclosure is shown.

[0010] Figure 5 An example of an image processing apparatus according to aspects of the present disclosure is shown.

[0011] Figure 6 An example of a method for detecting a composite region of an image according to aspects of the present disclosure is shown.

[0012] Figure 7 and Figure 8 Examples of methods for training machine learning models according to aspects of the present disclosure are shown.

[0013] Fig. 9 Examples of training machine learning models according to aspects of the present disclosure are shown.

[0014] Fig.10 Examples of detecting synthetically generated images based on a trained machine learning model in accordance with aspects of the present disclosure are shown.

[0015] Fig.11 An example of a computing device for a machine learning model according to aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0016] Various aspects of the present disclosure provide methods, non-transitory computer-readable media, devices, and systems for image detection. According to one aspect of the present disclosure, a machine learning model is trained to generate annotation information indicating whether a pixel in a set of pixels in an image is synthetically generated. In some examples, the machine learning model receives training data, the training data comprising a set of real images and a set of generated images corresponding to the set of real images. In some embodiments, the machine learning model combines a portion of a real image from the training data with a portion of a generated image to obtain a combined image. The machine learning model is also trained based on the combined image.

[0017] A subfield in image detection involves detecting artifacts in images using machine learning models to distinguish images of traditional sources (or real images) (such as photographs, documents, and drawings of traditional sources) from AI-generated images (or synthetic images). Traditional machine learning models for detecting synthetic images are trained using a dataset that includes real images and synthetic images, where the synthetic images are generated from a single image generator. However, when the output of the different image generators is not used as training data for the traditional machine learning model for detecting synthetic images, the traditional machine learning model performs poorly when used to detect synthetic images generated from different image generators.

[0018] Accordingly, unlike conventional models, embodiments of the present disclosure detect synthetic images generated by multiple different image generation models (including models trained after the machine learning model used to detect synthetic images). In addition, some embodiments detect synthetic portions in images that include both real portions and synthetic portions (e.g., created using generative inpainting). For example, some embodiments generate per-pixel information that indicates whether the pixel is synthetically generated.

[0019] By detecting composite images from multiple image generation models and by detecting per-pixel composite image data, embodiments of the present disclosure enable applications such as determining whether an image is authentic (e.g., for inclusion in news media), resolving intellectual property issues, and identifying or incentivizing creators. In addition, even if the original image is not available for comparison, the user can identify the composite areas of the image and perform additional repairs or modifications to those portions without affecting the original portions of the image. Furthermore, embodiments of the present invention enable these applications even if the source of the image is unknown (i.e., it is not known which image generation model (if any) was used to create or modify the image).

[0020] One or more aspects of the present disclosure include training a machine learning model using real images and synthetic images generated by different image generators. In some embodiments, the synthetic images and the corresponding real images are divided into multiple batches and sorted, where the order is based on information indicating the chronological order of the different image generators, such as the date or release date of the image generator. By training the machine learning model with a chronologically sorted data set (including real images and synthetic images), the machine learning model is able to detect synthetic images generated from an image generator without corresponding real images in the machine learning model.

[0021] In some cases, the synthetic image is entirely computer generated. In some cases, the synthetic image is inpainted, where the synthetic image includes regions of the computer generated image and regions of the real image. Traditional machine learning models are trained based on synthetic images (e.g., inpainted images) and corresponding real images and ground truth masks. However, some traditional machine learning models cannot identify inpainted regions of synthetic images without corresponding training data.

[0022] Accordingly, aspects of the present disclosure include detecting synthetic images at a pixel level. In some embodiments, a machine learning model is trained to generate annotation information indicating whether a pixel of a patched image is synthetically generated. In some embodiments, the machine learning model combines a portion of a real image from training data and a portion of a generated image to obtain a combined image. The machine learning model is also trained based on the combined image. By training the machine learning model with the combined image, the machine learning model is able to detect synthetic regions in the generated image without a corresponding real image and a reference truth mask representing the synthetic region. In some embodiments, the machine learning model generates a heat map representing the synthetic region in the generated image.

[0023] Embodiments of the present disclosure may be used in the context of image detection, image verification, and document authentication. For example, an image processing device including a machine learning model based on the present disclosure detects synthetically generated areas in a synthetically generated image or an image generated from an "unseen" generator without a corresponding training image. An "unseen" generator refers to a generative machine learning model, where the output of the generative machine learning model is not used in the training data set of the machine learning model. The image processing device detects synthetically generated areas of an image at the pixel level.

[0024] refer to Figure 2 An example application of the present inventive concept in image detection is provided. Figure 5 Provides details about the architecture of the image processing device. Figure 6 Provides examples of procedures for image processing. Figure 7-Figure 9 A description of an example training procedure is provided.

[0025] As used herein, the terms "synthetically generated," "synthetic image," or "artificial image" refer to pixels or images that are artificially created or generated by using a computer, rather than images captured or recorded from the real world using, for example, sensors and cameras. For example, a synthetically generated image contains artifacts that can be detected by a machine learning model. Real images do not contain artifacts that can be detected by a machine learning model.

[0026] Image Detection

[0027] An apparatus, system, and method for image processing are described. One or more aspects of the apparatus, system, and method include: at least one processor; at least one memory including instructions executable by the at least one processor; and a machine learning model including parameters stored in the at least one memory, wherein the machine learning model is trained to generate annotation information indicating whether each of a plurality of pixels of an input image is synthetically generated based on training data from a plurality of generative machine learning models.

[0028] exist Figure 1-Figure 4 A method, apparatus, non-transitory computer-readable medium, and system for image processing are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include: obtaining an input image including a plurality of pixels; generating annotation information using a machine learning model, the annotation information indicating whether each of the plurality of pixels is synthetically generated; and generating a combined image based on the annotation information, wherein the combined image shows a synthetically generated region of the input image.

[0029] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include encoding the input image to obtain the image embedding. Some examples further include decoding the image embedding to obtain the annotation information.

[0030] In some aspects, the machine learning model is trained using training data comprising synthetic images generated by a plurality of generative machine learning models. In some aspects, the plurality of generative machine learning models comprises a plurality of diffusion models having different architectures.

[0031] In some aspects, the annotation information includes a heat map indicating a likelihood that each pixel in the plurality of pixels is synthetically generated. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include combining the input image with a mask representing the annotation information.

[0032] Figure 1An example of an image processing system according to aspects of the present disclosure is shown. The example shown includes a user 100, a user device 105, an image processing apparatus 110, a cloud 115, and a database 120. The image processing apparatus 110 is a reference Figure 5 Examples of corresponding elements described, or including references Figure 5 Describes aspects of the corresponding elements.

[0033] refer to Figure 1 According to some embodiments, user 100 provides an input image to image processing apparatus 110 via user device 105 and cloud 115. In some cases, image processing apparatus 110 generates a combined image based on the input image. The combined image includes a mask area on the input image indicating synthetically generated pixels. Image processing apparatus 110 displays the combined image to user 100 on user device 105 (e.g., via cloud 115).

[0034] In some embodiments, the image processing device 110 generates annotation information indicating whether each pixel in the input image is synthetically generated. In some embodiments, the synthetically generated pixels or the likelihood of synthetically generated pixels are represented in a heat map, where the heat map is combined with the input image into a combined image. In some embodiments, the image processing device 110 displays the combined image to the user 100 via the user device 105 and the cloud.

[0035] User device 105 may be a personal computer, laptop computer, mainframe computer, palmtop computer, personal assistant, mobile device, or any other suitable processing device. In some examples, user device 105 includes software in conjunction with an image detection application. In some examples, the image detection application on user device 105 may include the functionality of image processing device 110.

[0036] The user interface may enable user 100 to interact with user device 105. In some embodiments, the user interface may include an audio device (such as an external speaker system), an external display device (such as a display screen), or an input device (e.g., a remote control device that interfaces with the user interface directly or through an I / O controller module). In some cases, the user interface may be a graphical user interface (GUI). In some examples, the user interface may be represented in code, where the code is sent to user device 105 and rendered locally by a browser. Reference Figure 2 The process of using the image processing device 110 is further described.

[0037] The image processing device 110 is a reference Figure 5 and Fig.11 Examples of corresponding elements described, or including references Figure 5 and Fig.11 According to some aspects, the image processing device 110 includes a computer-implemented network including a machine learning model, an encoder, a decoder, and a class sampler. The image processing device 110 also includes a processor unit, a memory unit, an I / O module, and a training component. In some embodiments, the image processing device 110 also includes a reference Fig.11 In addition, the image processing apparatus 110 communicates with the user device 105 and the database 125 via the cloud 115. Figure 2 and Figure 4 Further details regarding the operation of the image processing apparatus 110 are provided.

[0038] In some cases, the image processing device 110 is implemented on a server. The server provides one or more functions to users linked through one or more various networks. In some cases, the server includes a single microprocessor board containing a microprocessor responsible for controlling all aspects of the server. In some cases, the server uses a microprocessor and a protocol to exchange data with other devices / users on one or more networks via hypertext transfer protocol (HTTP) and simple mail transfer protocol (SMTP), although other protocols such as file transfer protocol (FTP) and simple network management protocol (SNMP) can also be used. In some cases, the server is configured to send and receive files (e.g., for displaying web pages) in hypertext markup language (HTML) format. In various embodiments, the server includes a general computing device, a personal computer, a laptop computer, a mainframe computer, a supercomputer, or any other suitable processing device.

[0039] Cloud 115 is a computer network configured to provide on-demand availability of computer system resources (such as data storage and computing power). In some examples, cloud 115 provides resources without active management by a user (e.g., user 100). The term "cloud" is sometimes used to describe a data center that is available to many users via the Internet. The functions of some large cloud networks are distributed over multiple locations in a central server. If a server has a direct or close connection with a user, the server is designated as an edge server. In some cases, cloud 115 is limited to a single organization. In other examples, cloud 115 is available to many organizations. In one example, cloud 115 includes a multi-layer communication network including multiple edge routers and core routers. In another example, cloud 115 is based on a local collection of switches in a single physical location.

[0040] Database 120 is an organized collection of data. For example, database 120 stores data in a specific format called a schema. Database 120 can be constructed as a single database, a distributed database, multiple distributed databases, or an emergency backup database. In some cases, a database controller can manage data storage and processing in database 120. In some cases, a user (e.g., user 100) interacts with the database controller. In other cases, the database controller can operate automatically without user interaction.

[0041] Figure 2 An example of a method 200 for detecting a composite image according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. In general, the operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0042] refer to Figure 2 According to some aspects, an image processing device (such as reference Figure 1 and Figure 5 The image processing apparatus described herein generates a combined image based on an input image by generating annotation information indicating whether each pixel in the input image is synthetically generated and by generating a combined image based on the annotation information. In some aspects, the combined image shows a synthetically generated area of ​​the input image. For example, the synthetically generated area includes pixels indicated as synthetically generated. For example, the combined image includes one or more portions of the input image. In some examples, the one or more portions are separated.

[0043] In some embodiments, the image processing device is trained by combining a portion of a real image from the training data and a portion of a synthetically generated image to obtain a combined training image. The image processing device is also trained based on the combined training image to detect synthetically generated pixels. In some embodiments, the image processing device is trained using the training data (including real images and synthetic images) in a chronological order (e.g., the image processing device is trained using a first training data set generated from a first image generator, and the first training data set is updated using a second data set generated from a second image generator, where the second generator is later than the first image generator). By training the image processing device in this order, the image processing device is able to detect synthetic images generated from an image generator without corresponding training data (e.g., real images).

[0044] At operation 205, the system provides an input image. In some cases, the operation of this step involves referring to Figure 1 Describes the user, or can be referenced by Figure 1 For example, in some cases, a user performs a Figure 1 The user interface provided by the image processing device on the user equipment described in the present invention provides a user interface to the image processing device (for example, reference Figure 1 The image processing apparatus described herein provides an input image. In some cases, the input image is a patched image, which includes a synthetically generated area and a real area. In some cases, the input image is a synthetically generated image.

[0045] In operation 210, the system generates annotation information based on the image. In some cases, the operation of this step involves referring to Figure 1 and Figure 5 The image processing device described herein may be referred to as Figure 1 and Figure 5 The image processing apparatus described above is used to perform. For example, in some cases, an encoder is used to encode an input image to generate an image embedding. A decoder is used to decode the image embedding to obtain annotation information.

[0046] In operation 215, the system detects whether the input image is a composite image based on the annotation information. In some cases, the operation of this step involves referring to Figure 1 and Figure 5 The image processing device described herein may be referred to as Figure 1 and Figure 5 For example, in some cases, the annotation information includes the likelihood of whether a pixel in the input image is synthetically generated.

[0047] At operation 220, the system outputs the test results. In some cases, the operation of this step involves reference to Figure 1 and Figure 5 The image processing device described herein may be referred to as Figure 1 and Figure 5 In some cases, the image processing apparatus outputs the combined image to a user via a user interface provided on a user device.

[0048] Figure 3 An example of detecting a synthesized region of an inpainted image according to aspects of the present disclosure is shown. The example shown includes a ground truth image 305 , a synthesized image 310 , a ground truth mask 315 , and a combined image 320 .

[0049] refer to Figure 3 , machine learning models (e.g., ref. Figure 5 The machine learning model described in the example above receives the synthesized image 310 and generates a combined image 320. The synthesized image 310 is generated by an “unseen” image generator (e.g., Firefly inpainting model) is generated based on the ground truth image 305. For example, the machine learning model has not yet utilized Firefly inpainting model is trained on a dataset that includes generated / synthesized images, ground truth masks, and ground truth images. In some cases, The Firefly inpainting model generates a ground truth mask 315 depicting synthetically generated regions / pixels. Inpainting refers to a method of filling in missing or damaged portions of an image using information derived from surrounding pixels. In some examples, inpainting is used to produce a visually coherent finish that is consistent with the surrounding area. The combined image 320 generated by the machine learning model includes multiple mask regions depicting synthetically generated pixels. In some cases, the mask regions are represented by heat maps. Figure 3 It shows that, in the absence of a corresponding training data set in the machine learning model according to aspects of the present disclosure, the combined image 320 generated by the machine learning model successfully captures the Generated pixels in the synthetic image 310 generated by the Firefly inpainting model. For example, the masked area in the combined image 320 (including the synthetically generated pixels) and the masked area in the ground truth mask 315 are substantially the same.

[0050] Figure 4An example of a method 400 for detecting a composite area of ​​an image at a pixel level according to various aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. In general, the operations are performed according to the methods and processes described in accordance with various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in combination with other operations.

[0051] In operation 405, the system obtains an input image comprising a set of pixels. In some cases, the operation of this step involves referring to Figure 5 and Fig. 9 The machine learning model described here can be referenced by Figure 5 and Fig. 9 Describe the machine learning model to execute.

[0052] According to some embodiments of the present disclosure, the input image is an image generated from a generative machine learning model. For example, the machine learning model receives generated images from 14 different generative machine learning models released between June 2020 and March 2023. These generative machine learning models are grouped into a first set for training machine learning models and a second set for testing machine learning models. The first generative machine learning model set includes DDPM (denoising diffusion probabilistic model), DDIM (denoising diffusion implicit model), GLIDE (Guided Language to Image Diffusion for Generation and Editing), LDM (latent diffusion model), Dalle 2, Midj v2, RDM (retrievalaugmented diffusion model), SD (stable diffusion) v1, Midj v3 and SD v2. The second set of generative machine learning models includes DiT (diffusion model with transformer), Midj v4, Midj v5, and Firefly. The input image may include an image generated from, for example, Firefly. In some cases, the input image includes one or more synthetically generated regions. For example, a region of the input image may include synthetically generated pixels, and the remaining region of the input image may include non-artificial (or real) pixels.

[0053] At operation 410, the system generates annotation information using the machine learning model, the annotation information indicating whether each pixel in the pixel set is synthetically generated. In some cases, the operation of this step involves referring to Figure 5 and Fig. 9 The machine learning model described here can be referenced by Figure 5 and Fig. 9The machine learning model described in the present invention is performed. In some embodiments, the machine learning model is trained to identify artifacts in each pixel and generate annotation information based on the artifacts. Each generative machine learning model generates a synthetic image with detectable artifacts based on the model architecture. For example, generative machine learning models with pixel diffusion architecture are grouped, including DDPM, DDIM, and GLIDE. Generative machine learning models with latent diffusion models (LDM) are also grouped. Each of these generative machine learning model groups can include detectable artifacts.

[0054] At operation 415, the system generates a combined image based on the annotation information, wherein the combined image shows the synthetically generated area of ​​the input image. In some cases, the operation of this step involves referring to Figure 5 and Fig. 9 The machine learning model described here can be referenced by Figure 5 and Fig. 9 In some embodiments, the machine learning model generates a mask representing an area of ​​synthetically generated pixels. In some cases, the mask includes a heat map indicating the likelihood that each pixel in the plurality of pixels is synthetically generated. In some cases, a combined image is displayed to a user. The combined image is a reference image. Figure 3 Examples of corresponding elements described, or including references Figure 3 Describes aspects of the corresponding elements.

[0055] Network Architecture

[0056] exist Figure 5 In the invention, an apparatus, system and method for image processing are described. One or more aspects of the apparatus, system and method include: at least one processor; at least one memory including instructions executable by at least one processor; and a machine learning model including parameters stored in at least one memory, wherein the machine learning model is trained to generate annotation information indicating whether each of a plurality of pixels of an input image is synthetically generated based on training data from a plurality of generative machine learning models.

[0057] Some examples of the apparatus, system, and method also include generating a combined image based on the annotation information, wherein the combined image shows a synthetically generated region of the input image. Some examples of the apparatus, system, and method also include an encoder configured to encode the input image to obtain an image embedding. Some examples also include a decoder configured to decode the image embedding to obtain the annotation information.

[0058] In some aspects, the encoder includes a machine learning model including a convolutional neural network (CNN). In some aspects, the decoder includes a fully convolutional network (FCN).

[0059] Some examples of the apparatus, system, and method also include a training component configured to train the machine learning model. In some aspects, the training data includes a partially synthesized training image, which includes a real part and a synthesized part, and a reference truth annotation data indicating the real part and the synthesized part.

[0060] Figure 5 FIG. 5 shows an example of an image processing apparatus 500 according to aspects of the present disclosure. Figure 1 Examples of corresponding elements described, or including references Figure 1 In one aspect, the image processing device 500 includes a processor unit 505, an I / O module 510, a memory unit 515, and a training component 540. In one aspect, the memory unit 515 includes a machine learning model 520, an encoder 525, a decoder 530, and a class sampler 535.

[0061] The processor unit 505 is an intelligent hardware device, such as a general processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof. In some cases, the processor unit 505 is configured to operate a memory array using a memory controller. In other cases, the memory controller is integrated into the processor. In some cases, the processor unit 505 is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor unit 505 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. The processor unit 505 is a reference Fig.11 An example of a processor described herein, or including reference Fig.11 Describes aspects of the processor.

[0062] I / O module 510 (e.g., input / output interface) may include an I / O controller. An I / O controller may manage input signals and output signals of a device. An I / O controller may also manage peripheral devices that are not integrated into the device. In some cases, an I / O controller may represent a physical connection or port to an external peripheral device. In some cases, an I / O controller may utilize an operating system, such as Or another known operating system. In other cases, the I / O controller may represent or interact with a modem, keyboard, mouse, touch screen, or similar device. In some cases, the I / O controller may be implemented as part of the processor. In some cases, a user may interact with the device via the I / O controller or via hardware components controlled by the I / O controller.

[0063] In some examples, the I / O module 510 includes a user interface. The user interface can enable a user to interact with the device. In some embodiments, the user interface can include an audio device (such as an external speaker system), an external display device (such as a display screen) or an input device (for example, a remote control device directly or through an I / O controller module and docked with the user interface). In some cases, the user interface can be a graphical user interface (GUI). In some examples, the communication interface operates at the boundary between the communication entity and the channel, and can also record and process communication. A communication interface is provided herein so that a processing system can be coupled to a transceiver (for example, a transmitter and / or a receiver). In some examples, a transceiver is configured to transmit (or send) and receive signals for a communication device via an antenna.

[0064] Examples of memory unit 515 include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory unit 515 include solid-state memory and hard disk drives. In some examples, memory unit 515 is used to store computer-readable computer executable software including instructions that, when executed, cause the processor to perform the various functions described herein. In some cases, memory unit 515 includes a basic input / output system (BIOS) that controls basic hardware or software operations (such as interaction with peripheral components or devices), etc. In some cases, a memory controller operates a memory cell. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, the storage cells within memory unit 515 store information in the form of logical states. Memory unit 515 is a reference to Fig.11 An example of a memory subsystem described herein, or including reference to Fig.11 Aspects of the memory subsystem are described.

[0065] In some examples, memory unit 515 includes instructions executable by at least one processor unit 505. Memory unit 515 includes machine learning model 520 or stores parameters of the machine learning model.

[0066] According to some aspects, the machine learning model 520 obtains an input image including a set of pixels. In some examples, the machine learning model 520 generates annotation information indicating whether each pixel in the set of pixels is synthetically generated. In some examples, the machine learning model 520 generates a combined image based on the annotation information, wherein the combined image shows a synthetically generated area of ​​the input image.

[0067] In some aspects, the machine learning model 520 is trained using training data including synthetic images generated by a collection of generative machine learning models. In some aspects, the collection of generative machine learning models includes a collection of diffusion models with different architectures. In some aspects, the annotation information includes a heat map indicating the likelihood that each pixel in the collection of pixels is synthetically generated. In some examples, the machine learning model 520 combines an input image with a mask representing the annotation information.

[0068] According to some aspects, the machine learning model 520 obtains training data, the training data comprising a set of real images and a set of synthetic images generated at least in part by a set of different generative machine learning models. In some examples, the machine learning model 520 divides the training data into a set of training batches based on which generative machine learning model in the set of different generative machine learning models generated each synthetic image in the set of synthetic images. In some examples, the machine learning model 520 sorts the set of different generative machine learning models according to a date corresponding to each generative machine learning model in the set of different generative machine learning models, respectively, wherein the training data is divided based on the sorting.

[0069] In some examples, the machine learning model 520 obtains a training real image and a training synthetic image. In some examples, the machine learning model 520 combines at least a portion of the training real image with at least a portion of the training synthetic image to obtain a combined image, wherein the training data includes the combined image. In some aspects, the machine learning model 520 is trained to generate annotation information indicating whether each pixel in the pixel set is synthetically generated based on the combined image. In some aspects, the annotation information includes a heat map having a per-pixel value corresponding to the pixel set.

[0070] In some examples, machine learning model 520 identifies additional generative machine learning models after training machine learning model 520. In some examples, machine learning model 520 obtains additional training data generated by the additional generative machine learning models.

[0071] According to some aspects, the machine learning model 520 includes parameters stored in at least one memory, wherein the machine learning model 520 is trained to generate annotation information indicating whether each of a plurality of pixels of an input image is synthetically generated based on training data from a plurality of generative machine learning models. In some examples, the machine learning model 520 generates a combined image based on the annotation information, wherein the combined image shows a synthetically generated region of the input image.

[0072] According to some aspects, the training data includes partially synthesized training images including real parts and synthesized parts, and reference truth annotation data indicating the real parts and the synthesized parts. The machine learning model 520 is a reference Fig. 9 Examples of corresponding elements described, or including references Fig. 9 Describes aspects of the corresponding elements.

[0073] According to some aspects, the encoder 525 encodes the input image to obtain the image embedding. According to some aspects, the encoder 525 is configured to encode the input image to obtain the image embedding. In some aspects, the encoder 525 includes a machine learning model including a convolutional neural network (CNN).

[0074] CNN is a type of neural network commonly used in computer vision or image classification systems. In some cases, CNN can be used to process digital images with minimal preprocessing. CNN can be characterized by the use of convolutional (or cross-correlation) hidden layers. These layers apply convolution operations to the input and then signal the results to the next layer. Each convolution node can process data in a limited input domain (e.g., receptive field). During the forward pass of the CNN, the filter at each layer can be convolved across the input volume to calculate the dot product between the filter and the input. During the training process, the filter can be modified so that the filter activates when the filter detects specific features in the input.

[0075] According to some aspects, decoder 530 decodes the image embedding to obtain the annotation information. According to some aspects, decoder 530 is configured to decode the image embedding to obtain the annotation information. In some aspects, decoder 530 comprises a fully convolutional network (FCN).

[0076] FCN is a class of deep neural networks used primarily in computer vision or for image processing tasks. FCN receives an input image of arbitrary size and outputs an image of the same spatial dimensions, where each pixel in the output image corresponds to a pixel in the input image and contains features and / or class information. FCN includes convolutional layers to extract features from the input image. FCN includes skip connections to combine feature maps from different levels of the network. The final layer of the FCN includes a convolutional layer with a kernel size that matches the desired output resolution. The final layer produces pixel-by-pixel predictions, where each pixel is associated with a class label or feature.

[0077] According to some aspects, the category sampler 535 uses the category sampler 535 to balance the distribution of training data based on a set of categories corresponding to a set of different generative machine learning models, respectively.

[0078] According to some aspects, training component 540 trains machine learning model 520 based on training data to detect whether an image is a synthetic image. In some examples, training component 540 updates machine learning model 520 based on additional training data. According to some aspects, training component 540 is configured to train machine learning model 520.

[0079] According to some embodiments, the training component 540 is implemented as software stored in the memory unit 515 and executable by a processor in the processor unit 505 of a separate computing device, firmware in a separate computing device, one or more hardware circuits of a separate computing device, or a combination thereof. In some examples, the training component 540 is part of another device different from the image processing device 500 and communicates with the image processing device 500. In some examples, the training component 540 is part of the image processing device 500.

[0080] Image Detection

[0081] Figure 6 An example of a method 600 for detecting a composite area of ​​an image according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. In general, the operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0082] At operation 605, the system encodes the input image to obtain an image embedding. In some cases, the operation of this step involves referring to Figure 5 The encoder described in Figure 5For example, an image embedding is a numerical representation of an image in a high-dimensional vector space. Image embeddings contain semantic information and features in a format that can be used by machine learning models (see Figure 5 described).

[0083] In operation 610, the system decodes the image embedding to obtain the annotation information. In some cases, the operation of this step involves referring to Figure 5 The decoder described in Figure 5 In some cases, the decoder decodes the image embedding and generates an output image, wherein the output image includes annotation information. For example, the annotation information indicates the likelihood of whether a pixel among a plurality of pixels in the input image is synthetically generated.

[0084] At operation 615, the system combines the input image with a mask representing the annotation information. In some cases, the operation of this step involves reference Figure 5 and Fig. 9 The machine learning model described here can be referenced by Figure 5 and Fig. 9 In some cases, the mask comprises a synthetically generated pixel region in the input image. In some cases, the mask is represented by a heat map.

[0085] Training and Evaluation

[0086] exist Figure 7-10 In the invention, a method, apparatus, non-transitory computer-readable medium, and system for image processing are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include: obtaining training data, the training data including a plurality of real images and a plurality of synthetic images generated at least in part by a plurality of different generative machine learning models; and training a machine learning model based on the training data to detect whether an image is a synthetic image.

[0087] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include: dividing the training data into a plurality of training batches based on which of the plurality of different generative machine learning models generated each of the plurality of synthetic images. Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include: sorting the plurality of different generative machine learning models according to a date corresponding to each of the plurality of different generative machine learning models, respectively, wherein the training data is divided based on the sorting.

[0088] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include obtaining a training real image and a training synthetic image. Some examples also include combining at least a portion of the training real image with at least a portion of the training synthetic image to obtain a combined image, wherein the training data includes the combined image. In some aspects, the machine learning model is trained to generate annotation information indicating whether each pixel in the plurality of pixels is synthetically generated based on the combined image. In some aspects, the annotation information includes a heat map having per-pixel values ​​corresponding to the plurality of pixels.

[0089] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include identifying an additional generative machine learning model after training the machine learning model. Some examples further include obtaining additional training data generated by the additional generative machine learning model. Some examples further include updating the machine learning model based on the additional training data. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include: using a category sampler to balance the distribution of the training data based on multiple categories corresponding to multiple different generative machine learning models, respectively.

[0090] Figure 7 An example of a method 700 for training a machine learning model according to various aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. In general, the operations are performed according to the methods and processes described in accordance with various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0091] At operation 705, the system obtains training data, the training data comprising a set of real images and a set of synthetic images generated at least in part by a set of different generative machine learning models. In some cases, the operation of this step involves reference Figure 5 and Fig. 9 The machine learning model described here can be referenced by Figure 5 and Fig. 9 Describe the machine learning model to execute.

[0092] According to an embodiment of the present disclosure, the machine learning model receives generated images from 14 different generative machine learning models (e.g., released between June 2020 and March 2023). These different generative machine learning models reflect the rapid development of realistic synthetic image generation. In some embodiments, the different generative machine learning models are grouped by the architecture of each generative machine learning model that generates the final pixel.

[0093] The training dataset is used to evaluate the generalization of the detector to unseen generative machine learning models. For example, the generated images and the corresponding release date of each generative machine learning model are used to simulate an online learning setting. For publicly available models, the model release date from the repository is used. For products that do not have model release dates but can be queried using the API (Application Programming Interface) (e.g., Midjourney and Firefly), the date of querying the generative machine learning model is used. The dataset includes a total of 570,221 images (including 405,862 training images, 48,057 ground truth images, and 116,302 test images). Reference Fig. 9 Describe further details about the training data.

[0094] At operation 710, the system trains a machine learning model based on the training data to detect whether the image is a synthetic image. In some cases, the operation of this step involves referring to Figure 5 The training components described in this document may be referenced by Figure 5 The training components described are executed.

[0095] According to some embodiments of the present disclosure, the machine learning model progressively trains a binary classifier with a cross entropy loss to distinguish between "real" images of natural origin and images generated by AI (e.g., synthetic images). In some cases, a simple classifier can be generalized across different generators (e.g., different generative machine learning models). According to some embodiments, the machine learning model includes a CNN architecture and ResNet-50, and the machine learning model is pre-trained on ImageNet. In some embodiments, the machine learning model includes an encoder including a convolutional neural network (CNN) and a decoder including a fully convolutional network (FCN).

[0096] In some embodiments, the machine learning model applies a weighted Dice loss to compensate for data imbalance. During training, when training data is not available, full images can be utilized for pixel-by-pixel labeling by applying CutMix augmentation. CutMix blending is a simple way to cut and paste blocks of one image into another. When a patch dataset is available, the machine learning model detects synthetically generated patched images using random masks. Reference Fig. 9 Describes further details about training machine learning models.

[0097] According to some aspects of the present disclosure, a machine learning model is trained using SDv1. For example, when trained using training data generated from SDv1 including fully synthetic images, the machine learning model is able to detect inpainted images generated by SDv1 with an accuracy of approximately 77.6%. However, when the machine learning model is further trained and updated using the modified training data by applying CutMix augmentation, performance is improved in detecting inpainted images generated by SDv1 with an accuracy of approximately 83.6%.

[0098] In some cases, when the machine learning model was trained using training data including patched images generated from SDv1 patching, the machine learning model was able to detect the patched images generated by SDv1 with 99.0% accuracy. However, when the machine learning model was further trained and updated using the modified training data by applying CutMix augmentation, the performance was improved in detecting the patched images generated by SDv1 with approximately 99.2% accuracy.

[0099] Figure 8 An example of a method 800 for training a machine learning model according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. In general, the operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0100] At operation 805, the system identifies additional generative machine learning models after training the machine learning models. In some cases, the operation of this step involves reference to Figure 5 and Fig. 9 The machine learning model described here can be referenced by Figure 5 and Fig. 9 The additional generative machine learning model is executed based on the machine learning model described. For example, the machine learning model is trained based on a training dataset including a first set of real images and a first set of synthetic images generated by a first generative machine learning model. The additional generative machine learning model is released on a date that is later than the release date of the first generative machine learning model. In some cases, the additional generative machine learning model has the same architecture as the first generative machine learning model. In some cases, the additional generative machine learning model has a different architecture. Reference Fig. 9 and Fig.10 Describes further details about additional generative machine learning models.

[0101] At operation 810, the system obtains additional training data generated by additional generative machine learning models. In some cases, the operation of this step involves referring to Figure 5 and Fig. 9 The machine learning model described here can be referenced by Figure 5 and Fig. 9 The additional training data may be performed using the machine learning model described herein. For example, the additional training data may include an additional set of real images and an additional set of synthetic images generated at least in part by the additional generative machine learning model. In some cases, the additional synthetic images may include fully synthetic images. In some cases, the additional synthetic images may include inpainted images in which a portion of the image is synthetically generated and the remainder of the image is a real image. Fig. 9 and Fig.10 Describes further details about additional generative machine learning models.

[0102] At operation 815, the system updates the machine learning model based on the additional training data. In some cases, the operation of this step involves referring to Figure 5 The training components described in this document may be referenced by Figure 5 The training component described herein is performed. For example, when the machine learning model is updated based on additional training data, the machine learning model is continuously trained from the previous model weights including the existing data set. As a result, the training data set is expanded and the machine learning model is enabled to detect synthetic images generated from the "unseen" generative machine learning model.

[0103] Fig. 9 An example of training a machine learning model 900 according to aspects of the present disclosure is shown. The example shown includes a machine learning model 900, a timeline 905, a baseline data set 910, a first training data set 915, a second training data set 920, and a test set 925. The machine learning model 900 is a reference Figure 5 Examples of corresponding elements described, or including references Figure 5 Describes aspects of the corresponding elements.

[0104] refer to Fig. 9, the machine learning model 900 is progressively trained using different data sets from different generative machine learning models. For example, the machine learning model 900 includes a baseline data set 910. In some cases, the baseline data set 910 includes non-generated images from LAION (Large-scale AIOpen Network)-400M at the beginning of training and remains unchanged throughout the training process. The machine learning model 900 also receives a first training data set 915 generated from a first generative machine learning model (e.g., DDPM). The first training data set 915 includes a set of first real images and a set of first synthetic images. In some cases, the set of first synthetic images includes synthetically generated pixels.

[0105] The machine learning model 900 also receives a second training data set 920 generated from a second generative machine learning model (e.g., DDIM). In some cases, the second generative machine learning model is referred to as an additional generative machine learning model. The release date of the additional generative machine learning model is later than the release date of the first generative machine learning model. For example, the training data of each generative model in the set of different generative models is provided to the machine learning model 900 in chronological order based on the timeline 905. Similarly, the second training data set 920 includes a set of second real images and a set of second synthetic images.

[0106] According to some embodiments of the present disclosure, the second generative machine learning model includes an architecture that is different from the architecture in the first generative machine learning model. For example, the second generative machine learning model is a latent diffusion model (LDM). By sorting the training data based on the release date of each generative machine learning model, the machine learning model 900 is able to simulate a real-world learning setting. In addition, the machine learning model 900 is able to detect synthetic images generated from an untrained generative machine learning model without a corresponding training-based training data set.

[0107] In some embodiments, the test set 925 includes generated images from generative machine learning models (e.g., DiT, Midj v4, Midj v5, and Firefly). The machine learning model 900 obtains the generated images in the test set 925 as input images. The machine learning model 900 outputs a detection indicating whether the input image is a synthetic image. In some cases, the input images in the test set 925 include inpainted images, in which an area of ​​the input image is synthetically generated and the remaining area of ​​the input image is a real image. The machine learning model 900 is trained to detect synthetic pixels based on the progressive learning method described in the present disclosure. The machine learning model 900 detects and outputs a combined image including a mask area representing a synthetic pixel and an input image.

[0108] According to some embodiments, machine learning model 900 receives generated images from 14 different generative machine learning models (shown on timeline 905) released between June 2020 and March 2023. These different generative machine learning models reflect the rapid development of realistic synthetic image generation. In some embodiments, the different generative machine learning models are grouped by the architecture of each generative machine learning model that generates the final pixel.

[0109] According to an embodiment of the present disclosure, 14 generative machine learning models are grouped based on architecture. For example, generative machine learning models with pixel diffusion architecture are first grouped, including DDPM, DDIM, and GLIDE. These generative machine learning models have U-Net and diffusion-based target architectures. For example, DDPM and DDIM train unconditional models on smaller datasets, while GLIDE generates synthetic images based on text-to-image generation.

[0110] In addition, DALL E 2 (a high-profile model from OpenAI) is also grouped. At the time of data collection, DALL E 2 was available through a web interface and does not include large-scale publicly available datasets. More precisely, the corresponding dataset was obtained via DALL E 2 Reddit by retaining 1024×1024 images (to filter out irrelevant content such as memes). In some cases, the web interface generates a watermark that can be easily identified by the classifier and crops off the bottom 16 pixels of the image.

[0111] Secondly, the largest family of generative machine learning models that follow is the latent diffusion model (LDM). These generative machine learning models use U-Net to perform diffusion in the latent domain, and then use a decoder trained as part of a variational autoencoder to decode the latent signal in combination with GAN (Generative Adversarial Network) and LPIPS (Learned Perceptual Image Patch Similarity) perceptual loss. These generative machine learning models include "stable diffusion" and "stable diffusion" 2, which are scaled-up versions of latent diffusion trained on large-scale network data, containing multiple sub-versions. In addition, some generative machine learning models use transformers based on ViT (Vision Transformer) to change the diffusion U-Net, which improves the identification task.

[0112] According to aspects of the present disclosure, unpublished generative machine learning models are used as test sets. For example, according to aspects of the present disclosure, images from Midjourney and Adobe Firefly are tested on machine learning models. In some cases, images generated by Midjourney are obtained by crawling the Discord API. Since these generative machine learning models do not include release data, these models are sorted based on crawl date. In some cases, Adobe Firefly test images are obtained through a web interface without a watermark.

[0113] Additionally, for generative machine learning models including GLIDE, LDM, RDM, Firefly (all datasets), and "Stable Diffusion" (both training and ground truth datasets), prompts from DiffusionDB are used as input to generate a set of synthetic images. Prompts can include textual prompts, color prompts, and image prompts. For the "Stable Diffusion" dataset, prompts from various web sources are used. Unique prompts are sampled so that the training, ground truth, and test datasets do not overlap.

[0114] In some cases, the composite image includes a composite of AI-generated pixels and an image from a traditional source. In some cases, the composite image is generated by "painting," where masked areas of an image are seamlessly filled in with generated content. In one aspect, the machine learning model receives input from Firefly and "stable diffusion" inpainting models. In some cases, ground truth masks may be difficult to obtain in real-world settings (e.g., using closed models). Accordingly, the machine learning model applies Cut-Mix to the full image to obtain pixel-by-pixel labels. The machine learning model is also trained by using synthetic images (e.g., inpainted images) generated from other generative machine learning models (e.g., SDv1 inpainting model, SDv2 inpainting model, and Firefly).

[0115] In one aspect, the machine learning model receives three patched datasets, including “stable diffusion” (SDv1 and SDv2) and Firefly. In some embodiments, the training dataset includes the model release date, input source (e.g., LAION), mask pixel distribution, and dataset size. For example, the machine learning model samples input images and corresponding prompts from the LAION-400M dataset. In some cases, the machine learning model resizes the short side of the image to 512 pixels, applies a center crop, and generates a mask corresponding to the image. The machine learning model generates masks covering 15% to 35% of each image using random overlapping strokes and shapes (where a portion of the shapes are based on the image content) or using dilation and jitter augmentation.

[0116] In some cases, a complete image dataset including modified images is generated by using equal-sized samples of LAION input and some generative machine learning models. The machine learning model maintains the quality of the non-masked area and isolates the generated pixels from the original pixels by copying the original image back into the non-masked area. Accordingly, the modified image does not contain visible seams.

[0117] Fig.10 An example of detecting synthetically generated images based on a trained machine learning model according to various aspects of the present disclosure is shown. The example shown includes a generative model 1000 and a test set 1005.

[0118] refer to Fig.10 , the machine learning model is progressively trained by each generative model 1000. For example, the machine learning model is first trained using training data generated from a first generative model (e.g., DDPM). In one aspect, the test set 1005 includes synthetic images generated from the generative model 1000. The trained machine learning model detects the synthetic images from the test set 1005. For example, when the machine learning model is trained using DDPM, the machine learning model is able to detect synthetic images generated from DDPM and DDIM with an accuracy of approximately 0.96. However, the machine learning model cannot detect other synthetic images generated by different generative models with high accuracy.

[0119] The machine learning model is updated based on the second generative model in the generative model 1000. For example, when the machine learning model is trained using training data generated from the second generative model (e.g., LDM), the updated machine learning model can increase the accuracy of detecting synthetic images. For example, the updated machine learning model can detect synthetic images generated from Midj v2 with an accuracy of 0.83, while before the update, the machine learning model can detect synthetic images generated from Midj v2 with an accuracy of 0.02-0.3. For example, the updated machine learning model can detect synthetic images generated from Firefly with an accuracy of 0.3, while before the update, the machine learning model cannot detect synthetic images generated from Firefly (e.g., the accuracy is 0).

[0120] like Fig.10 It is shown that when a machine learning model is progressively trained by updating the training dataset, the accuracy increases when detecting synthetic images generated from a generative model that has not yet been trained. The machine learning model is a reference Figure 5 and Fig. 9 Examples of the same elements described, or including references Figure 5 and Fig. 9 Aspects of the same elements described.

[0121] Fig.10 The results of detecting synthetic images in an online setting are further represented. Generative model 1000 represents the training progress of the machine learning model over time. Each column represents a step in the learning process, where the classifier is trained on all models up to that cell. The generative models are sorted by release date. For example, the "LDM" column includes training data generated from DDPM, DDIM, GLIDE, and LDM. The training data includes a set of real images (or benchmark truth images) and a set of synthetic images. Test set 1005 represents a set of generative machine learning models with release dates sorted from bottom to top (for example, the generative model at the bottom is the earliest released, and the generative model at the top is the latest released). Accordingly, the cell in the upper left triangle shows the generalization performance of the machine learning model to multiple generative machine learning models that have not yet been included in the training.

[0122] In one aspect, after initially training a machine learning model and updating the initially trained machine learning model with an additional dataset, the trained generative model (e.g., Fig.10 The performance of the 2D RGB-Wavelength Array (shown in the lower right) remains approximately the same level of accuracy. Fig.10As shown, the last column indicates that the machine learning model is trained on all generative models 1000, where the trained machine learning model produces near-perfect performance in the entire test set 1005. For example, a single detector may learn different artifacts from different generative machine learning models.

[0123] Fig.11 An example of a computing device 1100 for a machine learning model according to aspects of the present disclosure is shown. The example shown includes a computing device 1100, (multiple) processors, a memory subsystem 1110, a communication interface 1115, an I / O interface 1120, (multiple) user interface components, and a channel 1130. The machine learning model is a reference Figure 5 Examples of corresponding elements described, or including references Figure 5 Describes aspects of the corresponding elements.

[0124] In some embodiments, computing device 1100 is Figure 1 and Figure 5 An example of an image processing apparatus comprising Figure 1 and Figure 5 In some embodiments, the computing device 1100 includes one or more processors 1105, and the one or more processors 1105 can execute instructions stored in the memory subsystem 1110 to: obtain an input image including a plurality of pixels; generate annotation information using a machine learning model, the annotation information indicating whether each of the plurality of pixels is synthetically generated; and generate a combined image based on the annotation information, wherein the combined image shows a synthetically generated area of ​​the input image.

[0125] According to some embodiments, the computing device 1100 includes one or more processors 1105. In some cases, the processor is an intelligent hardware device, such as a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or a combination thereof. In some cases, the processor is configured to operate a memory array using a memory controller. In other cases, the memory controller is integrated into the processor. In some cases, the processor is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. The processor 1105 is a reference Figure 5 An example of a processor unit described, or including reference Figure 5 Aspects of a processor unit are described.

[0126] According to some embodiments, the memory subsystem 1110 includes one or more memory devices. Examples of memory devices include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable computer executable software including instructions that, when executed, cause the processor to perform the various functions described herein. In some cases, the memory contains a basic input / output system (BIOS) that controls basic hardware or software operations (such as interaction with peripheral components or devices), etc. In some cases, a memory controller operates storage cells. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, storage cells within the memory store information in the form of logical states. The memory subsystem 1110 is a reference Figure 5 An example of a memory cell described herein, or including reference Figure 5 Aspects of a memory cell are described.

[0127] According to some embodiments, the communication interface 1115 operates at the boundary between the communication entities (such as the computing device 1100, one or more user devices, the cloud, and one or more databases) and the channel 1130, and can record and process the communication. In some cases, the communication interface 1115 is provided to enable the processing system to couple to a transceiver (e.g., a transmitter and / or a receiver). In some examples, the transceiver is configured to transmit (or send) and receive signals for the communication device via an antenna.

[0128] According to some embodiments, I / O interface 1120 is controlled by an I / O controller to manage input signals and output signals of computing device 1100. In some cases, I / O interface 1120 manages peripheral devices that are not integrated into computing device 1100. In some cases, I / O interface 1120 represents a physical connection or port to an external peripheral device. In some cases, the I / O controller uses an operating system, such as or other known operating systems. In some cases, an I / O controller represents or interacts with a modem, keyboard, mouse, touch screen, or similar device. In some cases, an I / O controller is implemented as a component of a processor. In some cases, a user interacts with the device via an I / O interface 1120 or via a hardware component controlled by the I / O controller.

[0129] According to some embodiments, user interface component(s) 1125 enable a user to interact with computing device 1100. In some cases, user interface component 1125 includes an audio device (such as an external speaker system), an external display device (such as a display screen), an input device (e.g., a remote control device that interfaces with the user interface directly or through an I / O controller), or a combination thereof.

[0130] The performance of the apparatus, system and method of the present disclosure has been evaluated, and the results indicate that the embodiments of the present disclosure have achieved improved performance over the prior art (e.g., image detectors). Example experiments show that the machine learning model based on the present disclosure outperforms conventional image detectors.

[0131] The descriptions and drawings described herein represent example configurations and do not represent all implementations within the scope of the claims. For example, operations and steps may be rearranged, combined, or modified. In addition, structures and devices may be represented in the form of block diagrams to represent the relationships between components and avoid obscuring the concepts described. Similar components or features may have the same name, but may have different reference numerals corresponding to different drawings.

[0132] Some modifications to the present disclosure may be apparent to those skilled in the art, and the principles defined herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is in accordance with the broadest scope consistent with the principles and novel features disclosed herein.

[0133] The described methods may be implemented or performed by a device including a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general purpose processor may be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. The processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration). Therefore, the functions described herein may be implemented in hardware or software and may be performed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored in a computer readable medium in the form of instructions or code.

[0134] Computer-readable media include both non-transitory computer storage media and communication media (including any media that facilitate the transmission of code or data). Non-transitory storage media can be any available media that can be accessed by a computer. For example, non-transitory computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk (CD) or other optical disk storage device, magnetic disk storage device, or any other non-transitory medium for carrying or storing data or code.

[0135] In addition, a connecting component may be properly referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, or microwave signals) is used to transmit code or data from a website, server, or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology is included in the definition of medium. Combinations of media are also included within the scope of computer-readable media.

[0136] In this disclosure and the claims that follow, the word "or" indicates an inclusive list, such that a list such as X, Y, or Z means X or Y or Z or XY or XZ or YZ or XYZ. The phrase "based on" is also not used to represent a closed set of conditions. For example, a step described as "based on condition A" can be based on both condition A and condition B. In other words, the phrase "based on" should be interpreted to mean "based at least in part on." Additionally, the words "a" or "an" indicate "at least one."

Claims

1. A method for image processing, comprising: obtaining an input image comprising a plurality of pixels; generating annotation information using a machine learning model, the annotation information indicating whether each pixel of the plurality of pixels is synthetically generated; as well as A combined image is generated based on the annotation information, wherein the combined image shows a synthetically generated area of ​​the input image.

2. The method according to claim 1, wherein generating the annotation information comprises: encoding the input image to obtain an image embedding; as well as The image embedding is decoded to obtain the annotation information.

3. The method according to claim 1, wherein: The machine learning model is trained using training data comprising synthetic images generated by multiple generative machine learning models.

4. The method according to claim 3, wherein: The plurality of generative machine learning models include a plurality of diffusion models having different architectures.

5. The method according to claim 1, wherein: The annotation information includes a heat map indicating a likelihood that each pixel of the plurality of pixels is synthetically generated.

6. The method according to claim 1, further comprising: The input image is combined with a mask representing the annotation information.

7. A method for image processing, comprising: Obtaining training data, the training data comprising a plurality of real images and a plurality of synthetic images generated at least in part by a plurality of different generative machine learning models; as well as A machine learning model is trained based on the training data to detect whether an image is a synthetic image.

8. The method of claim 7, wherein training the machine learning model comprises: The training data is divided into a plurality of training batches based on which generative machine learning model among the plurality of different generative machine learning models generated each of the plurality of synthetic images.

9. The method according to claim 8, further comprising: The plurality of different generative machine learning models are sorted according to a date respectively corresponding to each of the plurality of different generative machine learning models, wherein the training data is divided based on the sorting.

10. The method according to claim 7, wherein obtaining the training data comprises: Obtaining real training images and synthetic training images; as well as At least a portion of the training real image is combined with at least a portion of the training synthetic image to obtain a combined image, wherein the training data includes the combined image.

11. The method according to claim 10, wherein: The machine learning model is trained to generate annotation information indicating whether each of a plurality of pixels is synthetically generated based on the combined image.

12. The method according to claim 11, wherein: The annotation information includes a heat map having per-pixel values ​​corresponding to the plurality of pixels.

13. The method according to claim 7, further comprising: identifying an additional generative machine learning model after training the machine learning model; obtaining additional training data generated by the additional generative machine learning model; as well as The machine learning model is updated based on the additional training data.

14. The method of claim 7, wherein training the machine learning model comprises: Based on a plurality of categories respectively corresponding to the plurality of different generative machine learning models, a category sampler is used to balance the distribution of the training data.

15. An apparatus comprising: at least one processor; at least one memory comprising instructions executable by the at least one processor; as well as A machine learning model comprising parameters stored in the at least one memory, wherein the machine learning model is trained to generate annotation information indicating whether each of a plurality of pixels of an input image is synthetically generated based on training data from a plurality of generative machine learning models.

16. The apparatus of claim 15, wherein the instructions are further executable to: A combined image is generated based on the annotation information, wherein the combined image shows a synthetically generated area of ​​the input image.

17. The apparatus of claim 15, wherein the machine learning model further comprises: an encoder configured to encode the input image to obtain an image embedding; as well as A decoder is configured to decode the image embedding to obtain the annotation information.

18. The device according to claim 17, wherein: The encoder includes a machine learning model including a convolutional neural network (CNN); and The decoder includes a fully convolutional network FCN.

19. The apparatus according to claim 15, further comprising: A training component is configured to train the machine learning model.

20. The apparatus of claim 15, wherein: The training data includes a partially synthesized training image including a real part and a synthesized part, and reference truth annotation data indicating the real part and the synthesized part.

Images