Apparatus and method for performing image authentication

Abstract

An electronic device may be provided to identify artifacts from an image processed via an image processor and / or an AI camera module using artificial intelligence (AI) technology to perform image authentication. An electronic device may be configured to: obtain an input image; obtaining a processed image of the input image via an AI-based model; generating authentication metadata indicating the dummy pixels that have been generated by the AI-based model; storing authentication metadata associated with the processed image in the at least one memory; and outputting the processed image with an indication of a dummy pixel.

Description

Technical Field

[0001] The present disclosure relates to an apparatus and method for identifying a pseudo pixel from an image processed via an image processor and / or an AI camera module using an artificial intelligence (AI) technology. Background Art

[0002] Digital visual media is one of the primary means of communication in today’s world. However, the widespread availability and ease of use of photo editing software often casts a shadow on the authenticity of digital visual information. While images suspected of having been tampered with using external photo editing applications are the subject of careful investigation, images that come straight out of a digital camera are considered pristine and outside of close scrutiny. Cameras use dedicated built-in hardware to process the light reaching the sensor and convert it into the final photo displayed to the user. While this processing has traditionally been based on signal processing, in recent years cameras have begun to employ neural or artificial intelligence (AI) processing instead. These AI modules can “hallucinate” image content, i.e., can generate scene content as textures or images that do not exist in the real scene. This means that the authenticity of images rendered by AI cameras cannot be guaranteed, a key point that is largely overlooked in modern digital image forensics.

[0003] To address this issue, there has been a need for a standard that camera manufacturers can adopt that allows verification of the authenticity of images captured by the camera. Summary of the invention

[0004] Technical Solution

[0005] According to one aspect of the present disclosure, there is provided an electronic device for performing image authentication on an image captured by an artificial intelligence (AI) camera module or processed by an AI image signal processor (ISP). The electronic device may include: at least one memory storing instructions; and at least one processor configured to execute instructions to perform the following operations: obtaining an input image, obtaining a processed image of the input image via an AI-based model; generating authentication metadata indicating pseudo pixels that have been generated by the AI-based model, storing the authentication metadata associated with the processed image in the at least one memory; and outputting the processed image with an indication of pseudo pixels.

[0006] According to one aspect of the present disclosure, there is provided a method for performing image authentication on an image captured by an artificial intelligence (AI) camera module or processed by an AI image signal processor (ISP). The method may include: obtaining an input image; obtaining a processed image of the original image via an artificial intelligence (AI) based model, generating authentication metadata indicating pseudo pixels that have been generated by the AI ​​based model, storing the authentication metadata associated with the processed image in at least one memory; and outputting the processed image with an indication of pseudo pixels.

[0007] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing instructions is provided, wherein the instructions are executed by at least one processor to perform operations corresponding to the method described in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other aspects, features and aspects of the embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0009] Figure 1A is a diagram of an electronic device for performing image authentication according to an embodiment of the present disclosure;

[0010] Figure 1B is a block diagram of an electronic device for performing image authentication according to an embodiment of the present disclosure;

[0011] Figure 2A is a diagram showing the structure of an AI-based image processing model according to an embodiment of the present disclosure;

[0012] Figure 2B is a diagram showing the structure of an encoder of an AI-based image processing model according to an embodiment of the present disclosure;

[0013] Figure 2C is a diagram showing the structure of a decoder of an AI-based image processing model according to an embodiment of the present disclosure;

[0014] Figure 3 is a diagram showing the structure of an AI-based image processing model according to an embodiment of the present disclosure;

[0015] Figure 4A is a diagram showing the structure of an AI-based image processing model according to an embodiment of the present disclosure;

[0016] Figure 4B The embodiment according to the present disclosure is shown Figure 4A An image processed by the AI-based image processing model and an authentication mask of the processed image is also shown;

[0017] Figure 5is a diagram illustrating a method of generating authentication metadata when an AI-based image processing model includes a plurality of neural networks according to an embodiment of the present disclosure;

[0018] Figure 6 is a diagram illustrating a method of generating authentication metadata according to an embodiment of the present disclosure;

[0019] Figure 7 is a diagram illustrating a method of generating an artificial intelligence model configured to generate authentication metadata according to an embodiment of the present disclosure;

[0020] Figure 8 is a diagram illustrating a method for generating authentication metadata based on an artificial intelligence model according to an embodiment of the present disclosure;

[0021] Fig. 9 is a flowchart illustrating a method for identifying a fake image according to an embodiment of the present disclosure;

[0022] Fig.10 is a flow chart illustrating a method for generating authentication metadata using a difference metric according to an embodiment of the present disclosure;

[0023] Fig.11 is a flowchart illustrating a method for generating authentication metadata using a neural network model according to an embodiment of the present disclosure;

[0024] Fig.12 shows an example authentication mask according to an embodiment of the present disclosure;

[0025] Fig.13 A graphical user interface for identifying pseudo pixels according to an embodiment of the present disclosure is shown;

[0026] Fig.14 is a diagram of an apparatus for generating authentication metadata according to an embodiment of the present disclosure;

[0027] Fig.15 According to the embodiment of the present disclosure Fig.14 a diagram of components of one or more devices; and

[0028] Fig.16 is a flowchart illustrating a method of performing image authentication according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Example embodiments are described in more detail below with reference to the accompanying drawings.

[0030] In the following description, the same reference numerals are used for the same elements even in different drawings. Matters defined in the description, such as detailed configurations and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. However, it is apparent that the exemplary embodiments may be practiced without those specifically defined matters. In addition, well-known functions or configurations are not described in detail because they would obscure the description with unnecessary detail.

[0031] Expressions such as “at least one of…” when preceding a list of elements modify the entire list of elements rather than modifying the individual elements of the list. For example, the expression “at least one of a, b, and c” should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or any variation of the foregoing.

[0032] Although terms such as "first", "second", etc. may be used to describe various elements, these elements must not be limited to the above terms. The above terms may be used only to distinguish one element from another.

[0033] The term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0034] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0035] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claims.

[0036] Unless explicitly described as such, the elements, actions or instructions used herein should not be interpreted as critical or necessary. In addition, as used herein, the articles "one" and "an" are intended to include one or more items, and can be used interchangeably with "one or more". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more". In the case of only one item, the term "one" or similar language is used. In addition, as used herein, the terms "have", "have", "have" etc. are intended to be open terms. In addition, unless otherwise expressly stated, the phrase "based on" is intended to mean "based at least in part on".

[0037] One or more embodiments of the present disclosure provide a method and device for authenticating an image. An artificial intelligence (AI)-based image processing method may be performed on an original image to obtain a processed image (e.g., a composite image), and authentication metadata may be generated to indicate whether each pixel of the processed image is authentic (genuine) or forged.

[0038] The term "raw image" may refer to the unprocessed digital output of a camera's image sensor, or an image captured directly by a camera sensor without any processing by an image signal processor (ISP) or with minimal processing by an ISP. A raw image may also be referred to as a "raw burst image" or a "Bayer image". Light or photons incident from a scene are digitized and recorded by a camera sensor, and a raw image consists of digital pixel intensity values ​​recorded by the camera sensor before any processing is applied. For example, a raw image is an image that has not been processed by an image signal processor (ISP) or an image processing unit (IPU), and may have a raw Bayer format. When a camera sensor includes sensor elements arranged in a pattern of red, green, and blue color channels known as a Bayer array, an image recorded by a Bayer array on a camera sensor is known as a Bayer image. A raw burst image may refer to a series of images captured in rapid succession at a high frame rate. Raw burst images may be captured in a continuous burst mode, where the camera captures images as quickly as possible until a memory buffer is full or the user stops the burst mode. At one of the image processing steps on the ISP, the raw burst images (i.e., a group of images) are aligned and fused to produce a single raw image. Depending on the loss function used by the AI-based image processing model that performs the alignment and fusion, even the fused single image raw may have spurious pixel values, and there are situations where the fused raw image is not even guaranteed to be real.

[0039] Additionally, one or more embodiments of the present disclosure provide methods and apparatus for storing authentication metadata for processed images and displaying an indication of spurious pixels in the processed images.

[0040] The authentication metadata may indicate a pseudo pixel included in the image, and may be implemented in the form of a binary mask that indicates real pixels and pseudo pixels using binary values ​​"0" and "1". A binary mask may be an image with a value of "0" or "1" at each pixel position. Pixels at all two-dimensional positions in the binary mask with a value of "1" are considered to be fake (not real) and are considered to have been altered by image processing (e.g., AI-based image processing). The binary mask may also be referred to as an authentication map, an authentication mask, or an authentication binary mask.

[0041] Alternatively, the authentication metadata may be implemented in the form of a list of two-dimensional positions of pseudo-pixels.

[0042] The term “pseudo pixel” may refer to a pixel that is modified via image processing (eg, AI-based image processing) to change a scene object (eg, structure or texture) in an image.

[0043] From the camera's perspective, the authentication metadata encodes which parts of the image have been hallucinated via image processing (e.g., AI-based image processing). From the camera user's perspective, the authentication metadata reveals which parts of the image are not real.

[0044] When a user queries whether an image is authentic, for example, via a photo gallery application installed in the electronic device, the electronic device may recover the authentication mask from the stored metadata and may display the authentication mask to be overlaid on the image to facilitate visualization of false pixels.

[0045] Figure 1A is a diagram of an electronic device 101 for performing image authentication according to an embodiment of the present disclosure.

[0046] According to an embodiment of the present disclosure, the electronic device 101 may be a wearable device or a wearable device (such as an HMD) on which the electronic device can be mounted. For example, the electronic device 101 may represent an AR wearable device such as a head-mounted device with a display panel or smart glasses.

[0047] At image capture time, the electronic device 101 may generate authentication metadata to mark any scene content generated by an enhancement algorithm in an artificial intelligence (AI)-based image processing model. The authentication metadata may be embedded in a processed image output from the AI-based image processing model. For example, when a facial enhancement neural network redraws eyebrows in an image, the authentication metadata may indicate pseudo pixels located at the eyebrow region using binary values.

[0048] In detail, Figure 1AAs shown, the electronic device 101 may include a camera 100, at least one processor 200, at least one memory 300, an input interface 400, a display 500, and a communication interface 600. According to an embodiment of the present disclosure, the electronic device 101 may exclude at least one of these components, or may add at least one other component.

[0049] The camera 100 may include a lens 110 and an image sensor 120, and the image sensor 120 may include a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor and a color filter array (CFA). The CFA may have a red filter, a green filter, and a blue filter arranged in a Bayer pattern. The camera 100 may capture an image based on a user input received via the input interface 400, and may output an unprocessed raw image to at least one processor 200. For example, the output data of the CFA may be considered as raw image data. The input interface 400 may be implemented as at least one of a touch panel, a keyboard, a mouse, a button, a microphone, etc.

[0050] At least one processor 200 may include an artificial intelligence (AI)-based image processing model 210 and an image authenticator 220, the image authenticator 220 including one or both of a first metadata generator 221 and a second metadata generator 222. Authentication metadata may be obtained when an original image is processed via an image signal processor (ISP), the ISP including both a conventional signal processing-based ISP and an AI-based ISP, and only an AI-based ISP. The first metadata generator 221 may generate authentication metadata based on a difference metric between an input image and a processed output image, and the second metadata generator 222 may generate authentication metadata using a thresholding operation.

[0051] At least one processor 200 may receive an input image from the camera 100 or from an external device via the communication interface 600. The input image may be a raw image that is not processed by an image signal processor (ISP) and / or has a Bayer format. The image signal processor (ISP) may be a dedicated processor that is integrated into a camera module or connected to an image sensor interface, and may process the raw sensor image into a final display reference output of the camera module.

[0052] Based on user input to edit the input image or according to the default setting of the AI-based image processing model 210, at least one processor 200 can operate the AI-based image processing model 210 to modify the input image and output the modified input image as a processed image. The AI-based image processing model 210 may include one or more neural networks configured to perform demosaicing (for converting a Bayer pattern image captured by an image sensor into a full-color image), denoising (for removing random variations in pixel values ​​to improve image quality), super-resolution (for increasing the resolution of an image by combining multiple low-resolution images or using algorithms to generate new high-resolution pixels, fill in missing details and improve the overall quality of the image), exposure correction (for adjusting the brightness of an image to produce a well-exposed result, as well as other image processing functions), cropping (for removing portions of an image to focus on a specific area or changing the aspect ratio of an image), color correction (for adjusting the colors in an image to produce a more natural-looking result), white balance (for adjusting the colors in an image to account for changes in lighting conditions), sharpening (for enhancing edges in an image to make them appear clearer and more detailed), removing unwanted objects, adding new objects, etc.

[0053] Specifically, the AI-based image processing model 210 can be implemented as a monolithic AI-based image processor, in which a single neural network block replaces the entire conventional image signal processor ( Figure 2A ). Optionally, the AI-based image processing model 210 may be implemented as two neural network blocks for front-end and back-end processing ( Figure 3 ), more than two network blocks ( Figure 4A ) or a hybrid model in which only some signal processing blocks of a conventional image single processor are replaced by neural network blocks.

[0054] The AI-based image processing model 210 may output the processed image via the display 500, and may store the processed image in the memory 300. In addition, the AI-based image processing model 210 may provide the processed image to the image authenticator 220, so that the image authenticator 220 may generate authentication metadata indicating the authenticity of each pixel in the processed image. For example, the authentication metadata may be implemented as a binary authentication mask that assigns a value of 0 or 1 to each pixel to indicate authenticity. In an embodiment, the image authenticator 220 may assign a value of "0" to an area in the binary authentication mask corresponding to a pixel determined to be authentic, and may assign a value of "1" to an area in the binary authentication mask corresponding to a pixel determined to be forged.

[0055] In an embodiment of the present disclosure, the AI-based image processing model 210 may provide any image processed via the AI-based image processing model 210 to the image authenticator 220, regardless of whether the image includes a pseudo pixel. The image authenticator 220 may determine whether there are any pseudo pixels, and if so, generate authentication metadata to indicate the pseudo pixels in the image. The image authenticator 220 may store the authentication metadata in the memory 300 together with the image.

[0056] In an embodiment of the present disclosure, the AI-based image processing model 210 may only send an image processed by a neural network that changes a scene object, thereby creating a pseudo pixel or a phantom pixel, to the image authenticator 220. The determination of whether a certain neural network changes a scene object in an input image may be performed based on the network type or configuration information stored in the memory 300 (e.g., the information in Table 1 below). For example, when the AI-based image processing model 210 includes a first neural network trained to minimize adversarial loss and configured to perform demosaicing to reconstruct pixels and a second neural network trained to minimize reconstruction loss and configured to change the color or brightness of the input image, the AI-based image processing model 210 may determine that the first neural network performs a scene object change algorithm that generates pseudo pixels, and the second neural network does not perform a scene object change algorithm, based on information about the loss function types of the first neural network and the second neural network, respectively. In this case, the AI-based image processing model 210 may send the output image of the first neural network to the image authenticator 220, so that the image authenticator 220 may generate authentication metadata indicating a pseudo pixel area in the output image. The AI-based image processing model 210 may not send the output image of the second neural network to the image authenticator 220, so that the image authenticator 220 does not need to process the output image of the second neural network that does not include pseudo pixels.

[0057] The image authenticator 220 may identify pseudo pixels via the first metadata generator 221 or the second metadata generator 222. The first metadata generator 221 may be used in the case where the camera manufacturer purchases a "black box" ISP module from a third party without any information about the structure or configuration of the ISP module, or in the case where certain modules cannot be retrained on the ISP for other reasons. By applying the difference metric to the output of the AI-based image processing model 210, the first metadata generator 221 may operate with any AI-based image processing model 210, regardless of the neural network structure of the AI-based image processing model 210. On the other hand, if the camera manufacturer has the flexibility to redesign the model architecture for a given module and retrain it accordingly, the second metadata generator 222 may be utilized. When the AI-based image processing model 210 has a specific network structure (e.g., a structure having two separate neural networks trained using reconstruction loss and adversarial loss, respectively), the second metadata generator 222 may operate with the AI-based image processing model 210 to separate pseudo pixels from real pixels.

[0058] The first metadata generator 221 may receive an input image and an output image (i.e., a processed image) of the AI-based image processing model 210 to obtain a color transform that converts the color of the input image to the color of the processed image. The first metadata generator 221 may apply the color transform to the input image and calculate a difference metric between the color-transformed input image and the processed image. The color transform may modify only the color but not the scene content, so that the difference metric between the color-transformed image and the output of the neural network reveals pseudo pixels. The first metadata generator 221 may generate authentication metadata based on the difference metric between the color-transformed input image and the processed image. The authentication metadata may carry information indicating whether each pixel of the processed image is real or fake.

[0059] The second metadata generator 222 may interoperate with an AI-based image processing model 210, which includes a generator neural network trained using a discriminator neural network. The generator neural network may perform image processing on an input image in a manner that the processed input image may or may not contain pseudo pixels. The discriminator neural network may determine whether the image it receives is a real image or an image generated by a generator. The generator neural network may include a first neural network and a second neural network connected in cascade. The first neural network and the second neural network may be trained via a two-step process. In the first training step, the first neural network may be trained using a reconstruction loss (e.g., an L1 loss also known as a mean absolute error (MAE) loss, an L2 loss also known as a mean square error (MSE) loss, etc.). In the second training step, the weights of the first neural network are fixed, and the second neural network is cascaded to the first neural network and trained using an adversarial loss using the discriminator neural network. The second neural network is trained to predict a sparse residual that is effective in deceiving the discriminator neural network when added to the output layer of the first neural network. The residual may represent hallucinated content. Training the first neural network and the second neural network via the two-step training process discussed above may provide the advantage of directly separating fake pixels from real pixels. The discriminator neural network may be used during the training process and may be discarded during the inference process. According to an embodiment of the present disclosure, only the first neural network may be trained in the first training step, and the second neural network may be cascade-connected with the first neural network only during the second training step.

[0060] The second metadata generator 222 may generate authentication metadata based on the output of the second neural network of the generator neural network. The second metadata generator 222 may apply a thresholding operation having a zero value or a non-zero value as a threshold to the output of the second neural network. The second metadata generator 222 may identify pixel positions having a value higher than a selected threshold in the output of the second neural network as forged, and may output authentication metadata (e.g., a binary authentication mask) including the fake pixel positions. In addition, the output of the first neural network and the output of the second neural network are combined, for example, by an adder, and the combination of the output of the first neural network and the output of the second neural network is provided to the display 500, or a subsequent image processing module for additional image processing.

[0061] The AI-based image processing model 210 may be incorporated into the camera hardware, and falsification of pixel values ​​may occur at image capture time as part of the image capture process on the camera hardware. However, embodiments are not limited thereto, as the AI-based image processing model 210 may be provided external to the camera hardware, and falsification of pixel values ​​may occur during a post-capture process.

[0062] Display 500 may display the processed image and authentication metadata separately or together on the same display screen.For example, the authentication metadata may be provided in the form of a binary mask, and the binary mask may be overlaid on the processed image.

[0063] All elements of the electronic device may be included in a single device, or may be included in more than one device. For example, the camera 100, the input interface 400, and the display 500 may be included in a client device (e.g., a smart phone), and at least one of the AI-based image processing model 210 and the image authenticator 220 may be included in a server. When the image authenticator 220 is included in the server, the client device may request the server to provide authentication metadata, and may receive the authentication metadata from the server.

[0064] Figure 1B is a block diagram of an electronic device for performing image authentication according to an embodiment of the present disclosure.

[0065] According to an embodiment of the present disclosure, the electronic device may be a wearable device or a wearable device (such as an HMD) on which the electronic device can be mounted. For example, the electronic device may represent an AR wearable device, such as a head-mounted device with a display panel or smart glasses. Figure 1B As shown, the electronic device 101 may include at least one processor 200 and at least one memory 300, and the electronic device 101 may exclude at least one of these components or may add at least one other component.

[0066] At least one processor 200 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). In some embodiments, the processor 200 includes one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processor unit (GPU). The processor 200 is capable of performing control on at least one of the other components of the electronic device 101 and / or performing operations or data processing related to communication or other functions. As described in more detail below, the processor 200 may perform one or more operations for receiving an input image from a camera, processing the input image. The processor 200 may perform one or more operations for generating authentication metadata including the authenticity of each pixel in the processed image.

[0067] The memory 300 may include any suitable product for storing data and executable instructions, such as a random access memory (RAM), a read-only memory (ROM), a rewritable flash memory, a hard disk drive, etc. In addition, a program (e.g., an operating system) encoded on such a computer program product may also include instructions that can be executed by the processor(s) 200 to enable the electronic device 101 to provide various functionalities. For example, the memory 300 may store commands or data related to at least one other component of the electronic device 101.

[0068] Figure 2A is a diagram showing the structure of an AI-based image processing model 210 according to an embodiment of the present disclosure.

[0069] The AI-based image processing model 210 may be implemented as Figure 2A The single network shown, but can be implemented as Figure 3 and multiple neural networks shown in Figure 4. The AI-based image processing model 210 can replace all elements or parts of a conventional image signal processor (ISP).

[0070] Reference Figure 2A , the AI-based image processing model 210 may have a U-Net structure, and may include an encoder 201, a decoder 202, and a skip connection 203. The AI-based image processing model 210 may be implemented in a client device (eg, a mobile device) and / or a server.

[0071] The encoder 201 is configured to obtain (receive) an input image, obtain and send an intermediate representation of the input image. The encoder 201 includes the following Figure 2B The decoder 202 is configured to obtain and send an output image (e.g., a final image in JPG format) based on the obtained intermediate representation. The decoder 202 includes a neural network described in the following (such as, for example, a convolutional neural network (CNN) or a deep neural network (DNN)). Figure 2C The neural network described in (such as, for example, CNN or DNN). Via the encoder 201 and the decoder 202, the AI-based image processing model 210 may perform at least one of the following: effective area cropping, linearization, black level subtraction, normalization, demosaicing, denoising, white balance, color space conversion, digital exposure adjustment, local tone mapping, general color manipulation, selective color manipulation (e.g., skin tone manipulation), global tone mapping, YuV denoising, multi-frame fusion (e.g., multi-frame fusion in the YuV domain), final color space conversion, gamma correction, grain addition processing, resizing and resampling, and compression.

[0072] Figure 2Bis a diagram showing the structure of an encoder of an AI-based image processing model according to an embodiment of the present disclosure.

[0073] Reference Figure 2B Although the structure of the encoder 201 is not limited thereto, for example, the encoder 201 may have a U-Net structure including a first stage 204, a second stage 205, and a third stage 206. An input image may be transmitted through each layer of the first stage 204, the second stage 205, and the third stage 206 to obtain an intermediate representation output to the decoder 202.

[0074] The first stage 204 may include 3×3 convolutional layers 204a and 204b with stride 1 and padding 1, and rectified linear unit (ReLU) layers 204c and 204d. The ReLU layer 204c may be inserted between the convolutional layers 204a and 204b, and the convolutional layer 204b may be inserted between the ReLU layers 204c and 204d. For example, each of the convolutional layers 204a and 204b and the ReLU layers 204c and 204d may have a size of 128×128×24.

[0075] The second stage 205 may include 2×2 max pooling layers 205a and 205b with stride 2, 3×3 convolution layers 205c and 205d with stride 1 and padding 1, and ReLU layers 210e and 210f. The ReLU layer 210e may be inserted between the convolution layers 205c and 205d, the convolution layer 205d may be inserted between the ReLU layers 205e and 205f, and the convolution layers 205c and 205d and the ReLU layers 205e and 205f may be inserted between the max pooling layers 205a and 205b. For example, each of the convolution layers 205c and 205d and the ReLU layers 205e and 205f may have a size of 64×64×48, the max pooling layer 205a may have a size of 64×64×24, and the max pooling layer 205b may have a size of 32×32×48.

[0076] The third stage 206 may include 3×3 convolutional layers 206a and 206b with stride 1 and padding 1, ReLU layers 206c and 206d, and a 2×2 maximum pooling layer 206e with stride 2. The ReLU layer 206c may be inserted between the convolutional layers 206a and 206b, the convolutional layer 206b may be inserted between the ReLU layers 206c and 206, and the maximum pooling layer 206e may be disposed after the convolutional layers 206a and 206b and the ReLU layers 206c and 206d. For example, each of the convolutional layers 206a and 206b and the ReLU layers 206c and 206d may have a size of 16×16×192, and the maximum pooling layer 206e may have a size of 8×8×192.

[0077] Figure 2C is a diagram showing the structure of a decoder of an AI-based image processing model according to an embodiment of the present disclosure.

[0078] Reference Figure 2C , the decoder 202 includes a first stage 207, a second stage 208, a third stage 209, and a fourth stage 211. The intermediate representation may be transmitted through each layer in the first stage 207, the second stage 208, the third stage 209, and the fourth stage 211 to obtain an output image.

[0079] The first stage 207 may include ReLU layers 207a and 207b and 3×3 convolutional layers 207c and 207d with stride 1 and padding 1. The convolutional layer 207c may be inserted between the ReLU layers 207a and 207b, and the ReLU layer 207b may be inserted between the convolutional layers 207c and 207d. For example, each of the ReLU layers 207a and 207b and the convolutional layers 207c and 207d may have a size of 8×8×384.

[0080] The second stage 208 may include a 2×2 upsampling layer 208a, 3×3 convolutional layers 208b, 208c, and 208d with stride 1 and padding 1, a deep connection layer 208e, and ReLU layers 208f and 208g. The layers of the second stage 208 may be in the following order from input to output: an upsampling layer 208a, a convolutional layer 208b, a deep connection layer 208e, a ReLU layer 208f, a convolutional layer 208c, a ReLU layer 208g, and a convolutional layer 208d. For example, each of the upsampling layer 208a, the convolutional layers 208b, 208c, and 208d, and the ReLU layers 208f and 208g may have a size of 16×16×192, and the deep connection layer 208e may have a size of 16×16×384.

[0081] The third stage 209 may include a 2×2 upsampling layer 209a, 3×3 convolutional layers 209b, 209c, and 209d with stride 1 and padding 1, a depth connection layer 209e, and ReLU layers 209f and 209g. The layers of the third stage 209 may be in the following order from input to output: an upsampling layer 209a, a convolutional layer 209b, a depth connection layer 209e, a ReLU layer 209f, a convolutional layer 209c, a ReLU layer 209g, and a convolutional layer 209d. For example, each of the upsampling layer 209a, the convolutional layers 209b, 209c, and 209d, and the ReLU layers 209f and 209g may have a size of 64×64×48, and the depth connection layer 209e may have a size of 64×64×96.

[0082] The fourth stage 211 may include a 2×2 upsampling layer 211a, 3×3 convolutional layers 211b, 211c, and 211d with stride 1 and padding 1, a depth connection layer 211e, ReLU layers 211f and 211g, and a 1×1 convolutional layer 211h with stride 1 and padding 1. The layers of the fourth stage 211 may be in the following order from input to output: upsampling layer 211a, convolutional layer 211b, depth connection layer 211e, ReLU layer 211f, convolutional layer 211c, ReLU layer 211g, and convolutional layers 211d and 211h. For example, each of the upsampling layer 211a, the convolutional layers 211b, 211c, and 211d, and the ReLU layers 211f and 211g may have a size of 128×128×24, the depth connection layer 211e may have a size of 128×128×48, and the convolutional layer 211h may have a size of 128×128×3.

[0083] Figure 3 is a diagram showing the structure of an AI-based image processing model 210 according to an embodiment of the present disclosure.

[0084] like Figure 3 As shown, the AI-based image processing model 210 may include a Bayer processing front-end network 210A and a photofinishing back-end network 210B.

[0085] The Bayer processing front-end network 210A may include an encoder 201A and a decoder 202A. The Bayer processing front-end network 210A may perform at least one of active area clipping, linearization, black level subtraction, normalization, demosaicing, denoising, white balance, and color space conversion.

[0086] The photo finishing back-end network 210B may include an encoder 201B and a decoder 202B. The photo finishing back-end network 210B may perform digital exposure adjustment, local tone mapping, general color manipulation, selective color manipulation (e.g., skin tone manipulation), global tone mapping, YuV denoising, multi-frame fusion (e.g., multi-frame fusion in the YuV domain), final color space conversion, gamma correction, grain addition processing, resizing and resampling, and compression.

[0087] Each of the Bayer processing front-end network 210A and the photofinishing back-end network 210B may have Figure 2B and Figure 2C The AI-based image processing model 210 shown in FIG. 2 has the same or substantially the same structure as that of the AI-based image processing model 210 shown in FIG.

[0088] Figure 4A is a diagram showing the structure of an AI-based image processing model 210 according to an embodiment of the present disclosure.

[0089] like Figure 4AAs shown, the AI-based image processing model 210 may include a Bayer front-end network 210C, a colorimetric network 210D, a photo enhancement network 210E, and an output reference network 210F.

[0090] The Bayer front-end network 210C may perform joint demosaicing and denoising on the input image. The colorimetric network 210D may map colors from the camera sensor space to a canonical space such as CIE XYZ. The colorimetric network 210D may estimate the illumination in the scene in which the input image is captured, and may apply white balance using the estimated illumination. In addition, the colorimetric network 210D may transform the white-balanced input image to a canonical CIE XYZ space. The photo enhancement network 210E may perform enhancement or photofinishing processing to enhance the aesthetic presentation. The output reference network 210F may map the image to a display-referenced color space such as sRGB, and may further perform color manipulation and image compression.

[0091] Each of the Bayer front-end network 210C, the colorimetric network 210D, the photo enhancement network 210E, and the output reference network 210F may have Figure 2B and Figure 2C The AI-based image processing model 210 shown in FIG. 2 has the same or substantially the same structure as that of the AI-based image processing model 210 shown in FIG.

[0092] Figure 4B An example image processed by a plurality of neural networks included in the AI-based image processing model 210 is shown, and an authentication mask indicating pseudo pixels included in the image is also shown. For example, Figure 4B The multiple neural networks shown may have the following characteristics.

[0093] [Table 1]

[0094] Pseudo-Pixels Image processing type Network loss type Network 1 yes De-mosaicing and denoising Fighting Losses Network 2 no White balance and color space conversion Reconstruction losses Network 3 yes General Enhancements Fighting Losses Network 4 no Final color space conversion Reconstruction losses

[0095] Reference Figure 4B, the Bayer front-end network 210C may perform demosaicing on an input image (e.g., a single-channel Bayer raw image), and if the Bayer front-end network 210C is trained to minimize the adversarial loss, the Bayer front-end network 210C may hallucinate objects in the input image. For example, the Bayer front-end network 210C may hallucinate scene content 210G around the edges of building windows in the input image. The Bayer front-end network 210C may output a first intermediate image (e.g., a demosaiced 3-channel raw RGB image), in which the scene content around the edges of the building windows is hallucinated. Based on the first intermediate image, the processor 200 may generate a first authentication mask that indicates a pseudo pixel area 210G' corresponding to the scene content around the edges of the building windows as pseudo pixels. The processor 200 may store the first authentication mask as metadata of the first intermediate image.

[0096] The colorimetric network 210D receives a first intermediate image (e.g., a demosaiced 3-channel raw RGB image) and obtains a second intermediate image by performing white balance and color space transformation on the first intermediate image. If the reconstruction loss is applied to the colorimetric network 210D, the colorimetric network 210D does not change, delete, or add scene content of the first intermediate image, and does not generate pseudo pixels. Therefore, the processor 200 advances the first authentication mask from the previous stage to the current stage of the colorimetric network 210D, and uses the first authentication mask as the authentication mask of the second intermediate image (i.e., the second authentication mask). The processor 200 stores the second authentication mask as metadata of the second intermediate image.

[0097] The photo enhancement network 210E may receive the second intermediate image and may hallucinate objects in the second intermediate image if an adversarial loss is applied to the photo enhancement network 210E. For example, the photo enhancement network 210E may hallucinate underexposed or overexposed content 210H and 210I in the second intermediate image (e.g., Figure 4B The processor 200 may generate a third authentication mask based on the third intermediate image, which indicates pseudo pixel areas 210H' and 210I' corresponding to the phantom underexposed or overexposed content 210H and 210I as pseudo pixels. The processor 200 may store the third authentication mask as metadata of the third intermediate image.

[0098] The output reference network 210F receives the third intermediate image and obtains the final output image by mapping the third intermediate image to a display reference color space (e.g., sRGB) and by further performing color manipulation and image compression on the third intermediate image. If the reconstruction loss is applied to the output reference network 210F, the output reference network 210F does not change, delete, or add the scene content of the third intermediate image, and does not generate pseudo pixels. Therefore, the processor 200 advances the third authentication mask from the previous stage to the final stage, and uses the third authentication mask as the final authentication mask of the final output image. The processor 200 stores the final authentication mask as metadata of the final output image. The final authentication mask is an integration of all previous stage masks.

[0099] The processor 200 may store the first intermediate image together with the first authentication mask, the second intermediate image together with the second authentication mask, the third intermediate image together with the third authentication mask, and the final output image together with the final authentication mask.

[0100] exist Figure 4B In the example, the authentication mask of the current stage shows all pseudo pixels generated in all previous stages and the current stage, but the embodiment is not limited thereto. For example, the authentication mask of the current stage may only show pseudo pixels generated in the current stage, without showing pseudo pixels generated previously.

[0101] Figure 5 is a diagram illustrating a method of generating authentication metadata when the AI-based image processing model 210 includes a plurality of neural networks.

[0102] Reference Figure 5 , the AI-based image processing model 210 may include a plurality of neural networks, for example, a first neural network 212, a second neural network 213, a third neural network 214, and a fourth neural network 215. The first neural network 212, the second neural network 213, the third neural network 214, and the fourth neural network 215 may perform different image processing algorithms. For example, the first neural network 212, the second neural network 213, the third neural network 214, and the fourth neural network 215 may correspond to the Bayer front-end network 210C, the colorimetric network 210D, the photo enhancement network 210E, and the output reference network 210F, respectively.

[0103] At each stage of the plurality of neural networks 212-215, authentication metadata (e.g., authentication metadata in the form of a binary mask, where a value of 1 indicates a pseudo pixel) may be calculated and advanced to the next stage. The final authentication metadata is saved as metadata for the final output image of the plurality of neural networks 212-215. In addition, the authentication metadata calculated at each stage may be saved as metadata for the intermediate image output from each of the plurality of neural networks 212-215. The authentication metadata may indicate which parts of the image are not real (i.e., pseudo pixels). The term "pseudo pixel" may refer to a pixel value generated by adding new scene content that does not exist in the input image, changing existing scene content, or deleting existing scene content through AI image processing. The falsification of pixel values ​​may occur at capture time as part of the image capture process on the camera hardware. For example, when a group of images are captured as raw burst images and aligned and fused via a neural network using an adversarial loss function to produce a single raw image, the fused single image may have pseudo pixel values. If an AI module or neural network modifies an input image for the purpose of illusion of detail, and thereby changes, adds, or deletes scene content in the input image, the AI ​​module or neural network may be determined to generate pseudo pixels. In other words, a pixel is marked as fake only when it undergoes processing that intentionally hallucinates the content. For example, the processor may determine whether the AI ​​module or neural network generates fake pixels based on the type of loss used to train the AI ​​module or neural network. If the AI ​​module is trained to restore the original signal based on a reconstruction-based loss function (e.g., mean square error (MSE) or mean absolute error (MAE)), the processor may determine that the AI ​​module does not generate fake pixels. If the AI ​​module is trained to perform image enhancement (e.g., deblurring, filtering, and contrast algorithms) to hallucinate details or features that produce perceptually pleasing images based on a perceptual loss function (e.g., visual geometry group (VGG) loss or learning perceptual image block similarity (LPIPS) loss) or an adversarial loss function, the processor may determine that the AI ​​module generates fake pixels. For example, all types of generative models including but not limited to generative adversarial networks (GANs), variational autoencoders (VAEs), flow-based models, and diffusion models can be considered AI models that can potentially generate hallucinated content. Furthermore, if the AI ​​module only manipulates the color of the image and does not manipulate the scene content (eg, structure, texture, etc.), the processor may determine that the AI ​​module does not generate false pixels and may determine that the output image of the AI ​​module is real.

[0104] The authentication metadata may be downsampled and / or compressed to reduce the size of the authentication metadata. In addition, security keys, non-fungible tokens (NFTs), and blockchain technology may be used to protect the authentication metadata to prevent tampering.

[0105] The memory 300 may store network configuration information of the plurality of neural networks 212-215, for example, as shown in the above Table 1. When the first neural network 212, the second neural network 213, the third neural network 214, and the fourth neural network 215 correspond to the Bayer front-end network 210C, the colorimetric network 210D, the photo enhancement network 210E, and the output reference network 210F, respectively, the memory 300 may store information indicating that the first neural network 212 and the third neural network 214 generate pseudo pixels, and the second neural network 213 and the fourth neural network 215 do not generate pseudo pixels. The photo enhancement network 210E may perform super-resolution.

[0106] The processor 200 may determine whether to generate authentication metadata for each of the first intermediate image, the second intermediate image, the third intermediate image, and the final output image based on the network configuration information of each of the plurality of neural networks 212-215. Referring to Table 1, the processor 200 may determine to skip generating authentication metadata for the image processed by the second neural network 213 and the fourth neural network 215, and may advance the authentication metadata from the previous stage because the second neural network 213 and the fourth neural network 215 do not generate pseudo pixels.

[0107] The processor 200 may generate first authentication metadata and second authentication metadata for the first intermediate image and the third intermediate image respectively output from the first neural network 212 and the third neural network 214. Specifically, the AI-based image processing model 210 may provide the first intermediate image and the third intermediate image to the image authenticator 220. The image authenticator 220 may analyze the first intermediate image and the third intermediate image via the first metadata generator 221 or the second metadata generator 222. The image authenticator 220 may identify pseudo pixels included in the first intermediate image and the third intermediate image, and may generate first authentication metadata and second authentication metadata indicating positions of pseudo pixels included in the first intermediate image and the third intermediate image, respectively.

[0108] According to different embodiments, the second authentication metadata may indicate pseudo pixels generated only by the third neural network 214, or may indicate pseudo pixels generated by both the first neural network 212 and the third neural network 214. The image authenticator 220 may compare the input image and the output image (i.e., the second intermediate image and the third intermediate image) of the third neural network 214 to generate the second authentication metadata showing the pseudo pixels generated only by the third neural network 214. Alternatively, the image authenticator 220 may compare the input image of the artificial intelligence-based image processing model 210 (i.e., the input image of the first neural network 212) with the output image (i.e., the third intermediate image) of the third neural network 214 to generate the second authentication metadata showing the pseudo pixels generated by all the previous neural networks and the current neural network (i.e., the first neural network 212, the second neural network 213, and the third neural network 214).

[0109] The processor 200 may store the first authentication metadata and the second authentication metadata together with the first intermediate image and the third intermediate image in the memory 300. For example, the first authentication metadata and the second authentication metadata may be embedded in the JPEG annotation fields of the first intermediate image and the third intermediate image, respectively. When the authentication mask only displays the pseudo pixels generated at the current stage and does not display the pseudo pixels generated previously, the third intermediate image may have the second authentication metadata. On the other hand, when the authentication mask displays the pseudo pixels generated at the current stage in addition to the pseudo pixels generated previously, the third intermediate image may have the first authentication metadata and the second authentication metadata.

[0110] Figure 6 2 is a diagram showing a method for generating authentication metadata according to an embodiment of the present disclosure. For example, the first metadata generator 221 may perform the authentication based on the difference metric. Figure 6 The method shown.

[0111] In operation 221A, the first metadata generator 221 may estimate a color transform T that maps the color of the input image to the output image. The color transform T may be estimated using a linear mapping method (e.g., y=ax+b, where saturated pixels are excluded during the estimation of the color transform) or a nonlinear mapping method (e.g., a second-order polynomial or higher-order function having the form of ax2+bx+c).

[0112] The input image and the output image may be the input image and the output image of any one of the plurality of neural networks 212-215, or may be the input image of the first neural network 212 and the output image of the fourth neural network 215. A color transform T is applied to the input image to map the color values ​​of the input image to the output image, thereby obtaining a color-transformed input image.

[0113] More specifically, given an input image I1∈Rn×3 and an output image I2∈Rn×3 of n pixels of the same scene, the linear color transformation T∈R3×3 between the color values ​​of the input image I1 and the output image I2 can be expressed as I 2 ≈I 1 T.

[0114] T can be calculated using the pseudo-inverse as follows:

[0115] More specifically, given A represents the pixel values ​​in the R, G, B color channels of the input image I1, and B represents the pixel values ​​in the R, G, B color channels of the output image I2, the 3×3 color transformation matrix T between A and B is calculated as follows.

[0116] A×T=B

[0117]

[0118] In the matrices of A and B, the three columns correspond to the R, G, B color channels, and the rows correspond to the number of pixels in the input image I1 and the output image I2, respectively.

[0119] Using the pseudo-inverse equation, the 3×3 color transformation matrix T is calculated as follows:

[0120]

[0121] In this embodiment, a 3×3 color transformation matrix is ​​used because it is linear and accurate, and computationally efficient. However, the size of the color transformation matrix is ​​not limited thereto, and any 3×M color transformation matrix (where M≥3) may be used.

[0122] Return to reference Figure 6 , in operation 221B, a color transform is applied to the input image so that the colors of the input image are mapped to the colors of the output image.

[0123] In operation 221C, the color transformed input image is compared with the output image to calculate a difference metric between the color transformed input image and the output image, and authentication metadata is generated based on the difference metric. For example, the L1 difference between the color transformed input image and the output image is calculated for each pixel, and a thresholding operation is applied to the L1 difference to obtain a binary authentication mask as the authentication metadata. L1 differences less than a preset threshold may be converted to "0" to indicate a real pixel, and L1 differences greater than or equal to the current threshold may be converted to "1" in the binary authentication mask to indicate a pseudo pixel. Optionally, an L2 difference or any Lp difference may be calculated to generate an authentication mask (where p represents a real number greater than or equal to 1). When there are two vectors A=(x1, y1) and B=(x2, y2), the L1 difference may be calculated as |x1-2|+|y1-y2|, and the L2 difference may be calculated as And the Lp difference can be calculated as

[0124] When the camera 100 captures a 12-megapixel image with a resolution of 3000×4000 pixels, the display reference output of the camera 100 in a lossy JPEG compression format is approximately 2MB to 4MB, and the full-resolution lossless binary authentication mask may be approximately 150KB to 300KB. The processor 200 may perform downsampling on the lossless binary authentication mask to save storage in the memory 300. For example, the processor 200 may downsample the lossless binary authentication mask to half the resolution of 1500×2000 pixels, so that the file size is reduced to approximately 64KB to 96KB, which represents a very nominal overhead of 2% to 3%.

[0125] The processor 200 may encrypt the authentication mask and store an encrypted version of the authentication mask with the image, or may generate a key to retrieve the authentication mask and store the key with the image.

[0126] Figure 7 is a diagram illustrating a method for generating an artificial intelligence model configured to generate authentication metadata. Figure 7 The method shown is generated Figure 5 Each of the multiple neural networks 212-215 of the AI-based image processing model 210 is shown.

[0127] Reference Figure 7, each of the multiple neural networks in the AI-based image processing model 210 can be configured to have a structure of a generator neural network 2221, in which the image processing part based on the reconstruction loss and the residual image generation part are separated and cascaded. The generator neural network 2221 can be trained using a discriminator neural network 2222. The generator neural network 2221 is trained to generate realistic images (e.g., forged images) so as to deceive the discriminator neural network 2222 to classify the generated images as real. The discriminator neural network 2222 is trained to classify the generated images as forged. The discriminator neural network 2222 can be any type of neural network, including a convolutional neural network with a fully connected layer.

[0128] The generator neural network 2221 may include a first deep neural network (DNN) 2223, a second DNN 2224, and an adder 2225. Although the structures of the first DNN 2223 and the second DNN 2224 are not limited thereto, each of the first DNN 2223 and the second DNN 2224 may have a UNet encoder-decoder structure, such as Figure 2B and Figure 2C The first DNN 2223 and the second DNN 2224 may be any type of neural network, such as a fully connected network, a convolutional neural network (CNN), or a recurrent neural network (RNN).

[0129] Specifically, in the first training stage, the first DNN 2223 is trained to perform image processing on the image input to the first DNN 2223 to obtain the output image in a manner that minimizes the reconstruction loss between the output image and the ground-truth target image. For example, when the first DNN 2223 is implemented as a module that performs demosaicing, the input image may be a single-channel Bayer image, the output image may be a three-channel demosaiced image, and the first DNN 2223 may be trained to minimize the L1 or L2 reconstruction loss between the three-channel demosaiced image and the ground-truth target image. When the first DNN 2223 is implemented as a denoiser, the input image may be a short-exposure high ISO noise image, the output image may be a noise-free or low-noise image, and the first DNN 2223 may be trained to minimize the L1 or L2 reconstruction loss between the noise-free or low-noise image and the ground-truth target image (e.g., a long-exposure low ISO image of the same scene). When the loss reaches a local minimum and does not decrease after a preset number of iterations, or when the loss reaches a predetermined minimum, it is determined that the loss is minimized.

[0130] Once the training of the first DNN 2223 is completed, the weights and biases of the first DNN 2223 may be fixed in a second training phase in which the second DNN 2224 and the discriminator neural network 2222 are jointly trained. For example, the cascaded first DNN 2223 and second DNN 2224 operate as a generative neural network model in an adversarial training setting, although the first DNN 2223 and second DNN 2224 do not necessarily need to operate as a generative neural network model.

[0131] The discriminator neural network 2222 may take an image (real or generated) as input and output a probability score indicating whether the input image is a real image or an image generated by the generator neural network 2221.

[0132] In the second training phase, the output image of the first DNN 2223 is fed as input to the second DNN 2224. The second DNN 2224 outputs a residual image, which is added to the input of the second DNN 2224 via an adder 2225 to obtain a combined image. When the second DNN 2224 outputs a sparse residual image, only a limited number of pixels within a predetermined range may have a non-zero value, and the remaining pixels may have a zero value. In this case, the second metadata generator 222 may use a zero threshold to determine all pixel values ​​greater than zero as forged. When the second DNN 2224 outputs a non-sparse residual image, the second metadata generator 222 may use a non-zero threshold to identify pseudo pixels.

[0133] The combined image is fed as an input to the discriminator neural network 2222, and the discriminator neural network 2222 outputs a probability score indicating whether the image input to the discriminator neural network 2222 is a real image or an image generated by the generator neural network 2221. The second DNN 2224 and the discriminator neural network 2222 are jointly trained until the adversarial loss is minimized or converges to a constant value with a predetermined margin. According to an embodiment of the present disclosure, a reconstruction loss or a learning-perceptual image block similarity (LPIP) loss may be added to the adversarial loss to stabilize the training of the AI ​​model and improve the convergence of the loss.

[0134] In the second training phase, the parameters of the second DNN 2224 of the generator neural network 2221 and the discriminator neural network 2222 may be updated via joint training. The generator neural network 2221 may generate synthetic samples via the second DNN 2224, and then feed the synthetic samples together with the real data samples into the discriminator neural network 2222. The discriminator neural network 2222 may use an adversarial loss function to calculate the error between its prediction and the reference true value label (whether the image input to the discriminator neural network 2222 is a real image or an image generated by the generator neural network 2221), and may update its network parameters to reduce the error. In turn, the generator neural network 2221 may update its network parameters to maximize the error rate of the discriminator neural network 2222.

[0135] In the inference stage, the output of the second DNN 2224 can be used to identify whether the images output from each of the multiple neural networks (e.g., the first intermediate image output from the first neural network 212, the second intermediate image output from the second neural network 213, the third intermediate image output from the third neural network 214, and the final output image output from the fourth neural network 215) include pseudo pixels.

[0136] Figure 8 is a diagram illustrating a method for generating authentication metadata based on an artificial intelligence model according to an embodiment of the present disclosure.

[0137] Reference Figure 8 , in the inference stage, the discriminator neural network 2222 may be discarded, and a thresholding operator 2227 may be connected to the output layer of the second DNN 2224 to determine the pseudo pixels in the output of the generator neural network 2221. For example, the thresholding operator 2227 may use a zero value as a threshold to determine pixels having non-zero values ​​as pseudo pixels. Alternatively, the non-zero entries in the output of the second DNN may be directly classified as pseudo pixels without using the thresholding operator 2227.

[0138] In an embodiment, a non-zero value may be used as a threshold for identifying false pixels. The output of the thresholding operator 2227 or the direct classification result may be used as authentication metadata. In addition, a morphological operation (such as corrosion or expansion) may be applied to the thresholded output image to produce a final binary authentication mask.

[0139] Fig. 9 is a flowchart illustrating a method of identifying a fake image according to an embodiment of the present disclosure.

[0140] In operation S901 , the processor may receive an input image.

[0141] In operation S902, the processor may perform AI-based image processing on an input image via a neural network to obtain a processed image.

[0142] In operation S903, the processor may determine whether scene content in the input image is changed based on a loss type of the neural network.

[0143] When the neural network is trained to minimize the reconstruction loss, it is determined that the scene content in the input image has not been changed, and the method proceeds to operation S904 to store the processed image without generating authentication metadata.

[0144] When the neural network is trained without using the reconstruction loss but is trained using the adversarial loss, it is determined that the scene content in the input image may be changed, and the method proceeds to operation S905 to generate authentication metadata, for example, using a difference metric or a specific neural network structure, as described below with reference to Fig.10 and Fig.11 described.

[0145] In operation S906, the authentication metadata is stored as metadata of the processed image together with the processed image.

[0146] In operation S907, if there is additional image processing, the method proceeds to operation S901 to obtain a second processed image and generate second authentication metadata of the second processed image.

[0147] Operations S901 - S906 may be performed for each output image of the plurality of AI-based image processing modules, or may be performed only for final output images of the plurality of AI-based image processing modules.

[0148] Fig.10 is a flow chart illustrating a method for generating authentication metadata using a difference metric according to an embodiment of the present disclosure. Fig.10 In the above description, the input image and the output image may refer to the input image and the output image of the AI-based image processing model 210, or the input image and the output image of any one of the plurality of neural networks included in the AI-based image processing model 210 (e.g., in Figure 5 , the first intermediate image input to the second neural network 213 and the second intermediate image output from the second neural network 213, or the second intermediate image input to the third neural network 212 and the third intermediate image output from the third neural network 212).

[0149] In operation S1001, a color transform is calculated based on an input image and an output image to map the color of the input image to the color of the output image. Figure 6As described in , a linear mapping method or a nonlinear mapping method can be used to calculate color transformation.

[0150] In operation S1002, color transformation is applied to an input image to obtain a color transformed input image.

[0151] In operation S1003, a difference measure between the color transformed input image and the output image is obtained using an L1 difference (i.e., Manhattan distance), an L2 difference (i.e., Euclidean distance), or any Lp difference. The L1 difference, L2 difference, and Lp difference may represent the difference between two vectors or two sets of values. The L1 difference may be calculated by adding the absolute differences between the elements of the two vectors. The L2 difference may be obtained by calculating the square root of the sum of the squared differences between the elements of the two vectors. The Lp difference between two vectors x and y may be expressed as (|x 1 -y 1 | p +|x 2 -y 2 | p +…+|x n -y n | p ) 1 / p , where x 1 , x 2 , ..., x n is the n elements of vector x, y 1 ,y 2 , ..., y are the n elements of vector y, and p is a real number greater than or equal to 1.

[0152] In operation S1004, authentication metadata is generated based on the difference metric. For example, the L1 difference between the color transformed input image and the output image is calculated for each pixel, and a threshold operation is applied to the L1 difference to obtain a binary authentication mask. L1 differences less than a preset threshold can be converted to "0" to indicate a real pixel area, and L1 differences greater than or equal to the current threshold can be converted to "1" to indicate a fake pixel area in the binary authentication mask.

[0153] Fig.11 1 is a flow chart illustrating a method for generating authentication metadata using a neural network model according to an embodiment of the present disclosure. The neural network model may include a generator neural network trained using a discriminator neural network. The generator neural network may include a first deep neural network (DNN) and a second DNN.

[0154] In operation S1101, a training image is input to a first DNN of a generator neural network.

[0155] In operation S1102, a reconstructed image of the training image is obtained as an output of the first DNN.

[0156] In operation S1103 , the first DNN is trained until a reconstruction loss of the first DNN is minimized or converged.

[0157] Once the training of the first DNN is completed, the network parameters of the first DNN are fixed, and the method proceeds to operation S1104. In operation S1104, the reconstructed image is input to the second DNN of the generator neural network.

[0158] In operation S1105 , a sparse residual image is obtained as an output of the second DNN.

[0159] In operation S1106, the reconstructed image and the sparse residual image are combined to output a combined image.

[0160] In operation S1107, the combined image is input to the discriminator neural network, and as an output of the discriminator neural network, an authentication decision for the combined image is obtained. The authentication decision indicates whether the combined image is authentic or generated by the generator NN.

[0161] In operation S1108, while the network parameters of the first DNN are fixed, a training process of the second DNN is performed using the discriminator neural network until the discriminator neural network can no longer distinguish between real samples and generated samples, or until some other stopping criteria is reached. Specifically, the second DNN and the discriminator neural network are trained until the adversarial loss is minimized. When the adversarial loss has reached a predetermined minimum value or a constant value with a predetermined margin, it is determined that the adversarial loss is minimized.

[0162] In the inference phase, the discriminator neural network 2222 can be discarded and the output of the second DNN can provide information about pseudo pixels. In one embodiment, all non-zero entries in the output of the second DNN can be directly classified as pseudo pixels. Optionally, a thresholding operation can be applied to the output of the second DNN to identify pseudo pixels.

[0163] Fig.12 An example authentication mask according to an embodiment of the present disclosure is shown.

[0164] Reference Fig.12 , when the processed image includes pseudo pixels, a binary authentication mask is generated to indicate areas corresponding to real pixels and areas corresponding to pseudo pixels using binary values ​​"0" and "1". The binary authentication mask may be stored in the memory 300 together with the processed image as metadata of the processed image. For example, when the processed image is displayed in JPEG format, the binary authentication mask may be embedded as metadata in a JPEG comment field of the processed image.

[0165] Fig.13A graphical user interface for identifying pseudo pixels according to an embodiment of the present disclosure is shown.

[0166] like Fig.13 As shown, the electronic device 101 may include a display 500 configured to display an original image, a processed image, and the original image together with the processed image. Upon user request, a pseudo pixel may be identified in the processed image.

[0167] For example, the electronic device 101 may execute a photo gallery application and may receive a user request to identify pseudo pixels in a processed image via the photo gallery application. The photo gallery application may recover a binary authentication mask from metadata embedded in the processed image. For example, the photo gallery application may retrieve authentication metadata from a JPEG annotation field in metadata of the processed image and may recover a binary authentication mask based on the authentication metadata. The photo gallery application may display a binary authentication mask to be overlaid on the processed image to visually indicate pseudo pixels.

[0168] Fig.14 is a diagram of an apparatus for generating authentication metadata according to an embodiment of the present disclosure. Fig.14 Included are user device 1410, server 1420, and network 1430. User device 1410 and server 1420 may be interconnected via a wired connection, a wireless connection, or a combination of wired and wireless connections.

[0169] The user device 1410 includes one or more devices configured to generate an output image. For example, the user device 1410 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a camera device, a wearable device (e.g., a pair of smart glasses or a smart watch), or the like.

[0170] Server 1420 includes one or more devices configured to receive an image, perform AI-based image processing on the image to obtain a processed image, generate authentication metadata indicating pseudo pixels included in the processed image, and send the processed image and authentication metadata to user device 1410 in response to a request from user device 1410.

[0171] The network 1430 includes one or more wired and / or wireless networks. For example, the network 1430 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0172] Fig.14 The number and arrangement of devices and networks shown in the figure are provided as examples. In practice, Fig.14 There may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks than those shown in . In addition, Fig.14 Two or more of the devices shown in may be implemented in a single device, or Fig.14 A single device shown in the may be implemented as multiple distributed devices. Additionally or alternatively, a group of devices (eg, one or more devices) may perform one or more functions described as being performed by another group of devices.

[0173] Fig.15 According to the embodiment of the present disclosure Fig.14 The electronic device 2000 may correspond to the user device 1410 and / or the server 1420.

[0174] The electronic device 2000 includes a bus 2010 , a processor 2020 , a memory 2030 , an interface 2040 , and a display 2050 .

[0175] The bus 2010 includes a circuit for connecting the components 2020 to 2050 to each other. The bus 2010 serves as a communication system for transmitting data between the components 2020 to 2050 or between electronic devices.

[0176] The processor 2020 includes one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a multi-core processor (MIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a machine learning accelerator, and a neural processing unit (NPU). The processor 2020 may be a single-core processor or a multi-core processor. The processor 2020 is capable of performing control of any one or any combination of the other components of the electronic device 2000, and / or performing operations related to communication or data processing. For example, the processor 2020 may include Figure 1AThe processor 2020 executes one or more programs stored in the memory 2030.

[0177] The memory 2030 may include volatile and / or non-volatile memory. The memory 2030 stores information, such as one or more of commands, data, programs (one or more instructions), applications 2034, etc., which are related to at least one other component of the electronic device 2000 and are used to drive and control the electronic device 2000. For example, the commands and / or data may formulate an operating system (OS) 2032. The information stored in the memory 2030 may be executed by the processor 2020. Specifically, the memory 2030 may store original images, processed images, and authentication metadata of the processed images.

[0178] Application 2034 includes the embodiments discussed above. These functions may be performed by a single application or by multiple applications that each perform one or more of these functions. For example, application 2034 may include a photo gallery application. When the photo gallery application receives a user request to identify a pseudo pixel in a processed image, the photo gallery application may recover a binary authentication mask from the authentication metadata of the processed image stored in memory 2030. The photo gallery application may display the binary authentication mask overlaid on the processed image to visually indicate the pseudo pixel.

[0179] The display 2050 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 2050 may also be a depth perception display (such as a multi-focal display). The display 2050 can present, for example, various contents (such as text, images, videos, icons, and symbols).

[0180] The interface 2040 includes an input / output (I / O) interface 2042, a communication interface 2044, and / or one or more sensors 2046. The I / O interface 2042 serves as an interface that can transmit commands and / or data between, for example, a user and / or other external devices and other (one or more) components of the electronic device 2000.

[0181] The communication interface 2044 may enable communication between the electronic device 2000 and other external devices via a wired connection, a wireless connection, or a combination of a wired and wireless connection. The communication interface 2044 may allow the electronic device 2000 to receive information from another device and / or provide information to another device. For example, the communication interface 2044 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc. The communication interface 2044 may receive an original image, a processed image, and / or authentication metadata from an external device or send an original image, a processed image, and / or authentication metadata to an external device.

[0182] The (one or more) sensors 2046 of the interface 2040 may measure physical quantities or detect the activation state of the electronic device 2000, and convert the measured or detected information into electrical signals. For example, the (one or more) sensors 2046 may include one or more cameras (e.g., Figure 1A ) or other imaging sensors for capturing images of a scene. The sensor(s) 2046 may also include any one or any combination of a microphone, a keyboard, a mouse, and one or more buttons for touch input. The sensor(s) 2046 may also include an inertial measurement unit. In addition, the sensor(s) 2046 may include control circuitry for controlling at least one of the sensors included herein. Any of these sensor(s) 2046 may be located within or coupled to the electronic device 2000.

[0183] Fig.16 is a flow chart showing a method of performing image authentication according to an embodiment of the present disclosure. The operation may be performed by an electronic device, a server, or any apparatus that performs image authentication.

[0184] In operation S1601, an input image is obtained. The electronic device may obtain an input image from a camera. The input image may be a raw image that is not processed by an image signal processor (ISP) and / or has a Bayer format.

[0185] In operation S1602, a processed image of an input image via an AI-based model is obtained. At least one processor of the electronic device may operate the AI-based image processing model to modify the input image, and output the modified input image as the processed image. The AI-based model may include one or more neural networks configured to perform demosaicing, denoising, super resolution, exposure correction, cropping, color correction, white balance, sharpening, unwanted object removal, new object addition, etc.

[0186] In operation S1603, authentication metadata indicating the fake pixels that have been generated by the AI-based model is generated. In an embodiment of the present disclosure, the authentication metadata may be implemented as a binary authentication mask that assigns a value of 0 or 1 to each pixel to indicate authenticity. For example, the electronic device may assign a value of "0" to an area in the binary authentication mask corresponding to a pixel determined to be authentic, and may assign a value of "1" to an area in the binary authentication mask corresponding to a pixel determined to be counterfeit.

[0187] In operation S1604, authentication metadata associated with the processed image is stored. The electronic device may store authentication metadata indicating the pseudo pixels associated with the processed image in a memory.

[0188] In operation S1605, the processed image with an indication of pseudo pixels is an output of the processing. In an embodiment of the present disclosure, the electronic device may output the processed image with an indication of pseudo pixels. The output image may be processed by multiple neural networks. For example, the multiple neural networks include a first neural network using a reconstruction loss in a first training step, and a second neural network jointly trained with a discriminator neural network using an adversarial loss. The output image may be an output image of the second neural network.

[0189] The image authentication method may be written as a computer-executable program or instruction that may be stored in a medium.

[0190] The medium can store computer executable programs or instructions continuously, or temporarily store computer executable programs or instructions for execution or downloading. In addition, the medium can be any of various recording media or storage media that combine single or multiple pieces of hardware, and the medium is not limited to the medium directly connected to the electronic device, but can be distributed on the network. Examples of media include magnetic media (such as hard disks, floppy disks, and tapes) configured to store program instructions, optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as floppy disks), and ROM, RAM, and flash memory. Other examples of media include recording media and storage media managed by application stores that distribute applications or by websites, servers, etc. that supply or distribute various other types of software.

[0191] The image authentication method may be provided in the form of downloadable software. The computer program product may include a product in the form of a software program that is electronically distributed through a manufacturer or an electronic market (e.g., a downloadable application). For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server or a storage medium of a server.

[0192] The above-mentioned model related to the neural network can be implemented via a software module. When the model is implemented via a software module (eg, a program module including instructions), the model can be stored in a computer-readable recording medium.

[0193] In addition, the model can be part of the above-mentioned electronic device by integrating in the form of a hardware chip. For example, the model can be manufactured in the form of a dedicated hardware chip for artificial intelligence, or can be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU).

[0194] Although embodiments of the present disclosure have been described with reference to the accompanying drawings, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope defined by the following claims.

[0195] According to one aspect of the present disclosure, there is provided an electronic device for performing image authentication on an image captured by an artificial intelligence (AI) camera module or processed by an AI image signal processor (ISP). The electronic device may include: at least one memory storing instructions; and at least one processor configured to execute the instructions to perform the following operations: obtain an input image; obtain a processed image of the input image via an AI-based model; generate authentication metadata indicating pseudo pixels that have been generated by the AI-based model; store the authentication metadata associated with the processed image in at least one memory; and output the processed image with an indication of pseudo pixels.

[0196] According to an embodiment of the present disclosure, the authentication metadata is a binary authentication mask indicating pseudo pixels generated by adding, deleting, or modifying at least one scene content included in the input image.

[0197] According to an embodiment of the present disclosure, the AI-based model includes multiple neural networks, and at least one processor is also configured to: based on each of the multiple neural networks that generate pseudo pixels, generate multiple authentication metadata for multiple processed images output from the multiple neural networks, respectively.

[0198] According to an embodiment of the present disclosure, at least one processor is further configured to identify at least one neural network among the plurality of neural networks that generates pseudo pixels based on the loss type of each of the plurality of neural networks. The at least one processor is further configured to generate authentication metadata for at least one processed image output from the at least one neural network, and skip the generation of authentication metadata for at least one remaining neural network among the plurality of neural networks that is identified as not generating pseudo pixels.

[0199] According to an embodiment of the present disclosure, at least one processor is further configured to identify at least one neural network trained using perceptual loss or adversarial loss as at least one neural network that generates pseudo pixels. At least one processor is further configured to identify a neural network trained using reconstruction loss or configured to change the color of an input image without changing scene content as at least one remaining neural network that does not generate pseudo pixels.

[0200] According to an embodiment of the present disclosure, at least one processor is also configured to generate authentication metadata by the following operations: applying a color transformation to an input image that maps the color of the input image to an output image of at least one neural network of an AI-based model to obtain a color-transformed image, and calculating a difference metric between the color-transformed image and the output image for each pixel.

[0201] According to an embodiment of the present disclosure, at least one processor is further configured to generate authentication metadata via an AI-based model including a generator neural network. The generator neural network includes a first neural network trained using a reconstruction loss in a first training step, and a second neural network jointly trained with a discriminator neural network using an adversarial loss in a second training step, wherein the network parameters of the first neural network are fixed in the second training step.

[0202] According to one aspect of the present disclosure, there is provided a method for performing image authentication on an image captured by an artificial intelligence (AI) camera module or processed by an AI image signal processor (ISP). The method may include: obtaining an input image; obtaining a processed image of the original image via an artificial intelligence (AI)-based model, generating authentication metadata indicating pseudo pixels that have been generated by the AI-based model, storing the authentication metadata associated with the processed image in at least one memory; and outputting the processed image with an indication of pseudo pixels.

[0203] According to an aspect of the present disclosure, a non-transitory computer-readable storage medium storing instructions is provided, the instructions being executed by at least one processor to perform operations corresponding to the methods described in the present disclosure.

Claims

1. An electronic device for performing image authentication on an image captured by an artificial intelligence (AI) camera module or processed by an AI image signal processor (ISP), the electronic device comprising: at least one memory (300) storing instructions; as well as At least one processor (200) is configured to execute the instructions to perform the following operations: Get the input image; obtaining a processed image of the input image via an AI-based model; generating authentication metadata indicating pseudo pixels that have been generated by the AI-based model; storing authentication metadata associated with the processed image in the at least one memory; as well as The processed image is outputted with an indication of the spurious pixels.

2. The electronic device according to claim 1, wherein: The authentication metadata is a binary authentication mask indicating pseudo pixels generated by adding, deleting or modifying at least one scene content included in the input image.

3. The electronic device according to any one of claims 1 to 2, wherein: The AI-based model includes a plurality of neural networks, and The at least one processor is further configured to: Based on each of the plurality of neural networks that generates pseudo pixels, a plurality of authentication metadata is generated for a plurality of processed images output from the plurality of neural networks, respectively.

4. The electronic device according to any one of claims 1 to 3, wherein: The AI-based model includes a plurality of neural networks, and The at least one processor is further configured to: identifying at least one neural network among the plurality of neural networks that generates a pseudo pixel based on a loss type of each of the plurality of neural networks; generating authentication metadata for at least one processed image output from the at least one neural network; as well as The generation of authentication metadata is skipped for at least one remaining neural network of the plurality of neural networks that is identified as not generating pseudo pixels.

5. The electronic device according to claim 4, wherein: The at least one processor is further configured to: identifying the at least one neural network trained using a perceptual loss or an adversarial loss as the at least one neural network that generates pseudo pixels; as well as A neural network trained using a reconstruction loss or configured to change the color of the input image without changing the scene content is identified as the at least one remaining neural network that does not generate pseudo pixels.

6. The electronic device according to any one of claims 1 to 5, wherein: The at least one processor is further configured to: Generate authentication metadata by doing the following: applying a color transform to the input image to obtain a color transformed image, the color transform mapping the colors of the input image to an output image of at least one neural network of the AI-based model; and A difference measure between the color transformed image and the output image is calculated for each pixel.

7. The electronic device according to any one of claims 1 to 6, wherein: The at least one processor is further configured to: generating authentication metadata via the AI-based model comprising a generator neural network; Wherein, the generator neural network comprises: a first neural network trained in a first training step using a reconstruction loss; and A second neural network is trained jointly with the discriminator neural network using an adversarial loss in a second training step, wherein network parameters of the first neural network are fixed in the second training step.

8. A method for performing image authentication on an image captured by an artificial intelligence (AI) camera module or processed by an AI image signal processor (ISP), the method comprising: Get the input image; Obtaining a processed image of the original image via a model based on artificial intelligence (AI); generating authentication metadata indicating pseudo pixels that have been generated by an AI-based model; storing authentication metadata associated with the processed image in at least one memory; as well as The processed image is outputted with an indication of the spurious pixels.

9. The method according to claim 8, wherein: The authentication metadata is a binary authentication mask indicating pseudo pixels generated by adding, deleting or modifying at least one scene content included in the original image.

10. The method according to any one of claims 8 to 9, wherein: The AI-based model includes a plurality of neural networks, and The method further comprises: Based on each of the plurality of neural networks that generates pseudo pixels, a plurality of authentication metadata is generated for a plurality of processed images output from the plurality of neural networks, respectively.

11. The method according to any one of claims 8 to 10, wherein: The AI-based model includes a plurality of neural networks, and The method further comprises: identifying at least one neural network among the plurality of neural networks that generates a pseudo pixel based on a loss type of each of the plurality of neural networks; generating authentication metadata for at least one processed image output from the at least one neural network; and The generation of authentication metadata is skipped for at least one remaining neural network of the plurality of neural networks that is identified as not generating pseudo pixels.

12. The method according to claim 11, further comprising: identifying the at least one neural network trained using a perceptual loss or an adversarial loss as the at least one neural network that generates pseudo pixels; as well as A neural network trained using a reconstruction loss or configured to change the color of the input image without changing the scene content is identified as the at least one remaining neural network that does not generate pseudo pixels.

13. The method according to any one of claims 8 to 12, wherein: Generating the authentication metadata includes: applying a color transform to the input image to obtain a color transformed image, the color transform mapping the colors of the input image to an output image of at least one neural network of the AI-based model; calculating, for each pixel, a difference metric between the color transformed image and the output image; and The authentication metadata is generated based on the difference metric.

14. The method according to any one of claims 8 to 13, wherein: Generating the authentication metadata includes: generating the authentication metadata via the AI-based model comprising a generator neural network; Wherein, the generator neural network comprises: a first neural network trained in a first training step using a reconstruction loss; and A second neural network is trained jointly with the discriminator neural network using an adversarial loss in a second training step, wherein network parameters of the first neural network are fixed in the second training step.

15. A machine-readable medium storing instructions executable by at least one processor to perform the method according to any one of claims 8 to 14.

Images