Verification of image authenticity

Abstract

This document describes a technique and apparatus for verifying the authenticity of an image. In embodiments, the method comprises the steps of: receiving an image (210) to be verified by a decoder system (220); performing feature recognition on the received image (210) to determine a plurality of determined features (238) of the received image (210); generating a first output (236) that defines a value representing the determined features (238) of the received image (210); decoding the received image (210) by a message decoding neural network (252) of the decoder system (220) to extract a signature (254) embedded in the received image (210) - the signature (254) represents the recovered features (258) of the received image (210). The method includes the step of generating a second output (256) that defines a value representing a recovered feature (258) of a received image (210); the step of providing the first output (236) and the second output (256) to a manipulation detection neural network (272) of a decoder system (220); and the step of generating an estimate of the authenticity of the received image (210) using at least the first output (236) and the second output (256) by the manipulation detection neural network (272).

Description

Technology Field

[0001] This specification relates to the verification of the authenticity of an image. Background Technology

[0002] Artificial neural networks and computing processes can be used to create manipulated or "fake" images by illicitly altering images to include information that did not exist when the image was originally created. For example, artificial neural networks can be used to create deep learning fakes ("deepfakes"), and a video of a person making a first statement can be modified to make it appear as if that person made a second different statement. In another example, a "face swapping" application can be used to replace the face of a first person with the face of a second person in a still image.

[0003] Various approaches have been developed to detect fake images. For example, algorithms can be used to detect subtle inconsistencies (e.g., lighting patterns) in images that indicate modulation. Although these approaches have been created, the reliable detection of fake images has proven to be a difficult and challenging problem. To overcome the limitations of current technology, a new framework is needed to verify the authenticity of images.

[0004] Techniques and devices for verifying the authenticity of an image are described. In particular, the techniques and devices may include or otherwise utilize one or more machine learning models to verify the authenticity of a received image in order to detect image manipulation. Verification is based on determining whether an image is fake or manipulated by comparing detected image features with recovered image features. The comparison of detected and recovered features can be used to identify features indicating a fake image that cannot be detected by a human observer.

[0005] In some implementations, a feature extraction process is used to extract determined features from an input image, and a message embedding process is used to generate an output image by embedding a signature containing the determined features into the input image. The feature extraction process is utilized to extract an embedded signature containing features from the received image and to recover features from the signature. Then, a verification process is utilized to verify the authenticity of the image, for example, by comparing the determined features with the recovered features.

[0006] The embodiments described below include a method performed by a system comprising an encoder system and a decoder system. In the method, the system (e.g., a decoder system) receives an image to be verified. The system performs feature recognition on the received image to determine a plurality of determined features of the received image. The system generates a first output defining a value representing the determined features of the received image. The system includes a message decoding neural network that decodes the received image to extract an embedded signature from the received image. The embedded signature represents a recovered feature of the received image. The system generates a second output defining a value representing the recovered feature. The system provides the first output and the second output to a manipulation detection neural network. The manipulation detection neural network generates an estimate of the authenticity of the received image using at least the first output and the second output.

[0007] The embodiments described below also include a computing device having a processor and a computer-readable storage medium storing instructions that cause the processor to execute a procedure for verifying the authenticity of an image in response to execution by the processor. The embodiments described below include a method for generating an image and a method for verifying the authenticity of an image. An optional feature of one embodiment, such as the method described above, may be combined with another embodiment. Brief explanation of the drawing

[0008] A technology and device for verifying the authenticity of images using a machine learning model are described with reference to the drawings. Throughout the drawings, the same number is used to refer to the same functions and components. Figure 1 illustrates an environment in which a technology and device for verifying the authenticity of an image using a machine learning model can be implemented. Figure 2 illustrates another environment in which technology and devices for verifying the authenticity of an image using a machine learning model can be implemented. FIG. 3 illustrates block diagrams of an exemplary user computing system, an exemplary server computing system, and an exemplary training computing system according to some embodiments of the present disclosure. Figure 4 illustrates an exemplary system for training a feature extraction neural network to extract features from an image. Figure 5 illustrates an exemplary system for training a message coding neural network to encode / decode images. Figure 6 is a diagram illustrating a method for generating an image to be verified. Figure 7 is a diagram illustrating a method for verifying the authenticity of an image. FIG. 8 illustrates a block diagram of an exemplary machine learning model according to exemplary embodiments of the present disclosure. Specific details for implementing the invention

[0009] outline

[0010] As described above, the present disclosure relates to a technology and apparatus for verifying the authenticity of an image. In particular, the system and method of the present disclosure may include or otherwise utilize one or more machine learning models to verify the authenticity of an image.

[0011] As used herein, the term “image” includes, but is not limited to, still images, moving images (e.g., video), moving images associated with audio, moving images associated with text, etc. As used herein, the term “video” means a series or time sequence of moving images that may or may not be associated with other types of information, such as audio or text. For example, video may include a time sequence of moving images that includes a stream of audio information. In this specification, the term “fake image” means an image that is presented as an original image but is a manipulated copy of the original image or a fake copy of the original image. As used herein, the term “image feature” includes, but is not limited to, an image-independent matrix of measurements across the entire image, facial expressions, landmarks, human face landmarks in the image, landmarks, key points, user-provided features, edges, corners, blouks, ridges, motion, optical flow, raw pixels of the image, etc.

[0012] As used herein, the phrase referring to “at least one of the item list” refers to any combination of such items including a single member. For example, “at least one of a, b, or c” is intended to include all combinations with multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc and ccc or other orders of a, b, c), as well as a, b, c, ab, ac, bc and abc.

[0013] FIG. 1 illustrates an exemplary system (100) using a feature recognition engine (120) (e.g., a feature extraction neural network (122)) configured to extract image features from an input image (102) and a message encoder (140) (e.g., a message encoding neural network (142)) configured to generate an output image (150) from an input image (102).

[0014] FIG. 2 illustrates an exemplary system (200) using a feature recognition engine (230) (e.g., a feature extraction neural network (232)) configured to extract features (238) from a received image (210), a message decoder (250) (e.g., a message decoding neural network (252)) configured to extract a signature (254) (hidden message) containing features (258) from a received image (210), and / or an authentication engine (270) (e.g., a manipulation detection neural network (272)) configured to verify the authenticity of a received image (210) based on processing of the received image (210). In an embodiment, the system may include one or both of the encoder system (110) of FIG. 1 configured to generate an output image and the decoder system (220) of FIG. 2 configured to decode the received image and determine the authenticity of the received image. At least one of the system (100), the component of the system (200), or the component of the system (200) may be implemented in one or more of a user computing device, an embedded computing device, a server computing device, a model processing device, a training device, a virtual computing device, other computing devices or computing infrastructure, or a combination thereof, as described below with respect to FIG. 3, for example.

[0015] As illustrated in FIG. 1, in the system (100), an input image (102) is received by an encoder system (110) as an input for processing. For example, a user may submit an image (e.g., a photograph) to the system (100) as the input image (102). In another example, the user may capture the input image (102) on an imaging device (e.g., a digital camera, a camera of a computing device), and the encoder system (110) is implemented in the image pipeline of the imaging device. In another example, the input image (102) may be stored in an image storage (104) and provided to the encoder system (110). The image storage (104) may be included in the system (100) (e.g., by one or more local wired or wireless connections) or may be remote to the system (100) and may communicate with the system (100) via one or more wired or wireless connections (e.g., a local area network (LAN) or a wide area network (WAN) connection). The image storage (104) may be, for example, a database stored locally in the system (100), a database hosted on a server remote to the system (100), or a memory device implemented in a computing device.

[0016] A system (100) (e.g., an encoder system (110)) receives an input image (102), selects one or more features (128) of the input image (102), generates one or more signatures (144) based on the selected feature(s) (128), and embeds the signatures (144) into the input image (102) to generate an output image (150). In some embodiments, the output image (150) may be stored in a memory device.

[0017] A system (100) (e.g., an encoder system (110)) includes a feature recognition engine (120). The feature recognition engine (120) is configured to perform a feature extraction process on an input image (102) to determine a plurality of determined features (128) of an input image (102). For example, the feature recognition engine (120) may be configured to select a plurality of features from the input image (102) and process the selected image features (128) to generate an output (126) (e.g., data) that defines a value representing the determined features of the input image. In an embodiment, the feature recognition engine (120) selects a plurality of features from the input image (102) and processes the selected image features (128) to generate an output (126) in which each represents a selected feature (128) of the input image (102). For example, the features (128) of the input image (102) may include facial expressions, facial landmarks of a person in the image, landmarks, key points, user-provided features, edges, corners, blogs, ridges, motion, optical flow, raw pixels of the image, and an image-independent matrix of measurements.

[0018] The feature recognition engine (120) may include a feature extraction neural network (122). The feature extraction neural network (122) may be configured to receive an input image (102) and process the input image (102) to generate an output (126) that defines a value representing a selected feature (128) of the input image (102). In embodiments, each of the values ​​represents a selected feature (128) of the input image (102). For example, the feature extraction neural network (122) may receive the input image (102) and process the input image (102) or a part thereof to generate an output (126) that defines a value representing one or more image features corresponding to a facial expression, landmark, key point, edge, corner, blog, ridge, etc. In embodiments, the output (126) defines a value representing one or more image features corresponding to a facial expression, landmark, key point, edge, corner, blog, ridge, etc. In other implementations, the feature recognition engine (120) can generate image features through the use of other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transforms, or other algorithms.

[0019] The feature recognition engine (120) may receive one or more user-provided features (user-provided features) (124) provided by one or more user input components configured to receive input (selected image features) from a user. The user-provided features (124) may include facial expressions, human face landmarks in the image, raw pixels of the image, etc. The feature recognition engine (120) may include a feature extraction neural network (122) and user-provided features (124).

[0020] The system (100) (e.g., encoder system (110)) also performs a message embedding process for an input image (102). The system (100) transmits an output (126) specifying image features (128) derived from the input image (102) to a message encoder (140). The message encoder (140) may include a message encoding neural network (142). The message encoding neural network (142) may be configured as a message coding neural network (e.g., a message encoding / decoding neural network) used to encode and decode messages. The message encoding neural network (142) of FIG. 1 and the message decoding neural network of FIG. 2 are message encoding / decoding neural networks.

[0021] A message encoder (140) (e.g., a message encoding neural network (142)) receives an output (126) that specifies image features (128) derived from an input image (102) from the system (100) and generates a signature (144) representing the image features (128) of the input image (102) (e.g., including an output (126)). The message encoder (140) (e.g., a message encoding neural network (142)) processes the input image (102) by embedding the signature (144) (message) into the input image (102) as a digital message (e.g., a steganographic signal) to generate an output image (150). In some embodiments, the signature (144) is a perceptually invisible watermark.

[0022] Weights can be shared between the feature recognition engine (120) shown in FIG. 1 (e.g., first feature extraction neural network (122), user-provided features (124)) and the second feature recognition engine (230) shown in FIG. 2 (e.g., second feature extraction neural network (232), user-provided features (234)).

[0023] FIG. 2 illustrates a system (200) comprising a decoder system (220) configured to decode a received image (210) and verify the authenticity of the received image (210). The received image (210) is an image to be verified. In embodiments, the received image (210) is a copy of the output image (150). In embodiments, the received image (210) is a fake copy of the input image (102) or the output image (150). In embodiments, the received image (210) is a manipulated copy of the input image (102) or the output image (150). The decoder system (220) determines the authenticity of the received image (210), for example, whether the received image (210) is the output image (150) or whether the received image (210) is a fake version of the input image (102).

[0024] The system (100) may send an output image (150) generated by the encoder system (110) to the decoder system (220) as a received image (210). In some implementations, the received image (210) is provided to the decoder system (220) of the system (200); for example, a user may submit a photo to the system (200) as the received image (210). In other examples, the received image (210) may be provided to the decoder system (220) by an image repository such as the image repository (104) of FIG. 1.

[0025] In one example, a user uses the system (100) and / or the system (200) to verify the authenticity of an image. In another example, an online service provider utilizes the system (100) and / or the system (200) to verify the authenticity of an image. In another example, a third party utilizes the system (100) and / or the system (200) to verify the authenticity of an image.

[0026] The system (200) provides the received image (210) as input to the decoder system (220), and the decoder system (220) determines whether the received image (210) is a trusted copy (e.g., an authentic copy, an unaltered copy) or whether the received image (210) is an untrusted copy, e.g., a fake image or an altered copy of the output image (150) or the input image (102). As illustrated in FIG. 21, the system (200) (e.g., the decoder system (220)) includes a feature recognition engine (230), a message decoder (250), and an authentication engine (270).

[0027] The system (200) may provide the received image (210) as input to the feature recognition engine (230). The feature recognition engine (230) is configured to perform a feature extraction process (feature recognition) on the received image (210) to determine a plurality of determined features (238) of the received image (210). For example, the determined features (238) of the received image (210) may include an image-independent matrix of facial expressions, landmarks, key points, user-provided features, edges, corners, blogs, ridges, motion, optical flow, pixels, and measurements.

[0028] The feature recognition engine (230) selects a plurality of determined features (238) from the received image (210) and processes the determined features (238) to generate an output (236) (data) that defines a value representing the determined features of the received image (210). In an embodiment, the feature recognition engine (230) selects a plurality of determined features (238) from the received image (210) and processes the determined features (238) to generate an output (236) (data) that defines a value representing the determined features of the received image, each representing a selected feature of the received image (210). The output (236) of the determined features (238) is transmitted to the authentication engine (270) by the system (200).

[0029] The feature recognition engine (230) may include a second feature extraction neural network (232). The feature extraction neural network (232) performs a feature extraction process on the received image (210). The feature extraction neural network (232) may be configured to receive the received image (210) and process the received image (210) to generate an output (236) (data) that defines a value representing a determined feature (238) of the received image (210). For example, the feature extraction neural network (232) may be configured to receive the received image (210) and process the received image (210) to generate an output (236) that defines a value representing a determined feature (238) of the received image (210). In some implementations, the feature extraction neural network (232) is the same neural network as the feature extraction neural network (122) of FIG. 1. In some implementations, the feature extraction neural network (232) is a different feature extraction neural network from the feature extraction neural network (122) of FIG. 1.

[0030] The feature recognition engine (230) may receive an input comprising one or more user-provided features (234) (e.g., the user input component (322) of FIG. 3) provided by one or more user input components configured to receive input (selected image features) from a user. The user-provided features (234) may include, for example, facial expressions, human face landmarks in the image, raw pixels of the image, etc. In some implementations, the feature recognition engine (230) may include a second feature extraction neural network (232) and user-provided features (234).

[0031] The received image (210) may be provided as an input to a message decoder (250) of a decoder system (220). The message decoder (250) (e.g., a message decoding neural network (252)) decodes the received image (210). For example, the message decoder (250) may decode the received image (210) by performing a message extraction process on the received image (210). In the message extraction process, the message decoder (250) processes the received image (210) to extract a signature (254) embedded in the received image (210). The signature (254) represents a recovered feature (258) embedded in the received image (210) by a message encoder (e.g., the message encoder (140) of FIG. 1). In the embodiments, the system (200) detects whether a signature exists in the received image (210).

[0032] The message extraction process includes generating a second output (256) (data) that defines a value representing a recovered feature (258) of the received image (210). In an embodiment, the message extraction process includes generating a second output (256) that defines a value representing each of the recovered features (258) of the received image (210) recovered from the signature (254). The output (256) of the recovered feature (258) is transmitted by the system (200) to an authentication (authenticity) engine (270).

[0033] The output (236) from the feature recognition engine (230) and the output (256) (second output) of the message decoder (250) are provided to and used by the authentication engine (270) to generate a prediction (e.g., an estimate) about the authenticity of the received image (210). For example, the authentication engine (270) of the decoder system (220) determines whether the received image (210) is a trusted copy (e.g., an authentic, unaltered copy) or whether the received image (210) is an untrusted copy, e.g., a fake image or an altered copy.

[0034] The authentication engine (270) may include a manipulation detection neural network (272) that receives outputs (e.g., output (236), output (256)). The manipulation detection neural network (272) generates an estimate (prediction) of the authenticity of the received image (210) by using at least output (236) and output (256), for example, by comparing output (236) with output (256). Based on the estimate generated by the authentication engine (270),

[0035] The system (100) determines whether the received image (210) is a reliable copy (e.g., genuine, unaltered copy) or whether the received image (210) is an unreliable copy, e.g., a fake image or an altered copy.

[0036] Exemplary Examples

[0037] The system and method of the present disclosure may be implemented or otherwise executed by one or more computing systems. An exemplary computing system of the computing system (300) includes a user computing device (e.g., a mobile computing device such as a laptop, desktop, tablet, smartphone, wearable computing device, camera, etc.); an embedded computing device (e.g., a device embedded in a vehicle, camera, image sensor, industrial machine, satellite, game console or controller, home appliance such as a refrigerator, thermostat, energy meter, home energy manager, smart home assistant, etc.); a server computing device (e.g., a database server, parameter server, file server, mail server, print server, web server, game server, application server, etc.); a dedicated, specialized model processing or training device; a virtual computing device; and one or more of other computing devices or computing infrastructure or combinations thereof.

[0038] For example, FIG. 3 illustrates a block diagram of an exemplary computing system (300) capable of verifying the authenticity of an image according to an exemplary embodiment of the present disclosure. The computing system (300) includes one or more of a user computing system (302), a server computing system (330), or a training computing system (350) that are communicably connected via a network (380).

[0039] The server computing system (330) may include or be implemented by one or more server computing devices. If the server computing system (330) includes multiple server computing devices, these server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof. In some implementations, the training computing system (350) includes or is otherwise implemented by one or more server computing devices.

[0040] A user computing system (302) includes one or more processors (312) and one or more memory devices (314). A server computing system (330) includes one or more processor(s) (332) and memory devices (334). A training computing system (350) includes one or more processor(s) (352) and memory devices (354). The processor(s) (312, 332, 352) may be any suitable processing unit (e.g., a central processing unit (CPU); a visual processing unit (VPU); a graphics processing unit (GPU); a tensor processing unit (TPU); a neural processing unit (NPU); a neural processing engine; a core of a CPU, VPU, GPU, TPU, NPU, or other processing unit; an application-specific integrated circuit (ASIC); a field-programmable gate array (FPGA); a coprocessor; a controller; or a combination of the processing units described above) and may be a single processor or a plurality of processors operablely connected. The processor(s) (312, 332, 352) can be embedded within other hardware components, such as image sensors, accelerometers, etc.

[0041] Memory device(s) (314, 334, 354) may include one or more non-transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory device, magnetic disk, etc., and combinations thereof. Memory device(s) (314) may store data (316) and instructions (318) that are executed by processor(s) (312) to enable the user computing system (302) to perform operations. Memory device(s) (334) may store data (336) and instructions (338) that are executed by processor(s) (332) to enable the server computing system (330) to perform operations. Memory device(s) (354) may store data (356) and instructions (358) that are executed by processor(s) (352) to enable the training computing system (350) to perform operations.

[0042] One or more of the user computing system (302), server computing system (330), or training computing system (350) may include an encoder system (e.g., the encoder system (110) of FIG. 1). An exemplary encoder system may include a feature recognition engine (e.g., the feature recognition engine (120) of FIG. 1) and a message encoder (e.g., the message encoder (140) of FIG. 1). In some implementations, one or more of the user computing system (302), server computing system (330), or training computing system (350) may include a decoder system (e.g., the decoder system (220) of FIG. 2). An exemplary decoder system may include at least one of a feature recognition engine (e.g., the feature recognition engine (230) of FIG. 2), a message decoder (e.g., the message decoder (250) of FIG. 2), or an authentication engine (e.g., the authentication engine (270) of FIG. 2).

[0043] The user computing system (302) may store or include one or more machine learning models (320). For example, the machine learning model (320) may be or include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models including non-linear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. More specifically, the machine learning model (320) may include one or more of a feature extraction network (e.g., the feature extraction neural network (122) of FIG. 1 and / or the second feature extraction neural network (232) of FIG. 2), a message encoding neural network (e.g., the message encoding neural network (142) of FIG. 1), a message decoding neural network (e.g., the message decoding neural network (252) of FIG. 2), or a manipulation detection neural network (e.g., the manipulation detection neural network (272) of FIG. 2).

[0044] Additionally or alternatively, one or more machine learning models (340) may be included in a server computing system (330) that communicates with a user computing system (302) according to a client-server relationship, or otherwise stored and implemented. For example, the machine learning models (340) may be implemented by the server computing system (340) as part of a web service (e.g., an extreme multiclass or multilabel classification service, a language modeling service, a metric learning service). Thus, one or more models (320) may be stored and implemented in the user computing system (302) and / or one or more models (340) may be stored and implemented in the server computing system (330). In an implementation, the machine learning model (340) may include one or more of a feature extraction network (e.g., the feature extraction neural network (122) of FIG. 1 and / or the feature extraction neural network (232) of FIG. 2), a message encoding neural network (e.g., the message encoding neural network (142) of FIG. 1), a message decoding neural network (e.g., the message decoding neural network (252) of FIG. 2), or a manipulation detection neural network (e.g., the manipulation detection neural network (272) of FIG. 2).

[0045] The user computing system (302) may also include one or more user input components (322) that receive user input. For example, the user input component (322) may be a touch detection component (e.g., a touch detection display screen, a touchpad) sensitive to the touch of a user input object (e.g., a finger, a stylus). The touch detection component may serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which the user can provide user input.

[0046] As described above, the server computing system (330) may store or include one or more machine learning models (340). For example, the model (340) may be or include various machine learning models. Examples of machine learning models include neural networks or other multilayer nonlinear models. Examples of neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0047] The computing system (300) may include an image storage (104). For example, the image storage (104) may be included in the system (300) by one or more local wired or wireless connections, may be remote to the system (300), and may communicate with the system (300) through one or more wired or wireless connections, for example, a local area network (LAN) or a wide area network (WAN) connection. The image storage (104) may be, for example, a database stored locally in the user computing system (302) or a database hosted on a server remote to the system (300) (e.g., a server computing system (330), a training computing system (350)).

[0048] The user computing system (302) and / or server computing system (330) can train a model (320 and / or 340) through interaction with a training computing system (350) that is communicatedly connected via a network (380). The training computing system (350) may be separate from the server computing system (330) or may be part of the server computing system (330).

[0049] The training computing system (350) may include a model trainer (trainer) (360) that trains a machine learning model (320 and / or 340) stored in a user computing system (302) and / or a server computing system (330) using various training or learning techniques, such as comparing determined features and recovered features with the machine learning model (320 and / or 340) and verifying the authenticity of an image based on backpropagation of errors. In some implementations, performing backpropagation of errors may include performing backpropagation truncated over time. The model trainer (360) may perform a number of generalization techniques (e.g., weight devaluation, dropout) to improve the generalization ability of the model being trained.

[0050] The model trainer (360) can train a machine learning model (320 and / or 340) based on a set of training data (362). The training data (362) may include, for example, an example of input data to which a label corresponding to the output data has been assigned.

[0051] Training examples may be provided in the user computing system (302) if the user consents. Thus, in such implementations, the model (320) provided to the user computing system (302) may be trained by the training computing system (350) on user-specific data received from the user computing system (302). In some cases, this process may be referred to as model personalization.

[0052] The model trainer (360) includes computer logic utilized to provide desired functions. The model trainer (360) may be implemented as hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, the model trainer (360) includes a program file that is stored in a storage device, loaded into memory, and executed by one or more processors. In other implementations, the model trainer (360) includes one or more computer-executable instruction sets stored in a computer-readable storage medium of a type such as a RAM hard disk or optical or magnetic media.

[0053] The network (380) may be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. Generally, communication through the network (380) may be transmitted via any type of wired and / or wireless connection using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or format (e.g., HTML, XML), and / or protection methods (e.g., VPN, secure HTTP, SSL).

[0054] FIG. 3 illustrates an example of a computing system (300) that can be used to implement the present disclosure. Other computing systems may also be used. For example, in some implementations, a user computing system (302) may include a model trainer (360) and training data (362). In such implementations, a model (320) may be trained and used locally in the user computing system (302). In some such implementations, the user computing system (302) may implement the model trainer (360) to personalize the model (320) based on user-specific data.

[0055] Training

[0056] The framework described herein can be trained in various ways depending on how, for example, feature extraction neural networks, message coding neural networks (e.g., message encoding neural networks, message decoding neural networks, message encoding / decoding neural networks), and / or manipulation detection neural networks are trained. In some implementations, one or more neural networks may be trained separately. In some implementations, one or more of the neural networks are co-trained. In other implementations, all neural networks are trained together.

[0057] FIG. 4 illustrates an exemplary system (400) configured to train a first feature extraction neural network (422) (e.g., the first feature extraction neural network (122) of FIG. 1 and / or the second feature extraction neural network (232) of FIG. 2) to extract features from an image (e.g., input image (416)) with improved accuracy. In embodiments, the feature extraction neural network (422) is utilized in a system configured to verify the authenticity of an image (e.g., the system (100) of FIG. 1, the system (200) of FIG. 2).

[0058] The feature extraction neural network (422) may be trained separately from one or more of the message encoding neural network, the message decoding neural network, or the manipulation detection neural network. For example, in operation (412), an input image (416) is provided to an encoder system (410) that includes a feature recognition engine (420) for processing. For example, a user may submit an image (e.g., a photograph) to the system (400) as the input image (416). In another example, the feature recognition engine (420) is the feature extraction neural network (422), and an image repository (104) (as described in relation to FIG. 2) provides the input image (416) to the feature recognition engine (420). In an implementation, the feature recognition engine (462) may generate determined features (418) through the use of other feature extraction techniques and algorithms that do not rely on the neural network, such as principal component analysis, edge detection, Hough transform, or other algorithms.

[0059] A feature recognition engine (420) (e.g., a feature extraction neural network (422)) is configured to estimate (determine) features of an input image (416) and to generate an output of determined features (418) (e.g., features (128) of FIG. 1). In some embodiments, the feature extraction neural network (422) estimates determined features (418) having features that can be recognized as features learned by image recognition. The learned features must be sensitive to fake image detection; for example, if the image is manipulated or fake, the learned features are significantly different.

[0060] Optionally or additionally, the determined feature (418) may be extracted from the input image (416) through the use of user input, such as user-provided feature (424) (e.g., user-provided feature (124) described herein). In an implementation, the second feature recognition engine (462) may generate the recovered feature (466) through the use of other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transform, or other algorithms.

[0061] After the determined feature (418) is estimated, the feature recognition engine (420) sends the output (430) of the determined feature (418) to the discrimination feature loss computation engine (470). The determined feature (418) is provided to the discrimination feature loss computation engine (470) so that the discrimination feature loss computation engine (470) can calculate the feature loss (472) used to train the feature extraction neural network (422). This process may be repeated until the calculated loss is below a predetermined threshold or until other convergence criteria are met.

[0062] The feature recognition engine (420) may also generate an output image (450) by transmitting an output (426) having determined features (418) to a message encoder (440) including a message encoding neural network (442) (e.g., 142 in FIG. 1). In embodiments, one or more signature(s) are generated based on the determined features(s) (428), and the signature(s) are embedded in the input image (416) as digital messages (e.g., steganographic signals) to generate the output image (450). In an implementation, the feature recognition engine (420) may generate the determined features (418) through the use of other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transform, or other algorithms.

[0063] The feature extraction process is performed on an image (e.g., input image (416), received image (452)) by a feature recognition engine, for example, a first feature recognition engine (420) (e.g., a first feature extraction neural network (422) (e.g., the first feature extraction neural network (122) of FIG. 1)) and / or a second feature recognition engine (462) (e.g., a second feature extraction neural network (464) (e.g., the feature extraction neural network (232) of FIG. 2)). For example, as illustrated in FIG. 4, the decoder system (460) includes a second feature recognition engine (462) (e.g., a feature extraction neural network (464)). The second feature recognition engine (462) decodes the received image (452) (e.g., as described above in relation to FIG. 2) to extract a signature from the received image (452). The signature represents the recovered features (466) of the received image (452).

[0064] The feature recognition engine (e.g., the second feature recognition engine (462)) sends the output (468) along with the recovered features (466) to the identification feature loss calculation engine (470). The recovered features (466) are provided so that the identification feature loss calculation engine (470) can calculate a feature loss (identification feature loss) (472) that can be used to train the feature extraction neural network (422).

[0065] The system may perform a message coding neural network training procedure in which a discrimination feature loss calculation engine (470) applies a discrimination feature loss function to a recovered feature (466) and a determined feature (418) to determine an image feature loss (472) caused by an estimate performed by a feature extraction neural network (422). The discrimination feature loss function compares the determined feature (418) of the input image (416) (e.g., selected feature (128) of the input image (102) of FIG. 1) with the recovered feature (466) of the received image (452) (e.g., recovered feature (258) of the received image (210) of FIG. 2). If the tested image (e.g., received image (210) of FIG. 2) is genuine (e.g., not a fake copy of the output image (450), and not a manipulated copy of the output image (450)), the discrimination feature loss function should be small. However, if the tested image (e.g., received image (452)) is not genuine (e.g., a fake copy of the output image (450), a manipulated copy of the output image (450)), the identification feature loss function must be large.

[0066] The identification feature loss calculation engine (470) sends the feature loss (472) as output to the feature recognition engine (420) to be used for training the feature extraction neural network (422). The loss function determines the deviation between the determined features (418) estimated by the feature recognition engine (420) and the recovered features (466) generated by the second feature recognition engine (462). In some implementations, the loss function may represent the deviations as a loss of accuracy of the recovered features (466) from the received image (452) because the received image (452) is a fake image.

[0067] After the loss is calculated, the feature loss (472) is used to further optimize the feature extraction neural network (422). For example, the feature loss (472) can be used to perform parameter optimization for the layers of the feature extraction neural network (422).

[0068] This training loop can be repeated for multiple images from the image repository (104) to iteratively optimize the feature extraction neural network (422). Parameter optimization enables the feature extraction neural network (422) to more accurately estimate the features depicted in the images. Over time, through sufficient training via iterations of the training loop, the feature extraction neural network (422) can be improved to minimize the loss function so that the recovered features (466) provided to the identified feature loss calculation engine (470) by the second feature recognition engine (462) converge to the determined features (418) provided to the identified feature loss calculation engine (470) by the first feature recognition engine (420). The process illustrated in FIG. 4 can be repeated until the feature loss is below a predetermined threshold, or until other convergence criteria are met.

[0069] FIG. 5 illustrates an exemplary system (500) configured to train a message coding neural network (e.g., a message encoding / decoding neural network). For example, a message encoding neural network of a message encoder (540) (e.g., the message encoding neural network of FIG. 5 (542), the message encoding neural network of FIG. 1 (142)) and / or a message decoding neural network of a message decoder (561) of a decoder system (560) (e.g., the message decoding neural network of FIG. 5 (562), the message decoding neural network of FIG. 2 (252)) encode an output image and / or decode a received image with enhanced accuracy. In embodiments, at least one of the message coding neural network or the message decoding neural network is trained separately from a feature extraction neural network and / or a manipulation detection neural network. In embodiments, the message coding neural extraction neural network is co-trained with one or more of the feature extraction neural network or the manipulation detection neural network.

[0070] In operation (504), the input image (502) is provided to the feature recognition engine (520) for processing. For example, as described above with respect to FIG. 4, a user may submit an image (e.g., a photograph) to the system (500) as the input image (502). In an implementation, the input image (502) is modified to train at least one of the message encoding neural networks (542) to be robust to this kind of distortion. For example, the input image (502) may be modified (e.g., edited, resized, cropped) or may be a fake image.

[0071] In another example, the feature recognition engine (520) includes a first feature extraction neural network (522) of the encoder system (510). An image repository (104) (e.g., as described in relation to the image repository (104) of FIG. 4) provides an input image (502) to the feature recognition engine (520) as described in relation to FIG. 4. In operation (506), the input image (502) is also provided to an image loss calculation engine (580) for processing.

[0072] A feature recognition engine (520) (e.g., a feature extraction neural network (522)) estimates (determines) features of an input image (502) and generates an output of determined features (528). In embodiments, the feature extraction neural network (522) estimates determined features (528) having features that can be recognized as features learned by image recognition. The learned features must be sensitive to fake image detection, for example, if the image is manipulated or fake, the learned features are significantly different. In embodiments, user-provided features (e.g., user-provided features (124) of FIG. 1 and / or user-provided features (234) of FIG. 2)) may be provided by a user to be included in the determined features (528).

[0073] Optionally or additionally, the determined features (528) may be extracted from the input image (502) through the use of user-provided features as described above in relation to FIG. 4. In an implementation, the feature recognition engine (520) may generate the determined features (528) through the use of other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transform, or other algorithms.

[0074] After the estimated determined feature (528), the feature recognition engine (520) transmits the output (530) (third output) of the determined feature (528) to the feature loss calculation engine (identification feature loss calculation engine) (570). The determined feature (528) is provided to the feature loss calculation engine (570) so that the feature loss calculation engine (570) can calculate the feature loss (572) provided as an input to the total loss calculation engine (590).

[0075] The feature recognition engine (520) also transmits the output (526) having the determined feature (528) to the message encoding neural network (542) of the message encoder (540) (e.g., the message encoding neural network (142 in FIG. 1)) to generate an output image (550). In embodiments, one or more signature(s) are generated based on the determined feature(s) (528), and the signature(s) are embedded in the output image (550) as a digital message (e.g., a steganographic signal) to generate the output image (550).

[0076] In operation (554), the received image (552) is transmitted to an image loss calculation engine (580) for processing. In operation 556, the received image (552) is transmitted to a decoder system (560). In embodiments, the decoder system (560) includes a message decoder (561). In embodiments, the message decoder (561) includes a message decoding neural network (562). A message decoding neural network (562), such as the message decoding neural network (252) of FIG. 2, decodes the received image (552) to extract a signature from the received image (552). The signature represents a recovered feature (564) of the received image (552). In an implementation, the message decoding neural network (252) is a message encoding / decoding neural network configured to encode and decode images.

[0077] The message decoding neural network (562) is configured to transmit an output (563) (second output) containing recovered features (564) to a feature loss calculation engine (570). The recovered features (564) are provided to the feature loss calculation engine (570) so that the feature loss calculation engine (570) can calculate the feature loss (572) provided as an input to the total loss calculation engine (590).

[0078] The system (500) determines a feature loss (572). In an implementation, the system (500) includes a feature loss calculation engine (570) that applies a loss function to determine the feature loss (572). The loss function determines the deviation between the determined features (528) estimated by the feature recognition engine (520) in relation to the features of the input image (502) and the recovered features (564) extracted by the message decoding neural network (562) in relation to the features of the received image (552). In some implementations, the loss function may represent the deviations as a loss of accuracy of the recovered features (564) from the received image (552) because the received image (552) is a fake image. After the loss is calculated, the feature loss (572) is transmitted to the total loss calculation engine (590) as an output (first loss). The feature loss (572) is provided to the total loss calculation engine (590) to further optimize one or more of the feature extraction neural network (522), message encoding neural network (542), message decoding neural network (562), or message coding neural network (e.g., message encoding / decoding neural network).

[0079] The system (500) determines an image loss (582). In an implementation, the system (500) includes an image loss calculation engine (580) that receives an input image (502) and a received image (552) and applies a loss function to determine the image loss (582). The loss function determines the deviation between the input image (502) and the received image (552). In some implementations, the loss function may represent the deviation of the received image (552) as a result of a fake image or a manipulated image. After the image loss (582) is calculated, the image loss (582) is transmitted to a total loss calculation engine (590) and may be used to further optimize one or more of a feature extraction neural network (522), a message encoding neural network (542), or a message decoding neural network (562).

[0080] The system (500) determines a total loss (592) based on image loss (582) and feature loss (572). In embodiments, the system (500) includes a total loss calculation engine (590) that applies a loss function to the feature loss (572) and image loss (582) to determine the total loss (592). The total loss (592) may be caused by an estimate performed by at least one of the first feature extraction neural network (522) or the message decoding neural network (562). The loss function determines the deviation between the feature loss (572) calculated by the feature loss calculation engine (570) and the image loss (582) calculated by the image loss calculation engine (580).

[0081] The total loss function to be optimized can be represented by Equation 1.

[0082]

[0083] Here, the total loss is equal to the sum of the feature loss function (features, features recovered from the input image) and the image loss function (input image, output image). In the mathematical formula, the total loss is L, and the image loss function is L il , the feature loss function is L fl , the input image I i , the output image is I o , the feature is f, and the feature recovered from the input image is f ii am.

[0084] After calculating the total loss, the calculated total loss (592) is used to further optimize at least one of the feature extraction neural network (522), the message decoding neural network (562), or the message coding neural network (e.g., the message encoding / decoding neural network). For example, the calculated total loss (592) may be used to perform parameter optimization for one or more layers of the feature extraction neural network (522) or the message decoding neural network (562). The process illustrated in FIG. 5 may be repeated until the calculated total loss is below a predetermined threshold or until other convergence criteria are met. This training loop may be repeated for multiple images from the image repository (104) to iteratively optimize one or more of the feature extraction neural network (522), the message encoding neural network (542), or the message decoding neural network (562). In the implementation, parameter optimization enables the feature extraction neural network (522) to more accurately estimate the features depicted in the images. Over time, through sufficient training via iterations of the training loop, the feature extraction neural network (522) can be improved so that the recovered features (536) provided to the feature loss calculation engine (570) by the message decoding neural network (562) converge to the determined features (528) provided to the feature loss calculation engine (570) by the feature extraction neural network (522).

[0085] In an implementation, a manipulation detection neural network (e.g., the manipulation detection neural network (272) illustrated in FIG. 2) is trained separately from one or more of a message encoding neural network (e.g., the message encoding neural network (142) of FIG. 1), a message decoding neural network (e.g., the message decoding neural network (252) of FIG. 2, the message decoding neural network (562) of FIG. 5), or a feature extraction neural network (e.g., the feature extraction neural network (122) of FIG. 1, the feature extraction neural network (232) of FIG. 2, the feature extraction neural network (422) of FIG. 4, the second feature extraction neural network (464) of FIG. 4, and the first feature extraction neural network (522) of FIG. 5).

[0086] To train a manipulation detection neural network, at least one value of a feature extraction neural network, a message encoding neural network, a feature extraction neural network, or a message decoding neural network is fixed, and a training image is generated. The training image may include at least one image manipulation. In an embodiment, the input image may be a training image. In embodiments, the received image may be a training image.

[0087] The system sends training images to a manipulation detection neural network. The manipulation detection neural network determines the image manipulations applied to the training images. The manipulation detection neural network calculates a normal loss based on a normal loss function by utilizing the image manipulations applied to the training images. In the implementation, the normal loss is the cross-entropy loss. The manipulation detection neural network is trained based on at least the calculated normal loss. The loss function applied in the modality is a normal loss function for classification.

[0088] A feature extraction neural network can be co-trained with a message coding neural network (e.g., a message encoding / decoding neural network, a message encoding neural network, or one or more of a message encoding / decoding neural network) to extract features from images and / or encode / decode messages from images with improved accuracy. For example, through a combination of the separate feature extraction neural network training described above and the separate message encoding / decoding neural network training described above, the total loss function to be optimized can be expressed by the following Equation 2.

[0089]

[0090] Here, the total loss is equal to the image loss function (input image, output image) + feature loss function (features, features recovered from the input image) + identification feature loss function (features, features of the received image). In Equation 2, the total loss is L, and the image loss function is L il , the feature loss function is L fl , the input image I i , the output image is I o , the feature is f, and the feature recovered from the input image is f ii , the identification feature loss function is L dfl , , the characteristics of the received image are f r i am.

[0091] The input image may be perceptually identical to the output image, and even if some image manipulation (or fake) is present, the features recovered from the input image are as close as possible to the features extracted (recovered) from the input image. In this case, the learned features may be features that are easy for the message encoding neural network to hide and extract. Additionally, the learned features may be sensitive to image manipulation (or fake), and consequently, if the image is altered (e.g., manipulated, fake), the extracted features may differ significantly from the features extracted from the input image. In this manner, the manipulation detection neural network can be trained separately as described above.

[0092] A feature extraction neural network can be co-trained with a manipulation detection neural network to extract features from an image and / or to determine with enhanced accuracy whether the received image is a fake or manipulated version of the input image. In such an embodiment, a message coding neural network (e.g., the message encoding neural network (142) of FIG. 1 and the message decoding neural network (252) of FIG. 2) may be trained first. To train the message coding neural network, the image manipulation (or fake) to be detected may be applied to any input image having any features.

[0093] The total loss function to be optimized can be expressed by Equation 3.

[0094]

[0095] Here, the total loss is equal to the sum of the image loss function (input image, output image) and the feature loss function (random feature, recovered random feature). In Equation 3, the total loss is L, and the image loss function is L il , the feature loss function is L f , random features F ra , the recovered random features f rr am.

[0096] The input image can be perceptually identical to the output image, and random features are recovered from the input image. Then, the weights of the message coding neural network are fixed, and the feature extraction network and the manipulation detection neural network are trained together. Subsequently, the image manipulation or fake to be detected may or may not be applied randomly.

[0097] The total loss function to be optimized can be expressed by the following mathematical equation 4.

[0098]

[0099] Here, the total loss is equal to the sum of the cross-entropy loss (fake or real) to the identification feature loss function (features, features of the received image). In Equation 4, the total loss is L, and the identification feature loss function is L dfl , the characteristics of the received image are f r i , cross-entropy loss (fake or real) is L ce Through this training process, the feature extraction neural network learns the features best suited for detecting fake or real images.

[0100] A message coding neural network (e.g., the message encoding neural network (142) of FIG. 1 and the message decoding neural network (252) of FIG. 2)) can be co-trained with a manipulation detection neural network to encode / decode the message of an image and / or determine with improved accuracy whether the received image is a fake or manipulated version of the input image. In this embodiment, the feature extraction neural network is trained separately as described above. After the feature extraction neural network is trained, its weights are fixed. To co-train the message coding neural network and the manipulation detection neural network, the image manipulation or fake to be detected is randomly applied or not applied. The total loss function to be optimized can be represented by the following Equation 5.

[0101]

[0102] Here, the total loss is equal to the image loss function (input image, output image) + feature loss function (features, features recovered from the input image) + cross-entropy loss (fake or real). In Equation 5, the total loss is L, and the image loss function is L il , the feature loss function is L fl , the input image I i , the output image is I o , the feature is f, and the feature recovered from the input image is f ii , cross-entropy loss (fake or real) is L ce In this mode, the input image may be perceptually identical to the output image, and even if some image manipulation (or fake) exists, the features recovered from the input image are as close as possible to the features extracted (recovered) from the input image.

[0103] The feature extraction neural network is co-trained with message coding neural networks (e.g., message encoding / decoding neural networks, message encoding neural networks and message decoding neural networks) and manipulation detection neural networks to extract features from images with enhanced accuracy, encode / decode messages from images, and / or determine whether a received image is a fake or manipulated version of an input image with enhanced accuracy.

[0104] The manipulations or fakes to be detected may be applied randomly or not applied at all. The total loss function to be optimized can be expressed by the following mathematical equation 6.

[0105]

[0106] Here, the total loss is equal to the image loss function (input image, output image) + feature loss function (random feature, recovered random feature) + identification feature loss function (feature, features of the received image) + cross-entropy loss (fake or real). In Equation 6, the total loss is L, and the image loss function is L il , the feature loss function is L fl , random characteristics are F ra , the recovered random feature is f rr , the identification feature loss function is L dfl , the characteristics of the received image are f ri , cross-entropy loss (fake or real) is L ce am.

[0107] If the received image is a fake image, the discriminative feature loss function penalizes case features, and the features of the received image are close. If the received image is real, the discriminative feature loss function penalizes case features, and the features of the received image are far. It is desirable for the input image to be perceptually identical to the output image. It is desirable for the message coding neural network to be robust against image manipulation or forgery. It is desirable to require features that are sensitive to image manipulation or forgery.

[0108] Although the described techniques and device features and concepts for verifying the authenticity of an image may be implemented in any number of different environments, systems, devices, and / or various configurations, the mode of verifying the authenticity of an image is described in the context of the following example devices, systems, and configurations.

[0109] Where the technology and / or device described herein collects or can use personal information about a user, the user may control whether the program or function collects personal information, e.g., information about the user (e.g., information about the user's social networks, social activities or activities, occupation, preferences, or current location), and may control whether and / or how the system and / or method can perform tasks more relevant to the user. Additionally, certain data may be anonymized in one or more ways before being stored or used to remove personally identifiable information. For example, the user's identity may be anonymized so that personally identifiable information about the user cannot be determined, or the user's geographic location may be generalized from where the location information is obtained, such as at the city, zip code, or state level, so that the user's specific location cannot be identified. Thus, the user may control how information about themselves is collected and used.

[0110] method

[0111] FIG. 6 illustrates an exemplary method (600) for generating an image to be verified. The method (600) may be performed by a system (100) as previously described in relation to FIG. 1, utilizing one or more of the components described in relation to FIG. 1. In 602, the encoder system receives an input image. In 604, the encoder system performs feature recognition on the input image to determine a plurality of determined features of the input image. In 606, the encoder system generates a third output defining a value representing the determined features of the input image. In 608, the encoder system provides the third output to a message encoding neural network. In 610, the message encoding neural network generates a signature from the third output defining a value representing the determined features of the input image. In 612, the encoder system embeds the second signature into the input image to generate an output image. In embodiments, the method (600) includes the step of verifying the authenticity of the image. The method (600) may include additional or fewer actions than those exemplified, or be performed in a different order.

[0112] In an exemplary use case of the method (600), a user of a user computing device utilizes the camera module of the user computing device to take an input image (photo). An encoder system implemented in the user computing device receives the input image. The encoder system performs feature recognition on the input image to determine a plurality of determined features of the input image. The encoder system generates an output that defines a value representing the determined features of the input image. The encoder system provides the output to a message encoding neural network implemented in the memory device of the user computing device. The message encoding neural network generates a signature from the output that defines a value representing the determined features of the input image. The encoder system embeds the signature into the input image to generate an output image that is stored in the memory device of the user computing device.

[0113] In another exemplary use case for the method (600), the image is stored in an image storage. An encoder system implemented in a server computing device receives an input image from the image storage. For example, the image storage may be utilized by the operator of the server computing device to store images uploaded by users of a service provided by the operator. The encoder system performs feature recognition on the input image to determine multiple determined features of the input image. The encoder system generates an output that defines values ​​representing the determined features of the input image. The encoder system provides the output to a message encoding neural network implemented in the memory device of the server computing device. The message encoding neural network generates a signature from the output that defines values ​​representing the determined features of the input image. The encoder system embeds the signature into the input image to generate an output image that is stored in the memory device of the server computing device. The embedded signature can later be used to verify (check) the image.

[0114] FIG. 7 illustrates an exemplary method (700) for verifying the authenticity of an image. The method (700) may be performed by a system (200) as previously described in relation to FIG. 2, utilizing one or more of the components described in relation to FIG. 2. In 702, the decoder system receives the image to be verified. In 704, the decoder system performs feature recognition on the received image to determine a plurality of determined features of the received image. In 706, the decoder system generates a first output defining a value representing the determined features of the received image. In 708, the decoder system decodes the received image using a message decoding neural network to extract a signature embedded in the received image. The embedded signature represents a recovered feature of the received image. In 710, the decoder system generates a second output defining a value representing the recovered feature of the received image. In 712, the decoder system provides the first output and the second output to a manipulation detection neural network. In 714, the manipulation detection neural network generates an estimate of the authenticity of the received image using at least the first output and the second output. In embodiments, the method (700) includes the step of verifying the authenticity of the image. The method (700) may include additional or fewer operations than illustrated, or may be performed in a different order.

[0115] In an exemplary use case for the method (700), a decoder system implemented in a user computing device receives an image to be verified from a memory device implemented in the user computing device. The decoder system performs feature recognition on the received image to determine a plurality of determined features of the received image. The decoder system generates a first output that defines a value representing the determined features of the received image. The decoder system utilizes a message decoding neural network implemented in the user computing device to decode the received image in order to extract a signature embedded in the received image. The embedded signature represents a recovered feature of the received image. The decoder system generates a second output that defines a value representing the recovered feature of the received image. The decoder system provides the first output and the second output to a manipulation detection neural network implemented in the user computing device. The manipulation detection neural network generates an estimation of the authenticity of the received image using at least the first output and the second output.

[0116] In another exemplary use case for the method (700), a decoder system implemented on a server computing device receives an image to be verified from an image repository. For example, the image repository may be utilized by an operator of the server computing device to store images uploaded by users of a service provided by an operator. The decoder system performs feature recognition on the received image to determine a plurality of determined features of the received image. The decoder system generates a first output that defines a value representing the determined features of the received image. The decoder system utilizes a message decoding neural network implemented on the server computing device to decode the received image in order to extract a signature embedded in the received image. The embedded signature represents a recovered feature of the received image. The decoder system generates a second output that defines a value representing the recovered feature of the received image. The decoder system provides the first output and the second output to a manipulation detection neural network implemented on the server computing device. The manipulation detection neural network generates an estimate of the authenticity of a received image using at least a first output and a second output.

[0117] machine learning models

[0118] FIG. 8 illustrates a block diagram of an exemplary machine learning model (800) according to exemplary embodiments of the present disclosure. As illustrated in FIG. 8, the machine learning model (800) is trained to receive one or more types of input data and, in response, provide one or more types of output data. Thus, FIG. 8 illustrates a machine learning model (800) that performs inference.

[0119] Input data may include one or more features associated with an instance or example. In some implementations, one or more features associated with an instance or example may be organized into a feature vector. In some implementations, output data may include one or more predictions. Predictions can also be referred to as inference. Therefore, given features associated with a specific instance, a machine learning model can output a prediction for that instance based on those features.

[0120] A machine learning model may be one or more of various types of machine learning models, or may include them. In particular, in some implementations, machine learning models can perform classification, regression, clustering, association, anomaly detection, recommendation generation, and / or other tasks. Machine learning models can perform various types of classification based on input data. For example, machine learning models can perform binary classification or multi-class classification. In binary classification, the output data may involve classifying the input data into one of two different classes. In multi-class classification, the output data may involve classifying the input data into one (or more) of two or more classes. Classification can be single-label or multi-label.

[0121] Machine learning models can perform discrete categorical classification where input data is simply classified into one or more classes or categories.

[0122] A machine learning model can perform classification by providing a numerical value for each of one or more classes that indicates the degree to which the machine learning model believes the input data should be classified into that class. In some cases, the numerical value provided by the machine learning model may be referred to as "confidence scores," representing the respective confidence associated with classifying the input into that class. In some implementations, confidence scores may be compared to one or more thresholds to render discrete categorical predictions. In some implementations, only a specific number of classes (e.g., one) with the relatively largest confidence scores may be selected to render discrete categorical predictions.

[0123] Machine learning models can provide probabilistic classification. For example, given a sample input, a machine learning model can predict a probability distribution for a set of classes. Thus, a machine learning model can output not only the most likely class to which the sample input belongs, but also the probability that the sample input belongs to that class for each class. In some implementations, the sum of the probability distributions for all possible classes is 1. In some implementations, a softmax function or layer can be used to squash the set of real values ​​associated with each possible class into a set of real values ​​within the range (0, 1) where the sum is 1.

[0124] The probabilities provided by the probability distribution can be compared to one or more thresholds to render a discrete categorical prediction. In some implementations, only a specific number of classes (e.g., one) with the relatively largest prediction probability may be selected to render the discrete categorical prediction.

[0125] Machine learning models can be trained using supervised learning techniques in implementations that perform classification. For example, a machine learning model can be trained on a training dataset containing training examples labeled as belonging to (or not belonging to) one or more classes. Further details regarding supervised training techniques are provided below.

[0126] Machine learning models can perform regression to provide output data in the form of continuous numeric values. These continuous numeric values ​​can correspond to various metrics or numeric representations, including, for example, currency values, scores, or other numeric representations. For example, machine learning models can perform linear regression, polynomial regression, or non-linear regression. For example, machine learning models can perform simple regression or multiple regression. As described above, in some implementations, a softmax function or layer can be used to squash a set of real values ​​associated with two or more possible classes into a set of real values ​​in the range (0, 1) where the sum is 1.

[0127] Machine learning models can perform various types of clustering. For example, a machine learning model can identify one or more previously defined clusters that best match the input data. As another example, a machine learning model can identify one or more clusters within the input data. That is, if the input data contains multiple objects, documents, or other entities, the machine learning model can sort the multiple entities contained in the input data into multiple clusters. In some implementations where machine learning models perform clustering, the machine learning model may be trained using unsupervised learning techniques.

[0128] Machine learning models can perform anomaly detection or outlier detection. For example, machine learning models can identify input data that does not follow expected patterns or other characteristics (e.g., as previously observed in previous input data). For instance, anomaly detection can be used for fraud detection or system failure detection.

[0129] Machine learning models can provide output data in the form of one or more recommendations. For example, machine learning models can be included in recommendation systems or engines. For instance, given input data describing previous results for a specific entity (e.g., a score, rank, or rating indicating the degree of success or enjoyment), the machine learning model can output suggestions or recommendations for one or more additional entities expected to yield desired results based on the previous results (e.g., deriving a score, rank, or rating indicating success or enjoyment). As an example, given input data describing multiple products purchased or highly rated by a user, the recommendation system can output suggestions or recommendations for additional products that the user might enjoy or wish to purchase.

[0130] Machine learning models can act as agents within an environment. For example, machine learning models can be trained using reinforcement learning, which is explained in more detail below.

[0131] Machine learning models can be parametric models, but in other implementations, machine learning models can be non-parametric models. In some implementations, machine learning models can be linear models, whereas in others, machine learning models can be non-linear models.

[0132] As mentioned above, a machine learning model may be one or more of various different types of machine learning models, or may include them. Examples of these various types of machine learning models are provided below for illustrative purposes. One or more of the exemplary models described below may be used (e.g., combined) to provide output data in response to input data. Additional models other than the example models provided below may also be used.

[0133] A machine learning model may be or include one or more classifier models, such as a linear classification model, a second-order classification model, etc.

[0134] A machine learning model may be or include one or more regression models, such as, for example, a simple linear regression model, a multiple linear regression model, a logistic regression model, a stepwise regression model, a multivariate adaptive regression spline; a locally estimated scatterplot smoothing model, etc.

[0135] The machine learning model may be or include one or more decision tree-based models, such as, for example, classification and / or regression trees, ID3 (Iterative Dichotomiser 3) decision trees, C4.5 decision trees, CS (chi-squared) automatic interaction detection decision trees, decision trees, conditional decision trees, etc.

[0136] A machine learning model may be or include one or more kernel machines. In some implementations, a machine learning model may be or include one or more support vector machines.

[0137] A machine learning model may be or include one or more instance-based learning models, such as, for example, a learning vector quantization model, a self-organizing map model, a local weighted learning model, etc.

[0138] The machine learning model may be one or more nearest neighbor models, such as, for example, a k-nearest neighbor classification model, a k-nearest neighbor regression model, or may include such models.

[0139] A machine learning model may be one or more Bayesian models, such as, for example, a Naive Bayes model, a Gaussian Naive Bayes model, a polynomial Naive Bayes model, an AODE (averaged one-dependence estimators); a Bayesian network; a BB (Bayesian belief) network; a HM (hidden Markov) model, or may include such models.

[0140] A machine learning model may be or contain one or more artificial neural networks (also simply called neural networks). A neural network may contain groups of connected nodes, also called neurons or perceptrons. A neural network may consist of one or more layers. A neural network containing multiple layers may be called a "deep" network. A deep network may include an input layer, an output layer, and one or more hidden layers located between the input and output layers. Nodes in a neural network may be connected or completely unconnected.

[0141] A machine learning model may be or contain one or more feedforward neural networks. In a feedforward network, connections between nodes do not form cycles. For example, each connection can link a node in a previous layer to a node in a later layer.

[0142] A machine learning model may be or contain one or more recurrent neural networks. In some cases, at least some of the nodes of a recurrent neural network may form a cycle. Recurrent neural networks can be particularly useful for processing inherently sequential input data. In particular, in some cases, a recurrent neural network can use recurrent or directed recurrent node connections to transfer or retain information from an earlier part of an input data sequence to a subsequent part of an input data sequence.

[0143] For example, sequential input data may include time-series data (e.g., sensor data versus time or images captured at different times). For example, a recurrent neural network can analyze sensor data over time to detect or predict the swipe direction, perform handwriting recognition, etc. As another example, sequential input data may include words in a sentence (e.g., for natural language processing, speech detection or processing, etc.), notes in a musical composition; sequential actions taken by a user (e.g., for detecting or predicting sequential application usage), sequential object states, etc.

[0144] Exemplary recurrent neural networks include long-term short-term (LSTM) recurrent neural networks, gated recurrent units, bidirectional recurrent neural networks, continuous-time recurrent neural networks, neural hysteresis compressors, echo-state networks, Elman networks, Jordan networks, recurrent neural networks, Hopfield networks; full recurrent networks, sequence-to-sequence constructions, etc.

[0145] A machine learning model may be or contain one or more convolutional neural networks. In some cases, a convolutional neural network may include one or more convolutional layers that perform convolution on input data using learned filters. Filters are also called kernels. Convolutional neural networks can be particularly useful for visual problems, such as when the input data includes images like still images or videos. However, convolutional neural networks can also be applied to natural language processing.

[0146] A machine learning model may be or include one or more generative networks (GN), such as a generative adversarial (GA) network, for example. GNs can be used to generate new data, such as new images or other content.

[0147] Machine learning models are or may include autoencoders. In some cases, the purpose of an autoencoder is typically to learn a representation of a dataset (e.g., a low-dimensional encoding) for dimensionality reduction. For example, in some cases, an autoencoder can encode input data and then provide output data that reconstructs the input data from the encoding. Recently, the concept of autoencoders has become more widely used to train generative models of data. In some cases, autoencoders may involve additional loss beyond reconstructing input data.

[0148] Machine learning models may be or include one or more different forms of artificial neural networks, such as deep Boltzmann machines, deep belief networks, stacked autoencoders, etc. Any neural network described herein may be combined (e.g., stacked) to form a more complex network.

[0149] One or more neural networks can be used to provide embeddings based on input data. For example, an embedding may be a representation of knowledge abstracted from input data into one or more learned dimensions. In some cases, embeddings can serve as a useful source for identifying relevant entities. In some instances, embeddings may be extracted from the network's output, while in others, they may be extracted from hidden nodes or layers of the network (e.g., layers that are near the end but not the final layer of the network). Embeddings can be useful for tasks such as automatic next-door video suggestion, product recommendations, and entity or object recognition. In some cases, embeddings serve as useful inputs for downstream models. For example, embeddings can be useful for generalizing input data (e.g., search queries) to downstream models or processing systems.

[0150] Machine learning models may include one or more clustering models, such as, for example, k-means clustering models, k-median clustering models, expectation-maximization models, and hierarchical clustering models.

[0151] Machine learning models can perform one or more dimensionality reduction techniques, such as principal component analysis, kernel principal component analysis, graph-based kernel principal component analysis; principal component regression; partial least squares regression, Samon mapping, multidimensional scaling, projection pursuit (PP), linear discriminant analysis, mixture discriminant (MD) analysis, second-order discriminant analysis, generalized discriminant analysis, flexible discriminant analysis, and auto-encoding.

[0152] Machine learning models may perform or apply one or more reinforcement learning techniques, such as, for example, Markov decision processes, dynamic programming, Q-function or Q-learning, value function approaches, deep Q-networks, differentiable neural computers, asynchronous advantage actor-critics (AAAC), deterministic policy gradients, etc.

[0153] Machine learning models can be autoregressive models. In some cases, an autoregressive model can be specified so that the output data depends linearly on its previous values ​​and stochastic terms. In other cases, an autoregressive model can take the form of a stochastic difference mathematical expression. An example of an autoregressive model is WaveNet, a generative model for raw audio.

[0154] Machine learning models can include or form parts of a multi-model ensemble. For example, bootstrap aggregation can be performed, which is also known as "bagging." In bootstrap aggregation, the training dataset is split into multiple subsets, and multiple models are trained on each subset. During inference, the outputs of each of the multiple models can be summed (e.g., through averaging, voting, or other techniques) and used as the output of the ensemble.

[0155] One exemplary model ensemble is the Random Forest, also known as the Random Decision Forest. The Random Forest is an ensemble learning method for classification, regression, and other tasks. A Random Forest is generated by generating multiple decision trees during training time. In some cases, the class mode (classification) or the average prediction of individual trees (regression) can be used as the output of the forest during inference time. Random Decision Forests can correct the tendency of decision trees to overfit the training set.

[0156] Another exemplary ensemble technique is stacking, which in some cases can be referred to as stacked generalization. Stacking involves training a combiner model to mix or combine predictions from several different machine learning models. Thus, multiple machine learning models (e.g., of the same or different types) can be trained based on training data. Additionally, the combiner model can be trained to use predictions from other machine learning models as input and generate a final inference or prediction in response. In some cases, a single-layer logistic regression model can be used as the combiner model.

[0157] Another example of an ensemble technique is boosting. Boosting can involve iteratively training weak models and adding them to the final strong model to progressively build an ensemble. For example, in some cases, each new model can be trained to highlight training examples that the previous model misinterpreted (e.g., misclassified). For instance, the weights associated with each such misinterpreted example can be increased. One common implementation of boosting is "AdaBoost," also known as adaptive boosting. Other exemplary boosting techniques include Linear Programming Boosting (LPBoost); TotalBoost; BrownBoost; XGBoost; MadaBoost; LogitBoost; and Gradient Boosting.

[0158] In addition, all the models described above (e.g., regression models and artificial neural networks) can be combined to form an ensemble. For example, an ensemble may include a heuristic function that combines and / or weights the outputs of higher-level machine learning models or the models forming the ensemble.

[0159] Multiple machine learning models (e.g., forming an ensemble) can be linked and trained jointly (e.g., sequentially through backpropagation via the model ensemble). However, in some implementations, only a subset (e.g., one) of the jointly trained models is used for inference.

[0160] Machine learning models can perform dimensionality reduction techniques and embeddings (e.g., matrix decomposition, principal component analysis, singular value decomposition, Word2vec / GloVe and / or related approaches) (clustering; classification and regression for downstream consumption). Many of these techniques have been described above and will be further explained below.

[0161] Referring again to FIG. 8, and as described above, a machine learning model can be trained or configured to receive input data and provide output data in response. The input data may include various types, formats, or variations of the input data. For example, in various implementations, the input data may include determined image features and / or image features provided by the user.

[0162] Machine learning models can receive and use input data in its raw form. In some implementations, raw input data may be preprocessed. Therefore, either by adding to the raw input data or, alternatively, machine learning models can receive and use preprocessed input data.

[0163] Preprocessing input data may include extracting one or more additional features from raw input data. For example, one or more new additional features may be generated by applying feature extraction techniques to the input data. Examples of feature extraction techniques may include edge detection; corner detection; blob detection; ridge detection; scale-invariant feature transformation; motion detection; optical flow; Hough transform; etc.

[0164] Extracted features may involve transforming the input data into a different domain and / or dimension, or may be derived. For example, the extracted features may involve transforming the input data into the frequency domain or be derived from such transformation. For instance, additional features may be generated by performing a wavelet transform and / or a Fast Fourier transform on the input data.

[0165] The extracted features may include statistics calculated from the input data or specific parts or dimensions of the input data. Exemplary statistics include modes, averages, maximums, minimums, or other metrics of the input data or parts thereof.

[0166] As explained above, input data can be inherently sequential. In some examples, sequential input data can be generated by sampling a stream of input data or otherwise splitting it. As an example, frames can be extracted from video. In some implementations, sequential data can be made non-sequential through summarization.

[0167] As another exemplary preprocessing technique, a portion of the input data can be imputed. For example, additional synthetic input data can be generated through interpolation and / or extrapolation.

[0168] As other exemplary preprocessing techniques, some or all of the input data may be scaled, standardized, normalized, generalized, and / or normalized. Examples of normalization techniques include Ridge regression; LASSO (least absolute shrinkage and selection operator); EN (elastic net); least angle regression; cross-validation; L1 normalization; L2 normalization, etc. As an example, some or all of the input data may be normalized by subtracting the mean for a given dimension from each individual feature value and then dividing by the standard deviation or another metric.

[0169] As another exemplary preprocessing technique, part or all of the input data may be quantized or discretized. As yet another example, qualitative features or variables contained in the input data may be converted into quantitative features or variables. For example, one-hot encoding may be performed.

[0170] Dimensionality reduction techniques can be applied to input data before being fed into a machine learning model. Several examples of dimensionality reduction techniques are provided, including, for instance, principal component analysis, kernel principal component analysis; graph-based kernel principal component analysis; principal component regression; partial least squares regression; Samon mapping; multidimensional scaling; projection pursuit (PP); linear discriminant analysis; mixture discriminant (MD) analysis; quadratic discriminant analysis; generalized discriminant analysis; flexible discriminant analysis; auto-encoding, etc.

[0171] During training, input data may be intentionally modified in various ways to improve model robustness, generalization, or other qualities. Examples of techniques for modifying input data include adding noise; changing color, shade, or hue; magnification; segmentation; and amplification.

[0172] Referring again to FIG. 8, the machine learning model (800) may provide output data upon receiving input data. The output data may include various types, formats, or variations of the output data. For example, in various implementations, the output data may include values ​​representing features of an image (e.g., input image, output image, received image), values ​​representing image features embedded in the received image, and / or predictions (e.g., estimates) regarding the authenticity of the image.

[0173] As described above, the output data may include various types of classification data (e.g., binary classification, multi-class classification, single-label, multi-label, discrete classification, regression classification, stochastic classification, etc.) or various types of regression data (e.g., linear regression, polynomial regression, non-linear regression, simple regression, multiple regression, etc.). In other examples, the output data may include clustering data, anomaly detection data, recommendation data, or any other form of output data described above.

[0174] Output data can influence downstream processes or decision-making. For example, in some implementations, output data can be interpreted and / or acted upon by a rules-based regulator.

[0175] In an embodiment, when a machine learning model is stored in a computing system (e.g., a user computing system (302)), software encryption rules (e.g., a secure hashing algorithm) may be utilized to protect the integrity of the model and prevent a third party from tampering with the model (e.g., replacing a part of the machine learning model with another model). In embodiments, an alarm signal may be generated in response to the detection of an attempted tampering.

[0176] Accordingly, the present disclosure provides a system and method comprising or otherwise utilizing one or more machine learning models to generate a prediction (e.g., an estimate) regarding the authenticity of an image based on determined features and / or recovered image features embedded in the image. Input data of any other type or format described above may be combined with any other type or format of machine learning model described above to provide output data of any other type or format described above.

[0177] A machine learning model (800) may be stored in a computing system (e.g., the computing system (300) of FIG. 3) and / or implemented locally by the computing system. For example, the machine learning model (800) may be stored in a user computing device or an embedded computing device and / or implemented locally. Output data obtained through the local implementation of the machine learning model in the computing system may be used to improve the performance of the computing system (e.g., an application implemented by the computing system). As one example, FIG. 3 illustrates a block diagram of a user computing system (302) (e.g., a mobile computing device) that locally stores and implements a machine learning model (320).

[0178] A machine learning model may be stored in a server computing device and / or implemented by a server computing device (e.g., the server computing system (330) of FIG. 3). In some cases, output data obtained through the implementation of the machine learning model in the server computing device may be used to improve other server tasks or to improve services performed by other non-user devices. For example, the output data may improve other downstream processes performed by the server computing device for a user computing device or an embedded computing device. In other examples, output data obtained through the implementation of the machine learning model in the server computing device may be transmitted to a user computing device, an embedded computing device, or some other client device and used by it. For example, it may be said that the server computing device performs machine learning as a service. As one example, FIG. 3 illustrates a block diagram of an exemplary user computing system (302) that can communicate with an exemplary server computing system (330) containing a machine learning model (340) via a network (380).

[0179] Different parts of a machine learning model may be stored and / or implemented in some combination of user computing devices, embedded computing devices, server computing devices, etc.

[0180] A computing device can perform graph processing techniques or other machine learning techniques using one or more machine learning platforms, frameworks, and / or libraries, such as TensorFlow, Caffe / Caffe2, Theano, Torch / PyTorch, MXNet, Cognitive Toolkit (CNTK), etc.

[0181] Computing devices are distributed across different physical locations and can be connected via one or more networks. Distributed computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. In one example, distributed computing devices can be controlled or guided through the use of a parameter server.

[0182] Processing throughput can be increased by parallelizing multiple instances of a machine learning model. For example, multiple instances of a machine learning model can be parallelized on a single processing unit or computing unit, or across multiple processing units or computing units.

[0183] The machine learning model described herein may be trained in a training computing system and then provided for storage and / or implementation in one or more computing devices as described above. For example, the model trainer (360) may be located in the training computing system (350) as illustrated in FIG. 3. The training computing system (350) may be included in or separated from one or more computing devices that implement the machine learning model. For example, FIG. 3 illustrates a block diagram of an exemplary user computing system (302) communicating with an exemplary training computing system (350) that includes the model trainer (360).

[0184] Machine learning models can be trained offline or online. In offline training (also known as batch learning), the model is trained on the entire static training dataset. In online training, the model is continuously trained (or retrained) as new training data becomes available (e.g., while the model is being used to perform inference).

[0185] Model trainers can perform centralized training of machine learning models (e.g., based on a centrally stored dataset). In other implementations, machine learning models can be trained, updated, or personalized using distributed training techniques such as distributed training and federated learning.

[0186] The machine learning models described herein may be trained according to one or more of various different training types or techniques. For example, in some implementations, machine learning models may be trained using supervised learning, where the machine learning models are trained on a training dataset containing labeled instances or examples. Labels may be applied manually by experts, generated through crowdsourcing, or provided through other techniques (e.g., physics-based or complex mathematical models). In some implementations, if the user has provided consent, training examples may be provided by the user's computing device. In some implementations, this process may be referred to as model personalization.

[0187] Machine learning models can learn by optimizing an objective function. For example, in some implementations, the objective function may be or include a loss function that compares (e.g., determines the difference) the output data generated by the model from the training data and the labels associated with the training data (e.g., ground-truth (GT) labels). For example, the loss function may evaluate the sum or average of the squared differences between the output data and the labels. As another example, the objective function may be or include a cost function that describes the cost of a specific result or output data. Other objective functions may include margin-based techniques, such as triplet loss or maximum margin training.

[0188] One or more of various optimization techniques may be performed to optimize the objective function. For example, optimization techniques can minimize or maximize the objective function. Exemplary optimization techniques include Hessian-based and gradient-based techniques, such as coordinate descent, gradient descent (e.g., stochastic gradient descent), and subgradient methods. Other optimization techniques include black-box optimization techniques and heuristics.

[0189] Backpropagation of errors can be used in conjunction with optimization techniques (e.g., gradient-based techniques) to train models (e.g., multilayer models such as artificial neural networks). For instance, a model can be trained by performing iterative cycles of propagation and model parameter (e.g., weight) updates. Examples of backpropagation techniques include truncated backpropagation and Levenberg-Marquardt backpropagation.

[0190] The machine learning models described herein can be trained using unsupervised learning techniques. Unsupervised learning may involve inferring functions that describe hidden structures in unlabeled data. For example, classification or categorization may not be included in the data. Using unsupervised learning techniques, machine learning models capable of performing clustering, anomaly detection, learning latent variable models, or other tasks can be generated.

[0191] The machine learning model described here can be trained using semi-supervised techniques that combine aspects of supervised and unsupervised learning.

[0192] The machine learning models described here can be trained or generated through evolutionary techniques or genetic algorithms.

[0193] The machine learning models described here can be trained using reinforcement learning. In reinforcement learning, an agent (e.g., a model) can learn how to take actions in the environment to maximize rewards or minimize penalties resulting from such actions. Reinforcement learning may differ from supervised learning problems in that correct input / output pairs are not provided and suboptimal actions are not explicitly corrected.

[0194] One or more generalization techniques can be performed during training to improve the generalization of machine learning models. Generalization techniques can help reduce overfitting of machine learning models to training data. Examples of generalization techniques include dropout, weight decay, batch normalization, early stopping, subset selection, and stepwise selection.

[0195] The machine learning models described herein may include or be influenced by various hyperparameters, such as the learning rate, the number of layers, the number of nodes in each layer, the number of leaves in the tree, and the number of clusters. Hyperparameters can affect model performance. Hyperparameters can be selected manually or automatically through the application of techniques, such as grid search; black-box optimization techniques (e.g., Bayesian optimization, random search, etc.); and gradient-based optimization. Exemplary techniques and / or tools for performing automatic hyperparameter optimization include Hyperopt, Auto-WEKA, Spearmint; and Metric Optimization Engines (MOE).

[0196] Various techniques can be used to optimize and / or adjust the learning rate when a model is trained. Exemplary techniques and / or tools for performing learning rate optimization or adaptation include AdaGrad, ADAM (Adaptive Moment Estimation), ADADELTA, and RMSprop.

[0197] Transfer learning techniques may be used to provide an initial model to initiate training of the machine learning model described herein. The machine learning model described herein may be included in other parts of the computer-readable code of a computing device. In one example, the machine learning model may be included in a specific application or program and may be used (e.g., exclusively) by such specific application or program. Thus, in one example, the computing device may include multiple applications, and one or more of such applications may include their own respective machine learning libraries and machine learning model(s).

[0198] The machine learning models described herein may be embedded in the operating system of a computing device (e.g., at the central intelligence layer of the operating system) and may be called or otherwise used by one or more applications interacting with the operating system. In some implementations, each application may communicate with the central intelligence layer (and the model(s) stored therein) using an Application Programming Interface (API) (e.g., a public API common to all applications).

[0199] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized data store for the computing device. The central device data layer can communicate with various other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using APIs (e.g., private APIs).

[0200] The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information transmitted from these systems. The inherent flexibility of computer-based systems allows for a wide variety of configurations, combinations, and divisions of tasks and functions among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0201] Furthermore, the machine learning techniques described herein are easily interchangeable and combinable. While specific exemplary techniques have been described, many other techniques exist and can be used in conjunction with the embodiments of this disclosure.

[0202] Accordingly, although the gist of this document has been described in detail with respect to various specific exemplary implementations, each example is provided for illustrative purposes rather than as a limitation of this disclosure. Those skilled in the art may readily make changes, variations, and equivalents to such implementations. Accordingly, this disclosure does not exclude such modifications, variations, and / or additional inclusions to the subject matter as would be readily apparent to those skilled in the art. For example, a function exemplified or described as part of one implementation may be used with another implementation to create another implementation.

[0203] Examples

[0204] The next section explains a few examples.

[0205] Example 1: The method comprises the steps of: receiving an image (210) to be verified by a decoder system (220); performing feature recognition on the received image (210) to determine a plurality of determined features (238) of the received image (210); generating a first output (236) that defines a value representing the determined features (238) of the received image (210); decoding the received image (210) by a message decoding neural network (252) of the decoder system (220) to extract a signature (254) embedded in the received image (210) - the signature (254) represents a recovered feature (258) of the received image (210); and generating a second output (256) that defines a value representing the recovered feature (258) of the received image (210). The method includes the step of providing the first output (236) and the second output (256) to the manipulation detection neural network (272) of the decoder system (220); and the step of generating an estimate of the authenticity of the received image (210) using at least the first output (236) and the second output (256) by the manipulation detection neural network (272).

[0206] Example 2: In the method of Example 1, the step of performing feature recognition on the received image further includes: a step of selecting a plurality of features of the received image by processing the received image using a first feature extraction neural network; or a step of receiving a first user input for selecting a plurality of features of the received image.

[0207] Example 3: In the method of Example 1 or Example 2, the method further comprises the steps of: receiving an input image by an encoder system; performing feature recognition on the input image to determine a plurality of determined features of the input image; and generating a third output that defines a value representing the determined features of the input image.

[0208] Example 4: In the method of Example 3, the step of performing feature recognition on a received image includes the step of processing the received image using a first feature extraction neural network to select a plurality of features of the received image; the step of performing feature recognition on an input image includes the step of processing the input image using a second feature extraction neural network to select a plurality of features of the input image; the method further includes the step of providing a second output and a third output to a discrimination feature loss calculation engine; the step of calculating a first loss based on a first loss function based on at least the second output and the third output by the discrimination feature loss calculation engine; the step of calculating a second loss based on a second loss function based on at least the input image and the received image; the step of calculating a total loss based on a total loss function based on at least the first loss and the second loss; and the step of co-training a message decoding neural network with at least one of the first feature extraction neural network or the second feature extraction neural network based on at least the calculated total loss.

[0209] Example 5: In the method of Example 3 or Example 4, the method further comprises the steps of: providing a third output to a message encoding neural network; generating a second signature from the third output; and embedding the second signature into an input image to generate an output image.

[0210] Example 6: In the method of Example 3, the step of performing feature recognition on the input image further includes at least one of the steps of: processing the input image using a second feature extraction neural network to select a plurality of features of the input image; or receiving a second user input for selecting a plurality of features of the input image.

[0211] Example 7: In any one of Examples 4-6, the method further comprises the step of performing a feature extraction neural network training procedure, wherein the step of performing the feature extraction neural network training procedure further comprises the step of calculating a third loss based on a third loss function based on at least a second output and a third output; and the step of training at least one of a first feature extraction neural network or a second feature extraction neural network based on at least the calculated third loss.

[0212] Example 8: In any one of the methods of Examples 3-7, the step of receiving the input image comprises at least one of the step of receiving the input image from an image repository; or the step of receiving the input image in response to a user selection; and the step of receiving the received image to be verified comprises at least one of the step of receiving the image received from an image repository; or the step of receiving the image received in response to a user selection.

[0213] Example 9: In any one of the methods of Example 3, Example 5, or Example 8, the step of performing feature recognition on a received image includes the step of processing the received image using a first feature extraction neural network to select a plurality of features of the received image; the step of performing feature recognition on an input image includes the step of processing the input image using a second feature extraction neural network to select a plurality of features of the input image; the method further includes the step of fixing the value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; the step of generating a training image including at least one image manipulation; the step of transmitting the training image to the manipulation detection neural network; the step of determining the image manipulation applied to the training image; the step of calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and the step of training the manipulation detection neural network based on at least the calculated normal loss.

[0214] Example 10: In any one of the methods of Example 3, Example 5, or Example 8, the step of performing feature recognition on a received image includes the step of processing the received image using a first feature extraction neural network to select a plurality of features of the received image; the step of performing feature recognition on an input image includes the step of processing the input image using a second feature extraction neural network to select a plurality of features of the input image; the method further includes the step of calculating an image loss based on an image loss function based on the input image; the step of calculating a feature loss based on a feature loss function based on the input image; the step of calculating a total loss based on the image loss and the feature loss; and the step of training at least one of the message encoding neural network or the message decoding neural network based on at least the calculated total loss.

[0215] Example 11: In any one of the methods of Examples 3 to 8 or Example 10, the method further comprises the step of applying at least one image manipulation to the input image.

[0216] Example 12: In the method of Example 10, the method further comprises the steps of: fixing the value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image including at least one image manipulation; transmitting the training image to a manipulation detection neural network; determining the image manipulation applied to the training image; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and training the manipulation detection neural network based on at least the calculated normal loss.

[0217] Example 13: In the method of Example 5 or Example 8, the method comprises the step of performing a feature extraction neural network training procedure—the step of performing the feature extraction neural network training procedure comprises: calculating a third loss based on a third loss function based on at least a second output and a third output; and training at least one of the first feature extraction neural network or the second feature extraction neural network based on at least the calculated third loss—; and the step of performing a manipulation detection neural network training procedure—the step of performing the manipulation detection neural network training procedure comprises: fixing the value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image including at least one image manipulation; transmitting the training image to the manipulation detection neural network; determining the image manipulation applied to the training image; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and training the manipulation detection neural network based on at least the calculated normal loss—; And it further includes the step of jointly training the feature extraction neural network and the manipulation detection neural network.

[0218] Example 14: In the method of Example 3 or Example 8, the step of performing feature recognition on a received image includes the step of processing the received image using a first feature extraction neural network; the step of performing feature recognition on an input image includes the step of processing the input image using a second feature extraction neural network; the method comprises the step of providing a third output to a message encoding neural network; the step of generating a second signature from the third output; the step of embedding the second signature into the input image to generate an output image; the step of providing the second output and the third output to an identification feature loss calculation engine; and the step of performing a message coding neural network training procedure—the step of performing the message coding neural network training procedure comprises the step of calculating a first loss based on a first loss function based on at least the second output and the third output by the identification feature loss calculation engine; the step of calculating a second loss based on a second loss function based on at least the input image and the received image; and the step of calculating a total loss based on a total loss function based on at least the first loss and the second loss. The method comprises the step of training at least one of a first feature extraction neural network, a second feature extraction neural network, or a message decoding neural network based on at least the total loss calculated above; the step of performing a manipulation detection neural network training procedure; the step of performing a manipulation detection neural network training procedure comprises: fixing the value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image including at least one image manipulation; transmitting the training image to the manipulation detection neural network; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and training the manipulation detection neural network based on at least the calculated normal loss.And it further includes a step of co-training the message coding neural network and the manipulation detection neural network.

[0219] Example 15: In the method of Example 3 or Example 8, the step of performing feature recognition on a received image includes the step of processing the received image using a first feature extraction neural network; the step of performing feature recognition on an input image includes the step of processing the input image using a second feature extraction neural network; the method comprises the step of providing a third output to a message encoding neural network; the step of generating a second signature from the third output; the step of embedding the second signature into an input image to generate an output image; the step of providing the second output and the third output to an identification feature loss calculation engine; the step of performing a message coding neural network training procedure—the step of performing a message coding neural network training procedure comprises the step of calculating a first loss based on a first loss function based on at least the second output and the third output by the identification feature loss calculation engine; the step of calculating a second loss based on a second loss function based on at least the input image and the received image; and the step of calculating a total loss based on a total loss function based on at least the first loss and the second loss. and includes the step of training at least one of a first feature extraction neural network, a second feature extraction neural network, or a message decoding neural network based on at least the calculated total loss-; the step of performing a feature extraction neural network training procedure - the step of performing a feature extraction neural network training procedure includes the step of calculating a third loss based on a third loss function based on at least the second output and the third output; and the step of training at least one of a first feature extraction neural network or a second feature extraction neural network based on at least the calculated third loss-; and the step of performing a manipulation detection neural network training procedure - the step of performing a manipulation detection neural network training procedure includes the step of fixing the value of at least one of a first feature extraction neural network, a second feature extraction neural network, a message encoding neural network, or a message decoding neural network;The method comprises: generating a training image including at least one image manipulation; transmitting the training image to a manipulation detection neural network; determining an image manipulation applied to the training image; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and training the manipulation detection neural network based on at least the calculated normal loss; and further comprises the step of co-training a feature extraction neural network, a message coding neural network, and a manipulation detection neural network.

[0220] Example 16: In any one of the methods of Examples 4-15, the first feature extraction neural network and the second feature extraction neural network are the same feature extraction neural network.

[0221] Example 17: In any one of Examples 9, 12, or 13-15, the normal loss is the cross-entropy loss.

[0222] Example 18: A computing device comprises a processor; and a computer-readable storage medium storing instructions that cause the processor to execute any one of Examples 1 to 17 in response to execution by the processor.

[0223] Example 19: The system includes one or more computers and one or more storage devices that store instructions that, when executed by one or more computers, cause one or more computers to perform an operation including any one of the methods of Examples 1 through 17.

[0224] Example 20: One or more non-transient computer storage media store instructions that, when executed by one or more computers, cause one or more computers to perform an operation including any one of the methods of Examples 1 through 19.

[0225] conclusion

[0226] Although the technology and apparatus for verifying the authenticity of an image have been described in language specific to the features and / or methods, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as exemplary implementations of the technology and apparatus for verifying the authenticity of an image.

Claims

Claim 1 As a method, the method comprises: receiving a first image (210) to be verified by a decoder system (220); performing feature recognition on the received first image (210) to determine a plurality of determined features (238) of the received first image (210) using a feature extraction neural network; generating a first output (236) that defines a value representing the determined features (238) of the received first image (210); decoding the received first image (210) by a message decoding neural network (252) of the decoder system (220) to extract a signature (254) embedded in the received first image (210) - the signature (254) represents a recovered feature (258) of the received first image (210); and a value representing the recovered feature (258) of the received first image (210). A step of generating a second output (256) that defines; a step of providing the first output (236) and the second output (256) to a manipulation detection neural network (272) of a decoder system (220); a step of generating an estimate of the authenticity of a received first image (210) using at least the first output (236) and the second output (256) by the manipulation detection neural network (272); a step of receiving an input image by an encoder system; a step of performing feature recognition on the input image to determine a plurality of determined features of the input image; a step of generating a third output that defines a value representing the determined features of the input image; A method comprising the step of jointly training the feature extraction neural network and the manipulation detection neural network, wherein the feature extraction neural network is trained based on a third loss calculated based on the second output and the third output, and the manipulation detection neural network is trained based on a normal loss calculated based on image manipulation determined according to the result of processing a training image using the manipulation detection neural network. Claim 2 A method according to claim 1, wherein the step of performing feature recognition on the received first image further comprises: a step of selecting a plurality of features of the received first image by processing the received first image using a first feature extraction neural network; or a step of receiving a first user input for selecting a plurality of features of the received first image. Claim 3 delete Claim 4 The method of claim 1 further comprises the step of performing feature recognition on a received first image, the step of processing the received first image using a first feature extraction neural network to select a plurality of features of the received first image; the step of performing feature recognition on an input image, the step of processing the input image using a second feature extraction neural network to select a plurality of features of the input image; and the method further comprises the step of providing a second output and a third output to a discrimination feature loss calculation engine; the step of calculating a first loss based on a first loss function based on at least the second output and the third output by the discrimination feature loss calculation engine; the step of calculating a second loss based on a second loss function based on at least the input image and the received first image; the step of calculating a total loss based on a total loss function based on at least the first loss and the second loss; and the step of co-training a message decoding neural network with at least one of the first feature extraction neural network or the second feature extraction neural network based on at least the calculated total loss. Claim 5 The method according to claim 1 further comprises the steps of: providing a third output to a message encoding neural network; generating a second signature from the third output; and embedding the second signature into an input image to generate an output image. Claim 6 A method according to claim 1, wherein the step of performing feature recognition on the input image further comprises at least one of the steps of: processing the input image using a second feature extraction neural network to select a plurality of features of the input image; or receiving a second user input for selecting a plurality of features of the input image. Claim 7 The method according to claim 4 further comprises the step of performing a feature extraction neural network training procedure, wherein the step of performing the feature extraction neural network training procedure further comprises the step of calculating a third loss based on a third loss function based on at least a second output and a third output; and the step of training at least one of a first feature extraction neural network or a second feature extraction neural network based on at least the calculated third loss. Claim 8 A method according to claim 1, wherein the step of receiving the input image comprises at least one of the step of receiving the input image from an image storage; or the step of receiving the input image in response to a user selection; and the step of receiving the first image to be verified comprises at least one of the step of receiving the first image received from an image storage; or the step of receiving the first image received in response to a user selection. Claim 9 In claim 1, the step of performing feature recognition on a received first image includes the step of processing the received first image using a first feature extraction neural network to select a plurality of features of the received first image; the step of performing feature recognition on an input image includes the step of processing the input image using a second feature extraction neural network to select a plurality of features of the input image; and the method further comprises the step of fixing the value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; the step of generating a training image including at least one image manipulation; the step of transmitting the training image to the manipulation detection neural network; the step of determining the image manipulation applied to the training image; the step of calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and the step of training the manipulation detection neural network based on at least the calculated normal loss. Claim 10 In claim 1, the step of performing feature recognition on a received first image includes the step of processing the received first image using a first feature extraction neural network to select a plurality of features of the received first image; the step of performing feature recognition on an input image includes the step of processing the input image using a second feature extraction neural network to select a plurality of features of the input image; the method further comprises the step of calculating an image loss based on an image loss function based on the input image; the step of calculating a feature loss based on a feature loss function based on the input image; the step of calculating a total loss based on the image loss and the feature loss; and the step of training at least one of the message encoding neural network of the encoder system or the message decoding neural network based on the calculated total loss. Claim 11 A method according to claim 1, wherein the method further comprises the step of applying at least one image manipulation to the input image. Claim 12 In claim 10, the method further comprises the steps of: fixing the value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image including at least one image manipulation; transmitting the training image to a manipulation detection neural network; and determining the image manipulation applied to the training image. Claim 13 delete Claim 14 In claim 1, the step of performing feature recognition on a received first image includes the step of processing the received first image using a first feature extraction neural network; the step of performing feature recognition on an input image includes the step of processing the input image using a second feature extraction neural network; the method comprises the steps of: providing a third output to a message encoding neural network; generating a second signature from the third output; embedding the second signature into the input image to generate an output image; providing the second output and the third output to an identification feature loss calculation engine; performing a message coding neural network training procedure; and performing a manipulation detection neural network training procedure. The step of performing a manipulation detection neural network training procedure comprises: fixing the value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image including at least one image manipulation; transmitting the training image to the manipulation detection neural network; and calculating a normal loss based on a normal loss function using the image manipulation applied to the training image. A method characterized by including at least the step of training the manipulation detection neural network based on the calculated normal loss; and further including the step of co-training the message coding neural network and the manipulation detection neural network. Claim 15 In claim 1, the step of performing feature recognition on a received first image includes the step of processing the received first image using a first feature extraction neural network; the step of performing feature recognition on an input image includes the step of processing the input image using a second feature extraction neural network; the method comprises the steps of: providing a third output to a message encoding neural network; generating a second signature from the third output; embedding the second signature into an input image to generate an output image; providing the second output and the third output to an identification feature loss calculation engine; and performing a message coding neural network training procedure. The step of performing a message coding neural network training procedure comprises: calculating a first loss based on a first loss function based on at least the second output and the third output by the identification feature loss calculation engine; calculating a second loss based on a second loss function based on at least the input image and the received first image; and calculating a total loss based on a total loss function based on at least the first loss and the second loss. And, comprising the step of training at least one of a first feature extraction neural network, a second feature extraction neural network, or a message decoding neural network based on at least a calculated total loss; a step of performing a feature extraction neural network training procedure - the step of performing a feature extraction neural network training procedure includes the step of calculating a third loss based on a third loss function based on at least a second output and a third output; and the step of training at least one of a first feature extraction neural network or a second feature extraction neural network based on at least a calculated third loss; and, a step of performing a manipulation detection neural network training procedure - the step of performing a manipulation detection neural network training procedure includes the step of fixing the value of at least one of a first feature extraction neural network, a second feature extraction neural network, a message encoding neural network, or a message decoding neural network; and the step of generating a training image including at least one image manipulation;A method comprising: transmitting a training image to a manipulation detection neural network; determining an image manipulation applied to the training image; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and at least training the manipulation detection neural network based on the calculated normal loss; and further comprising the step of jointly training a feature extraction neural network, a message coding neural network, and a manipulation detection neural network. Claim 16 A computing device comprising: a processor; and a computer-readable storage medium storing instructions that, in response to execution by said processor, cause said processor to execute the method of any one of claims 1, 2, 4 through 12, 14, and 15.

Images