Image Authenticity Verification Using Decoding Neural Networks

Abstract

This document describes techniques and apparatus for verifying the authenticity of an image. In various aspects, the method includes: receiving an image to be verified (210) by a decoder system (220); performing feature recognition on the received image to determine a determined feature (238) of the received image; generating a first output (236) defining a value representing the determined feature; decoding the received image by a message decoding neural network (252) to extract a signature (254) embedded in the received image, the embedded signature representing a recovered feature (258) of the received image; generating a second output (256) defining a value representing the recovered feature; providing the first and second outputs to a manipulation detection neural network (272); and generating, by the manipulation detection neural network, an estimate of the authenticity of the received image using at least the first and second outputs.

Description

Background Art

[0001] Using artificial neural networks and computational processes, images can be deceptively altered to include information that was not present when the image was originally created, thereby creating a manipulated or "fake" image. For example, deep learning forgeries ("deepfakes") can be created using artificial neural networks, where a video of a person making a first statement is modified to show that person making a second, different statement. In another example, a "face swap" application can be used to swap the face of a first person with the face of a second person on a still image.

[0002] Various methods have been developed to detect fake images. For example, algorithms can be used to detect subtle inconsistencies in images (e.g., lighting patterns) that indicate tampering. Despite these methods, reliable detection of fake images has proven to be a challenging and difficult problem. New frameworks are needed to verify the authenticity of images that address the limitations of existing techniques. Summary of the Invention

[0003] Techniques and apparatus for verifying the authenticity of images are described. Specifically, the techniques and apparatus may include or otherwise utilize one or more machine learning models to verify the authenticity of received images to detect manipulation of the images. Verification is based on comparing detected image features with recovered image features to determine whether the image is forged or manipulated. Comparison of the detected and recovered features can be used to identify features that indicate an inauthentic image that might not be detectable by a human observer.

[0004] In some embodiments, a feature extraction process is used to extract certain features from an input image, and a message embedding process is used to embed a signature including the certain features into the input image to generate an output image. The feature extraction process is used to extract the embedded signature including the features from the received image and recover the features from the signature. The authenticity of the image is then verified using an authenticity process, for example, by comparing the certain features with the recovered features.

[0005] Aspects described below include a method performed by a system including an encoder system and a decoder system. In the method, a system (e.g., a decoder system) receives an image to be verified. The system performs feature recognition on the received image to determine a plurality of determined features of the received image. The system generates a first output that defines a value representing the determined features of the received image. The system includes a message decoding neural network that decodes the received image to extract a signature embedded in the received image. The embedded signature represents recovered features of the received image. The system generates a second output that defines a value representing the recovered features. The system provides the first output and the second output to a manipulation detection neural network. The manipulation detection neural network generates an estimate of the authenticity of the received image using at least the first output and the second output.

[0006] The aspects described below also include a computing device having a processor and a computer-readable storage medium storing instructions that, in response to execution by the processor, cause the processor to perform a process for verifying the authenticity of an image. The aspects described below include methods for generating an image and methods for verifying the authenticity of an image. Optional features of one aspect, such as the above-described methods, may be combined with other aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The following figures describe techniques and apparatus for verifying image authenticity using machine learning models. The same numbers are used throughout the figures to refer to similar features and components:

[0008] Figure 1 An environment in which techniques and apparatus for verifying the authenticity of images using machine learning models are implemented is shown;

[0009] Figure 2 Another environment in which techniques and apparatus for verifying image authenticity using machine learning models can be implemented is shown;

[0010] Figure 3 depicts a block diagram of an example user computing system, an example server computing system, and an example training computing system according to some embodiments of the present disclosure;

[0011] Figure 4 An example system for training a feature extraction neural network to extract features from an image is shown;

[0012] Figure 5 An example system for training a message encoding / decoding neural network to encode / decode images is shown;

[0013] Figure 6 A method for generating an image to be verified is shown;

[0014] Figure 7 A method of verifying the authenticity of an image is shown; and

[0015] Figure 8 A block diagram of an example machine learning model according to an example embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0016] Overview

[0017] As described above, the present disclosure relates to techniques and apparatus for verifying the authenticity of an image. Specifically, the systems and methods of the present disclosure may include or utilize one or more machine learning models to verify the authenticity of an image.

[0018] As used herein, the phrase "image" includes, but is not limited to, a still image, a moving image (e.g., video), a moving image associated with audio, a moving image associated with text, and the like. As used herein, the phrase "video" refers to a series or time sequence of moving images that may or may not be associated with other types of information, such as audio or text. For example, a video may include a time sequence of moving images containing a stream of audio information. As used herein, the phrase "pseudo-image" refers to an image that is presented as an original image, but is a manipulated copy of the original image, or a pseudo-copy of the original image. As used herein, the phrase "image features" includes, but is not limited to, image-independent measurement matrices of the entire image, facial expressions, landmarks, facial landmarks of people in the image, landmarks, key points, user-provided features, edges, corners, blogs, ridges, motion, optical flow, raw pixels of the image, and the like.

[0019] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. For example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, bbc, cc, and cccc, or any other order of a, b, and c).

[0020] Figure 1 An example system 100 is shown that uses a feature recognition engine 120 (e.g., a feature extraction neural network 122) configured to extract image features from an input image 102 and a message encoder 140 (e.g., a message encoding neural network 142) configured to generate an output image 150 from the input image 102.

[0021] Figure 2An example system 200 is shown that uses a feature recognition engine 230 (e.g., a feature extraction neural network 232) configured to extract features 238 from a received image 210, a message decoder 250 configured to extract a signature 254 (hidden message) including features 258 from the received image 210, and / or an authenticity engine 270 (e.g., a manipulation detection neural network 272) configured to verify the authenticity of the received image 210 based on processing of the received image 210. In various aspects, the system may include a processor configured to generate an output image. Figure 1 The encoder system 110 is configured to decode the received image and determine the authenticity of the received image. Figure 2 One or both of the decoder systems 220. System 100, a component of system 100, system 200, or at least one of the components of system 200 may be implemented on one or more of a user computing device, an embedded computing device, a server computing device, a model processing device, a training device, a virtual computing device, other computing devices or computing infrastructure, or a combination thereof, such as described below with reference to Figure 3 Descriptive.

[0022] like Figure 1 As shown, in system 100, encoder system 110 receives input image 102 as input for processing. For example, a user may submit an image (e.g., a photograph) to system 100 as input image 102. In another example, a user may capture input image 102 on an imaging device (e.g., a digital camera, a camera of a computing device), and encoder system 110 is implemented in an image pipeline of the imaging device. In another example, input image 102 may be stored in image repository 104 and provided to encoder system 110. Image repository 104 may be included in system 100 (e.g., via one or more local wired or wireless connections), or may be remote from system 100 and in communication with system 100 via one or more wired or wireless connections (e.g., a local area network (LAN) or wide area network (WAN) connection). Image repository 104 may be, for example, a database stored locally on system 100, a database located at a server remote from system 100, or a memory device implemented on a computing device.

[0023] The system 100 (e.g., the encoder system 110) receives an input image 102, selects one or more features 128 of the input image 102, generates one or more signatures 144 based on the selected features 128, and embeds the signatures 144 into the input image 102 to create an output image 150. In some aspects, the output image 150 can be stored on a memory device.

[0024] The system 100 (e.g., the encoder system 110) includes a feature recognition engine 120. The feature recognition engine 120 is configured to perform a feature extraction process on an input image 102 to determine a plurality of determined features 128 of the input image 102. For example, the feature recognition engine 120 can be configured to select a plurality of features from the input image 102 and process the selected image features 128 to generate outputs 126 (e.g., data) defining values representing the determined features of the input image. In some aspects, the feature recognition engine 120 selects a plurality of features from the input image 102 and processes the selected image features 128 to generate outputs 126, each output 126 representing a selected feature 128 of the input image 102. For example, the features 128 of the input image 102 can include image-independent measurement matrices, facial expressions, facial landmarks of people in the image, landmarks, keypoints, user-provided features, edges, corners, corners, ridges, motion, optical flow, raw pixels of the image, and the like.

[0025] The feature recognition engine 120 may include a feature extraction neural network 122. The feature extraction neural network 122 may be configured to receive an input image 102 and process the input image 102 to generate an output 126 defining values representing selected features 128 of the input image 102. In some aspects, each of these values represents a selected feature 128 of the input image 102. For example, the feature extraction neural network 122 may receive an input image 102 and process the input image 102 or a portion thereof to generate an output 126 defining values representing one or more image features corresponding to facial expressions, landmarks, key points, edges, corners, corners, ridges, and the like. In some aspects, the output 126 defines values, each representing one or more image features corresponding to facial expressions, landmarks, key points, edges, corners, corners, ridges, and the like. In other embodiments, the feature recognition engine 120 may generate image features using other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transform, or other algorithms.

[0026] The feature recognition engine 120 may receive one or more user-provided features 124 provided by one or more user input components configured to receive input (selected image features) from a user. The user-provided features 124 may include facial expressions, facial landmarks of people in the image, raw pixels of the image, etc. The feature recognition engine 120 may include a feature extraction neural network 122 and the user-provided features 124.

[0027] The system 100 (e.g., encoder system 110) also performs a message embedding process on the input image 102. The system 100 sends an output 126 specifying image features 128 derived from the input image 102 to a message encoder 140. The message encoder 140 may include a message encoding neural network 142. The message encoding neural network 142 may be configured as a message encoding / decoding neural network (e.g., a message encoding / decoding neural network) for encoding and decoding messages. Figure 1 Message encoding neural network 142 and Figure 2 The message decoding neural network 252 is a message encoding / decoding neural network.

[0028] The message encoder 140 (e.g., a message encoding neural network 142) receives the output 126 specifying the image features 128 derived from the input image 102 from the system 100 and generates a signature 144 (e.g., including the output 126) representing the image features 128 of the input image 102. The message encoder 140 (e.g., the message encoding neural network 142) processes the input image 102 by embedding the signature 144 (the message) as a digital message (e.g., a steganographic signal) into the input image 102 to generate an output image 150. In some aspects, the signature 144 is a perceptually invisible watermark.

[0029] The weight can be Figure 1 The feature recognition engine 120 shown (e.g., first feature extraction neural network 122, user-provided features 124) and Figure 2 The second feature recognition engine 230 shown (e.g., the second feature extraction neural network 232, the user-provided features 234) is shared between them.

[0030] Figure 2 System 200 is shown including a decoder system 220 configured to decode a received image 210 and verify the authenticity of received image 210. Received image 210 is the image to be verified. In some aspects, received image 210 is a copy of output image 150. In some aspects, received image 210 is a pseudo-copy of input image 102 or output image 150. In some aspects, received image 210 is a manipulated copy of input image 102 or output image 150. Decoder system 220 determines the authenticity of received image 210, e.g., whether received image 210 is output image 150 or whether received image 210 is a pseudo-version of input image 102.

[0031] System 100 can send output image 150 generated by encoder system 110 to decoder system 220 as received image 210. In some embodiments, received image 210 is provided to decoder system 220 of system 200; for example, a user can submit a photo as received image 210 to system 200.

[0032] In another example, the received image 210 may be obtained from an image repository such as Figure 1 The image library 104) is provided to the decoder system 220.

[0033] In one example, a user verifies the authenticity of an image using system 100 and / or system 200. In another example, an online service provider verifies the authenticity of an image using system 100 and / or system 200. In another example, a third party verifies the authenticity of an image using system 100 and / or system 200.

[0034] The system 200 provides the received image 210 as input to the decoder system 220, and the decoder system 220 determines whether the received image 210 is a reliable copy (e.g., a true copy, an unaltered copy) or whether the received image 210 is an unreliable copy, e.g., a fake or altered copy of the output image 150 or the input image 102. Figure 2 As shown, system 200 (eg, decoder system 220 ) includes a feature recognition engine 230 , a message decoder 250 , and an authenticity engine 270 .

[0035] The system 200 may provide a received image 210 as input to a feature recognition engine 230. The feature recognition engine 230 is configured to perform a feature extraction process (feature recognition) on the received image 210 to determine a plurality of determined features 238 of the received image 210. For example, the determined features 238 of the received image 210 may include image-independent measurement matrices, facial expressions, landmarks, key points, user-provided features, edges, corners, corners, ridges, motion, optical flow, pixels, and the like.

[0036] The feature recognition engine 230 selects a plurality of determined features 238 from the received image 210 and processes the determined features 238 to generate an output 236 (data) defining values representing determined features of the received image 210. In various aspects, the feature recognition engine 230 selects a plurality of determined features 238 from the received image 210 and processes the determined features 238 to generate an output 236 (data) defining values representing determined features of the received image, each determined feature representing a selected feature of the received image 210. The system 200 sends the output 236 of the determined features 238 to the authenticity engine 270.

[0037] The feature recognition engine 230 may include a second feature extraction neural network 232. The feature extraction neural network 232 performs a feature extraction process on the received image 210. The feature extraction neural network 232 may be configured to receive the received image 210 and process the received image 210 to generate an output 236 (data), the output 236 defining a value representing a determined feature 238 of the received image 210. For example, the feature extraction neural network 232 may be configured to receive the received image 210 and process the received image 210 to generate an output 236, the output 236 defining a value each representing a determined feature 238 of the received image 210. In some embodiments, the feature extraction neural network 232 is associated with Figure 1 In some embodiments, the feature extraction neural network 232 is the same as the feature extraction neural network 122 of FIG. Figure 1 The feature extraction neural network 122 is a different feature extraction neural network.

[0038] The feature recognition engine 230 may receive input including one or more user input components (eg, Figure 3 The feature recognition engine 230 may include input from one or more user-provided features 234 provided by a user input component 322 of the image. The user-provided features 234 may include, for example, facial expressions, facial landmarks of people in the image, raw pixels of the image, etc. In some implementations, the feature recognition engine 230 may include a second feature extraction neural network 232 and the user-provided features 234.

[0039] The received image 210 may be provided as an input to a message decoder 250 of the decoder system 220. The message decoder 250 (e.g., a message decoding neural network 252) decodes the received image 210. For example, the message decoder 250 decodes the received image 210 by performing a message extraction process on the received image 210. During the message extraction process, the message decoder 250 processes the received image 210 to extract a signature 254 embedded in the received image 210. The signature 254 represents a signature generated by the message encoder (e.g., Figure 1 The message encoder 140 of FIG. 1 is used to embed the recovered features 258 in the received image 210. In some aspects, the system 200 detects whether the signature is present in the received image 210.

[0040] The message extraction process includes generating a second output 256 (data) defining values of recovered features 258 representing the received image 210. In various aspects, the message extraction process includes generating a second output 256 defining values of recovered features 258 each representing the received image 210 recovered from the signature 254. The system 200 sends the output 256 of recovered features 258 to the authenticity engine 270.

[0041] The output 236 (first output) from the feature recognition engine 230 and the output 256 (second output) from the message decoder 250 are provided to the authenticity engine 270 and used by the authenticity engine 270 to generate a prediction (e.g., an estimate) of the authenticity of the received image 210. For example, the authenticity engine 270 of the decoder system 220 determines whether the received image 210 is an authentic copy (e.g., an authentic copy, an unaltered copy) or whether the received image 210 is an unreliable copy, such as a fake image or an altered copy.

[0042] Authenticity engine 270 may include a manipulation detection neural network 272 that receives outputs (e.g., output 236, output 256). Manipulation detection neural network 272 uses at least output 236 and output 256 to generate an estimate (prediction) of the authenticity of received image 210, for example, by comparing output 236 and output 256. Based on the estimate generated by authenticity engine 270, system 100 determines whether received image 210 is an authentic copy (e.g., an authentic copy, an unaltered copy), or whether received image 210 is an unauthentic copy, for example, a fake image or an altered copy.

[0043] Example Embodiments

[0044] The systems and methods of the present disclosure may be implemented by or executed on one or more computing systems. Example computing systems in computing system 300 include one or more user computing devices (e.g., laptops, desktop computers, mobile computing devices such as tablets, smartphones, wearable computing devices, cameras, etc.); embedded computing devices (e.g., embedded in vehicles, cameras, image sensors, industrial machines, satellites, game consoles or controllers, or devices in home appliances such as refrigerators, thermostats, electric meters, home energy managers, smart home assistants, etc.); server computing devices (e.g., database servers, parameter servers, file servers, mail servers, print servers, network servers, game servers, application servers, etc.); dedicated, specialized model processing or training devices; virtual computing devices; other computing devices or computing infrastructure; or combinations thereof.

[0045] For example, Figure 3 A block diagram of an example computing system 300 that can verify image authenticity according to an example embodiment of the present disclosure is depicted. The computing system 300 includes one or more user computing systems 302, server computing systems 330, or training computing systems 350 communicatively coupled via a network 380.

[0046] The server computing system 330 may include one or more server computing devices, or may be implemented by one or more server computing devices. Where the server computing system 330 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof. In some embodiments, the training computing system 350 includes one or more server computing devices, or may be implemented by one or more server computing devices.

[0047] The user computing system 302 includes one or more processors 312 and one or more memory devices 314. The server computing system 330 includes one or more processors 332 and a memory device 334. The training computing system 350 includes one or more processors 352 and a memory device 354. The processors 312, 332, 352 can be any suitable processing device (e.g., a central processing unit (CPU); a visual processing unit (VPU); a graphics processing unit (GPU); a tensor processing unit (TPU); a neural processing unit (NPU); a neural processing engine; a core of a CPU, VPU, GPU, TPU, NPU, or other processing device; an application-specific integrated circuit (ASIC); a field-programmable gate array (FPGA); a coprocessor; a controller; or a combination of the foregoing), and can be a single processor or multiple processors operably connected. The processors 312, 332, 352 can be embedded in other hardware components, such as, for example, an image sensor, an accelerometer, etc.

[0048] Memory devices 314, 334, 354 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and the like, and combinations thereof. Memory device 314 may store data 316 and instructions 318, which are executed by processor 312 to cause user computing system 302 to perform operations. Memory device 334 may store data 336 and instructions 338, which are executed by processor 332 to cause server computing system 330 to perform operations. Memory device 354 may store data 356 and instructions 358, which are executed by processor 352 to cause training computing system 350 to perform operations.

[0049] One or more of the user computing system 302, the server computing system 330, or the training computing system 350 may include an encoder system (e.g., Figure 1 An example encoder system may include a feature recognition engine (e.g., Figure 1 feature recognition engine 120) and a message encoder (e.g., Figure 1In some implementations, one or more of the user computing system 302, the server computing system 330, or the training computing system 350 may include a decoder system (e.g., Figure 2 Decoder system 220). An example decoder system may include a feature recognition engine (e.g., Figure 2 Feature recognition engine 230), message decoder (e.g., Figure 2 message decoder 250) or an authenticity engine (e.g., Figure 2 at least one of the authenticity engines 270).

[0050] The user computing system 302 may store or include one or more machine learning models 320. For example, the machine learning model 320 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. More specifically, the machine learning model 320 may include one or more feature extraction networks (such as Figure 1 Feature extraction neural network 122 and / or Figure 2 A second feature extraction neural network 232), a message encoding neural network (such as Figure 1 message encoding neural network 142), message decoding neural network (such as Figure 2 message decoding neural network 252), or manipulation detection neural network (such as Figure 2 manipulation detection neural network 272).

[0051] Additionally or alternatively, one or more machine learning models 340 may be included in, or stored and implemented by, a server computing system 330 that communicates with the user computing system 302 according to a client-server relationship. For example, the machine learning models 340 may be implemented by the server computing system 340 as part of a web service (e.g., an extreme multi-class or multi-label classification service, a language modeling service, a metric learning service). Thus, one or more models 320 may be stored and implemented at the user computing system 302, and / or one or more models 340 may be stored and implemented at the server computing system 330. In embodiments, the machine learning models 340 may include one or more feature extraction networks (such as Figure 1 Feature extraction neural network 122 and / or Figure 2 feature extraction neural network 232), message encoding neural network (such as Figure 1 message encoding neural network 142), message decoding neural network (such as Figure 2message decoding neural network 252), or manipulation detection neural network (such as Figure 2 manipulation detection neural network 272).

[0052] The user computing system 302 may also include one or more user input components 322 for receiving user input. For example, the user input component 322 may be a touch-sensitive component (e.g., a touch-sensitive display screen, a touchpad) that is sensitive to the touch of a user input object (e.g., a finger, a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other components that a user may use to provide user input.

[0053] As described above, server computing system 330 may store or otherwise include one or more machine learning models 340. For example, model 340 may be or include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Examples of neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0054] Computing system 300 may include image repository 104. For example, image repository 104 may be included in system 300 via one or more local wired or wireless connections, or may be remote from system 300 and in communication with system 300 via one or more wired or wireless connections (e.g., a local area network (LAN) or wide area network (WAN) connection). For example, image repository 104 may be a database stored locally on user computing system 302, or a database located on a server remote from system 300 (e.g., server computing system 330, training computing system 350).

[0055] User computing system 302 and / or server computing system 330 may train models 320 and / or 340 via interaction with training computing system 350 communicatively coupled via network 380. Training computing system 350 may be separate from server computing system 330 or may be part of server computing system 330.

[0056] The training computing system 350 may include a model trainer 360 that trains the machine learning models 320 and / or 340 stored at the user computing system 302 and / or the server computing system 330 using various training or learning techniques, such as, for example, verifying the authenticity of an image based on comparing the determined features and the recovered features with the machine learning models 320 and / or 340 and backpropagation of errors. In some embodiments, performing backpropagation of errors may include performing truncated backpropagation over time. The model trainer 360 may perform a variety of generalization techniques (e.g., weight decay, dropout) to improve the generalization ability of the trained model.

[0057] Model trainer 360 can train machine learning models 320 and / or 340 based on a set of training data 362. Training data 362 can include, for example, examples of input data that have been assigned labels corresponding to output data.

[0058] If the user has provided consent, training examples may be provided by the user computing system 302. Thus, in such an implementation, the model 320 provided to the user computing system 302 may be trained by the training computing system 350 based on user-specific data received from the user computing system 302. In some cases, this process may be referred to as personalizing the model.

[0059] The model trainer 360 includes computer logic for providing the required functionality. The model trainer 360 can be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, the model trainer 360 includes a program file stored on a memory device, loaded into memory, and executed by one or more processors. In other embodiments, the model trainer 360 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as a RAM hard disk or optical or magnetic media.

[0060] Network 380 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications on network 380 can be carried over any type of wired and / or wireless connection using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0061] Figure 3 An example of a computing system 300 that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some embodiments, user computing system 302 may include model trainer 360 and training data 362. In such an implementation, model 320 may be trained and used locally on user computing system 302. In some such implementations, user computing system 302 may implement model trainer 360 to personalize model 320 based on user-specific data.

[0062] train

[0063] The framework described herein can be trained in a variety of ways, for example, depending on how the feature extraction neural network, the message encoding and decoding neural network (e.g., the message encoding neural network, the message decoding neural network, the message encoding / decoding neural network), and / or the manipulation detection neural network are trained. In some embodiments, one or more neural networks can be trained individually. In some embodiments, one or more neural networks are trained together. In other embodiments, all neural networks are trained together.

[0064] Figure 4 An example system 400 is depicted that is configured to train a first feature extraction neural network 422 (such as Figure 1 The first feature extraction neural network 122 and / or Figure 2 The second feature extraction neural network 232 of the present invention is used to extract features from an image (e.g., the input image 416) with improved accuracy. In some aspects, the feature extraction neural network 422 is used in a system configured to verify the authenticity of an image (e.g., Figure 1 System 100, Figure 2 system 200).

[0065] The feature extraction neural network 422 may be trained separately from one or more message encoding neural networks, message decoding neural networks, or manipulation detection neural networks. For example, at operation 412, an input image 416 is provided to an encoder system 410 including a feature recognition engine 420 for processing. For example, a user may submit an image (e.g., a photo) to the system 400 as the input image 416. In another example, the feature recognition engine 420 is a feature extraction neural network 422, and the image repository 104 (e.g., Figure 2 ) provides an input image 416 to a feature recognition engine 420. In embodiments, the feature recognition engine 462 may generate determined features 418 by using other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transform, or other algorithms.

[0066] The feature recognition engine 420 (eg, feature extraction neural network 422) is configured to estimate (determine) features of the input image 416 and generate determined features 418 (eg, Figure 1 In some aspects, the feature extraction neural network 422 estimates determined features 418 that have characteristics that can be identified as learned features by image recognition. The learned features should be sensitive to the detection of fake images; for example, if the image is manipulated or fake, the learned features will be very different.

[0067] Alternatively or additionally, determined features 418 can be extracted from the input image 416 using user input, such as user-provided features 424 (e.g., user-provided features 124 described herein). In embodiments, the second feature recognition engine 462 can generate recovered features 466 using other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transform, or other algorithms.

[0068] After estimating the determined features 418, the feature recognition engine 420 sends an output 430 of the determined features 418 to the discriminant feature loss calculation engine 470. The determined features 418 are provided to the discriminant feature loss calculation engine 470 to enable the discriminant feature loss calculation engine 470 to calculate a feature loss 472 for training the feature extraction neural network 422. This process may be repeated until the calculated loss is equal to or below a predetermined threshold, or until other convergence criteria are met.

[0069] The feature recognition engine 420 may also send an output 426 having the determined features 418 to a message encoder 440 (comprising a message encoding neural network 442 (e.g., Figure 1 4 and 5. The message encoding neural network 142 is used to generate the output image 450. In some aspects, one or more signatures are generated based on the determined features 428 and the signatures are embedded into the input image 416 as digital messages (e.g., steganographic signals) to generate the output image 450. In embodiments, the feature recognition engine 420 may generate the determined features 418 by using other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transform, or other algorithms.

[0070] By a feature recognition engine (e.g., by a first feature recognition engine 420 (e.g., a first feature extraction neural network 422 (e.g., Figure 1 122)) and / or by a second feature recognition engine 462 (e.g., a second feature extraction neural network 464 (such as Figure 2 The feature extraction neural network 232 performs a feature extraction process on an image (e.g., input image 416, received image 452). For example, Figure 4 As shown, the decoder system 460 includes a second feature recognition engine 462 (e.g., a feature extraction neural network 464). The second feature recognition engine 462 decodes the received image 452 to extract the signature from the received image 452 (e.g., as described above with reference to FIG. Figure 2 The signature represents the recovered features 466 of the received image 452 .

[0071] The feature recognition engine (e.g., the second feature recognition engine 462) sends an output 468 having recovered features 466 to a discriminative feature loss calculation engine 470. The recovered features 466 are provided to enable the discriminative feature loss calculation engine 470 to calculate a feature loss 472, which can be used to train the feature extraction neural network 422.

[0072] The system can perform a message encoding and decoding neural network training procedure, wherein a discriminative feature loss calculation engine 470 applies a discriminative feature loss function to the recovered features 466 and the determined features 418 to determine an image feature loss 472 caused by the estimation performed by the feature extraction neural network 422. The discriminative feature loss function applies the determined features 418 (such as Figure 1 The selected features 128 of the input image 102 and the restored features 466 of the received image 452 (such as Figure 2 If the test image (e.g., Figure 2 If the test image (e.g., received image 452) is genuine (e.g., not a pseudo copy of the output image 450, not a manipulated copy of the output image 450), then the discriminant feature loss function should be small. However, if the test image (e.g., received image 452) is not genuine (e.g., is a pseudo copy of the output image 450, is a manipulated copy of the output image 450), then the discriminant feature loss function should be large.

[0073] The discriminative feature loss calculation engine 470 sends the feature loss 472 as an output to the feature recognition engine 420 for use in training the feature extraction neural network 422. The loss function determines the deviation between the determined features 418 estimated by the feature recognition engine 420 and the recovered features 466 generated by the second feature recognition engine 462. In some embodiments, the loss function can express the deviation as a loss in accuracy of the recovered features 466 in the received image 452 due to the received image 452 being a pseudo image.

[0074] After the loss is calculated, the feature loss 472 is used to further optimize the feature extraction neural network 422. For example, the feature loss 472 can be used to optimize the parameters of the layers of the feature extraction neural network 422.

[0075] This training cycle can be repeated for multiple images from the image library 104 to iteratively optimize the feature extraction neural network 422. Parameter optimization enables the feature extraction neural network 422 to more accurately estimate the features depicted in the image. Over time, through sufficient training through iterations of the training cycle, the feature extraction neural network 422 can be improved to minimize the loss function so that the recovered features 466 provided by the second feature recognition engine 462 to the discriminative feature loss calculation engine 470 converge to the determined features 418 provided by the first feature recognition engine 420 to the discriminative feature loss calculation engine 470. This can be repeated. Figure 4 The process shown continues until the feature loss is equal to or below a predetermined threshold, or until other convergence criteria are met.

[0076] Figure 5 An example system 500 is depicted that is configured to train a message encoding / decoding neural network (e.g., a message encoding / decoding neural network). For example, a message encoding neural network (e.g., Figure 5 Message encoding neural network 542, Figure 1 The message encoding neural network 142 of the decoder system 560 and / or the message decoding neural network of the message decoder 561 of the decoder system 560 (e.g., Figure 5 Message decoding neural network 562, Figure 2 In some aspects, at least one of the message encoding neural network or the message decoding neural network is trained separately from the feature extraction neural network and / or the manipulation detection neural network. In some aspects, the message encoding and decoding neural network is trained together with one or more feature extraction neural networks or the manipulation detection neural network.

[0077] At operation 504, the input image 502 is provided to a feature recognition engine 520 for processing. Figure 4 As described above, a user may submit an image (e.g., a photo) to the system 500 as an input image 502. In embodiments, the input image 502 is modified to train at least one message encoding neural network 542 to be robust to this type of distortion. For example, the input image 502 may be modified (e.g., edited, resized, cropped) or may be a pseudo-image.

[0078] In another example, the feature recognition engine 520 includes a first feature extraction neural network 522 of the encoder system 510. The image library 104 (e.g., as described with respect to Figure 4 ) provides an input image 502 to a feature recognition engine 520, as described with respect to the image library 104 of FIG. Figure 4At operation 506 , the input image 502 is also provided to the image loss calculation engine 580 for processing.

[0079] The feature recognition engine 520 (e.g., feature extraction neural network 522) estimates (determines) features of the input image 502 and generates an output of determined features 528. In some aspects, the feature extraction neural network 522 estimates determined features 528 that have properties that can be identified as learned features by image recognition. The learned features should be sensitive to detection of fake images; for example, if the image is manipulated or fake, the learned features will be very different. In some aspects, user-provided features (such as Figure 1 User provided features 124 and / or Figure 2 User-provided features 234) can be provided by the user to include determined features 528.

[0080] Alternatively or additionally, the determined features 528 may be extracted from the input image 502 using user-provided features, as described above with reference to Figure 4 In embodiments, the feature recognition engine 520 may generate the determined features 528 by using other feature extraction techniques and algorithms that do not rely on neural networks, such as principal component analysis, edge detection, Hough transform, or other algorithms.

[0081] After estimating the determined features 528, the feature recognition engine 520 sends an output 530 (third output) of the determined features 528 to the feature loss calculation engine 570. The determined features 528 are provided to the feature loss calculation engine 570 so that the feature loss calculation engine 570 can calculate the feature loss 572, which is provided as input to the total loss calculation engine 590.

[0082] The feature recognition engine 520 also provides the message encoding neural network 542 (e.g., Figure 1 The message encoding neural network 142 of the embodiment of the present invention sends an output 526 having determined features 528 to generate an output image 550. In some aspects, one or more signatures 528 are generated based on the determined features, and the signatures are embedded in the output image 550 as a digital message (e.g., a steganographic signal) to generate the output image 550.

[0083] At operation 554, the received image 552 is sent to the image loss calculation engine 580 for processing. At operation 556, the received image 552 is sent to the decoder system 560. In some aspects, the decoder system 560 includes a message decoder 561. In some aspects, the message decoder 561 includes a message decoding neural network 562. The message decoding neural network 562 (such as Figure 2The message decoding neural network 252) decodes the received image 552 to extract a signature from the received image 552. The signature represents a recovered feature 564 of the received image 552. In an embodiment, the message decoding neural network 252 is a message encoding / decoding neural network configured to encode and decode images.

[0084] The message decoding neural network 562 is configured to send an output 563 (second output) including the recovered features 564 to the feature loss calculation engine 570. The recovered features 564 are provided to the feature loss calculation engine 570 so that the feature loss calculation engine 570 can calculate the feature loss 572 that is provided as input to the total loss calculation engine 590.

[0085] System 500 determines a feature loss 572. In embodiments, system 500 includes a feature loss calculation engine 570 that applies a loss function to determine feature loss 572. The loss function determines the deviation between determined features 528 estimated by feature recognition engine 520 relative to features in input image 502 and recovered features 564 extracted by message decoding neural network 562 relative to features in received image 552. In some embodiments, the loss function may represent the deviation as a loss in accuracy of recovered features 564 in received image 552 due to the fact that received image 552 is a pseudo-image. After the loss is calculated, feature loss 572 is sent as an output (a first loss) to a total loss calculation engine 590. Feature loss 572 is provided to total loss calculation engine 590 to further optimize one or more of feature extraction neural network 522, message encoding neural network 542, message decoding neural network 562, or a message encoding / decoding neural network (e.g., a message encoding / decoding neural network).

[0086] System 500 determines image loss 582. In embodiments, system 500 includes an image loss calculation engine 580 that receives input image 502 and received image 552 and applies a loss function to determine image loss 582. The loss function determines the deviation between input image 502 and received image 552. In some embodiments, the loss function may represent deviations in received image 552 that are the result of an artifact or manipulated image. After calculating image loss 582, image loss 582 is sent to a total loss calculation engine 590 and may be used to further optimize one or more of feature extraction neural network 522, message encoding neural network 542, or message decoding neural network 562.

[0087] The system 500 determines a total loss 592 based on the image loss 582 and the feature loss 572. In some aspects, the system 500 includes a total loss calculation engine 590 that applies a loss function to the feature loss 572 and the image loss 582 to determine the total loss 592. The total loss 592 may be caused by an estimation performed by at least one of the first feature extraction neural network 522 or the message decoding neural network 562. The loss function determines the deviation between the feature loss 572 calculated by the feature loss calculation engine 570 and the image loss 582 calculated by the image loss calculation engine 580.

[0088] The total loss function to be optimized can be expressed by the following equation:

[0089] L=L il (I i ,I o )+L fl (f,f ii ) (1)

[0090] Where the total loss is equal to the image loss function (input image, output image) plus the feature loss function (features, features recovered from the input image). In the equation, the total loss is L and the image loss function is L il , the feature loss function is L fl , the input image is I i , the output image is I o , the feature is f, and the feature recovered from the input image is f ii .

[0091] After calculating the total loss, the calculated total loss 592 is used to further optimize at least one of the feature extraction neural network 522, the message decoding neural network 562, or the message encoding / decoding neural network (e.g., the message encoding / decoding neural network). For example, the calculated total loss 592 may be used to optimize the parameters of one or more layers of the feature extraction neural network 522 or the message decoding neural network 562. Figure 5 The process shown is repeated until the calculated total loss is equal to or below a predetermined threshold, or until other convergence criteria are met. This training cycle can be repeated for multiple images from the image library 104 to iteratively optimize one or more of the feature extraction neural network 522, the message encoding neural network 542, or the message decoding neural network 562. In embodiments, parameter optimization enables the feature extraction neural network 522 to more accurately estimate features depicted in the image. Over time, through sufficient training through iterations of the training cycle, the feature extraction neural network 522 can improve such that the recovered features 536 provided by the message decoding neural network 562 to the feature loss calculation engine 570 converge to the determined features 528 provided by the feature extraction neural network 522 to the feature loss calculation engine 570.

[0092] In an embodiment, the detection neural network (e.g., Figure 2 The manipulation detection neural network 272 shown) is coupled to one or more message encoding neural networks (e.g., Figure 1 message encoding neural network 142), message decoding neural network (e.g., Figure 2 Message decoding neural network 252, Figure 5 message decoding neural network 562) or feature extraction neural network (e.g., Figure 1 Feature extraction neural network 122, Figure 2 Feature extraction neural network 232, Figure 4 Feature extraction neural network 422, Figure 4 The second feature extraction neural network 464, Figure 5 The first feature extraction neural network 522) is trained separately.

[0093] To train the manipulation detection neural network, the values of at least one of the feature extraction neural network, the message encoding neural network, the feature extraction neural network, or the message decoding neural network are fixed, and a training image is generated. The training image may include at least one image manipulation. In some aspects, the input image may be the training image. In some aspects, the received image may be the training image.

[0094] The system sends the training images to the manipulation detection neural network. The manipulation detection neural network determines image manipulations applied to the training images. Using the image manipulations applied to the training images, the manipulation detection neural network calculates a normal loss based on a normal loss function. In embodiments, the normal loss is a cross-entropy loss. The manipulation detection neural network is trained based at least on the calculated normal loss. In some aspects, the applied loss function is a normal loss function for classification.

[0095] The feature extraction neural network can be trained together with the message encoding / decoding neural network (e.g., a message encoding / decoding neural network, one or more message encoding neural networks, or a message encoding / decoding neural network) to extract features from the image and / or encode / decode the message in the image with improved accuracy. For example, by combining the above-mentioned separate feature extraction neural network training with the above-mentioned separate message encoding / decoding neural network training. The total loss function to be optimized can be represented by the following equation:

[0096] L=L il (I i ,I o )+L fl (f,f ii )+L dfl (f,f ri ) (2)

[0097] Where the total loss is equal to the image loss function (input image, output image) plus the feature loss function (features, features recovered from the input image) plus the discriminant feature loss function (features, features recovered from the received image). In this equation, the total loss is L and the image loss function is L il , the feature loss function is L fl , the input image is I i , the output image is I o , the feature is f, and the feature recovered from the input image is f ii , the discriminant feature loss function is L dfl , and the feature from the received image is f ri .

[0098] The input image may be perceptually identical to the output image, and the features recovered from the input image may be as close as possible to the features extracted (recovered) from the input image, even in the presence of some image manipulation (or forgery). In this case, the learned features may be features that are easier for the message encoding neural network to hide and extract. Furthermore, the learned features may be sensitive to image manipulation (or forgery), so that if the image is altered (e.g., manipulated, forged), the extracted features will be completely different from the features extracted from the input image. In such aspects, the manipulation detection neural network can be trained separately, as described above.

[0099] The feature extraction neural network can be trained together with the manipulation detection neural network to extract features from the image and / or determine with improved accuracy whether the received image is a forged or manipulated version of the input image. In such an aspect, the message encoding and decoding neural network (e.g., Figure 1 Message encoding neural network 142 and Figure 2 To train the message encoding and decoding neural network, the image manipulation (or forgery) that one wishes to detect may be applied to random input images having random features.

[0100] The total loss function to be optimized can be expressed by the following equation:

[0101] L=L il (I i ,I o )+L fl (f ra ,f rr ) (3)

[0102] The total loss is equal to the image loss function (input image, output image) plus the feature loss function (random features, recovered random features). In the equation, the total loss is L and the image loss function is L il , the feature loss function is L fl, the random feature is F ra , the recovered random feature is f rr .

[0103] The input image can be perceptually identical to the output image, and random features are recovered from the input image. The weights in the message encoding and decoding neural network are then fixed, and the feature extraction network and the manipulation detection neural network are jointly trained. The desired image manipulation or forgery can then be randomly applied or not.

[0104] The total loss function to be optimized can be expressed by the following equation:

[0105] L=L dfl (f,f ri )+L ce (4)

[0106] Where the total loss is equal to the discriminant feature loss function (features, features from the received image) plus the cross entropy loss (fake or real). In the equation, the total loss is L and the discriminant feature loss function is L dfl , the feature from the received image is f ri , the cross entropy loss (fake or real) is L ce Through such a training procedure, the feature extraction neural network will learn the best features for detecting fake or real images.

[0107] Message encoding and decoding neural networks (e.g., Figure 1 Message encoding neural network 142 and Figure 2 The message decoding neural network 252 can be trained together with the manipulation detection neural network to encode / decode messages in images and / or determine with improved accuracy whether a received image is a forged or manipulated version of an input image. In this regard, as described above, the feature extraction neural network is trained separately. After the feature extraction neural network is trained, the weights are fixed. To jointly train the message encoding and decoding neural network and the manipulation detection neural network, the image manipulation or forgery to be detected is randomly applied or not. The total loss function to be optimized can be represented by the following equation:

[0108] L=L il (I i ,I o )+L fl (f,f ii )+L ce (5)

[0109] Where the total loss is equal to the image loss function (input image, output image) plus the feature loss function (features, features recovered from the input image) plus the cross entropy loss (fake or real). In the equation, the total loss is L and the image loss function is L il, the feature loss function is L fl , the input image is I i , the output image is I o , the feature is f, and the feature recovered from the input image is f ii , the cross entropy loss (fake or real) is L ce In all respects, the input image can be perceptually identical to the output image, and the features recovered from the input image are as close as possible to the features extracted (restored) from the input image, even in the presence of some image manipulation (or forgery).

[0110] The feature extraction neural network can be trained together with the message encoding and decoding neural network (e.g., a message encoding / decoding neural network, a message encoding neural network, and a message decoding neural network) and the manipulation detection neural network to extract features from an image with improved accuracy, encode / decode messages in an image, and / or determine with improved accuracy whether a received image is a forged or manipulated version of an input image.

[0111] The manipulation or forgery that is desired to be detected may be applied or not applied randomly. The total loss function to be optimized can be expressed by the following equation:

[0112] L=L il (I i ,I o )+L fl (f,f ii )+L dfl (f,f ri )+L ce (6)

[0113] Where the total loss is equal to the image loss function (input image, output image) plus the feature loss function (random features, recovered random features) plus the discriminant feature loss function (features, features from the received image) plus the cross entropy loss (fake or real). In the equation, the total loss is L and the image loss function is L il , the feature loss function is L fl , the random feature is F ra , the recovered random feature is f rr , the discriminant feature loss function is L dfl , the feature from the received image is f ri , the cross entropy loss (fake or real) is L ce .

[0114] If the received image is fake, the discriminant feature loss function will penalize the case features, and the features from the received image are close. If the received image is real, the discriminant feature loss function will penalize the case features, and the features from the received image are far. It is desirable that the input image is perceptually identical to the output image. It is desirable that the message encoding and decoding neural network is robust to image manipulation or forgery. It is desirable that the extracted features are sensitive to image manipulation or forgery.

[0115] While the features and concepts of the described techniques and apparatus for verifying image authenticity can be implemented in any number of different environments, systems, devices, and / or various configurations, aspects of verifying image authenticity are described in the context of the following example devices, systems, and configurations.

[0116] To the extent that the techniques and / or devices discussed herein may collect personal information about a user or may utilize personal information, the user may be provided with an opportunity to control whether the program or feature collects personal information, such as information about the user's social network, social actions or activities, occupation, preferences, or current location, or to control whether and / or how the system and / or method may perform actions that are more relevant to the user. In addition, certain data may be anonymized in one or more ways before it is stored or used, thereby removing personally identifiable information. For example, the user's identity may be anonymized so that the user's personally identifiable information cannot be determined, or the user's geographic location may be summarized to the location where the location information was obtained, such as the city, zip code, or state level, so that the user's specific location cannot be determined. Thus, users can control how information about them is collected and used.

[0117] method

[0118] Figure 6 An example method 600 for generating an image to be verified is shown. The method 600 may be as described above with reference to Figure 1 The system 100 utilizes reference Figure 1 At 602, the encoder system receives an input image. At 604, the encoder system performs feature recognition on the input image to determine a plurality of determined features of the input image. At 606, the encoder system generates a third output defining values representing the determined features of the input image. At 608, the encoder system provides the third output to a message encoding neural network. At 610, the message encoding neural network generates a signature from the third output, the third output defining values representing the determined features of the input image. At 612, the encoder system embeds a second signature into the input image to generate an output image. In some aspects, the method 600 includes verifying the authenticity of the image. The method 600 may be performed to include more or fewer operations than those shown, or in a different order.

[0119] In an example use case of method 600, a user of a user computing device captures an input image (photograph) using a camera module of the user computing device. An encoder system implemented on the user computing device receives the input image. The encoder system performs feature recognition on the input image to determine a plurality of determined features of the input image. The encoder system generates an output defining values representing the determined features of the input image. The encoder system provides the output to a message encoding neural network implemented on a memory device of the user computing device. The message encoding neural network generates a signature from the output defining values representing the determined features of the input image. The encoder system embeds the signature into the input image to generate an output image stored on the memory device of the user computing device.

[0120] In another example use case of method 600, an image is stored in an image repository. An encoder system implemented on a server computing device receives an input image from the image repository. For example, an operator of the server computing device may utilize the image repository to store images uploaded by users of a service provided by the operator. The encoder system performs feature recognition on the input image to determine a plurality of determined features of the input image. The encoder system generates an output defining values representing the determined features of the input image. The encoder system provides the output to a message encoding neural network implemented on a memory device of the server computing device. The message encoding neural network generates a signature from the output defining values representing the determined features of the input image. The encoder system embeds the signature into the input image to generate an output image stored on the memory device of the server computing device. The embedded signature can be later used to authenticate the image.

[0121] Figure 7 An example method 700 for verifying the authenticity of an image is shown. The method 700 may be as described above with reference to Figure 2 The system 200 utilizes reference Figure 2The method 700 is performed by one or more components described herein. At 702, a decoder system receives an image to be verified. At 704, the decoder system performs feature recognition on the received image to determine a plurality of determined features of the received image. At 706, the decoder system generates a first output, the first output defining a value representing the determined features of the received image. At 708, the decoder system decodes the received image using a message decoding neural network to extract a signature embedded in the received image. The embedded signature represents the recovered features of the received image. At 710, the decoder system generates a second output, the second output defining a value representing the recovered features of the received image. At 712, the decoder system provides the first and second outputs to a manipulation detection neural network. At 714, the manipulation detection neural network generates an estimate of the authenticity of the received image using at least the first and second outputs. In some aspects, the method 700 includes verifying the authenticity of the image. The method 700 may be performed to include more or fewer operations than those shown, or in a different order.

[0122] In an example use case of method 700, a decoder system implemented on a user computing device receives an image to be authenticated from a memory device implemented on the user computing device. The decoder system performs feature recognition on the received image to determine a plurality of determined features of the received image. The decoder system generates a first output defining a value representing the determined features of the received image. The decoder system decodes the received image using a message decoding neural network implemented on the user computing device to extract a signature embedded in the received image. The embedded signature represents recovered features of the received image. The decoder system generates a second output defining a value representing the recovered features of the received image. The decoder system provides the first and second outputs to a manipulation detection neural network implemented on the user computing device. The manipulation detection neural network generates an estimate of the authenticity of the received image using at least the first and second outputs.

[0123] In another example use case of method 700, a decoder system implemented on a server computing device receives an image to be verified from an image repository. For example, an operator of the server computing device may utilize the image repository to store images uploaded by users of a service provided by the operator. The decoder system performs feature recognition on the received image to determine a plurality of determined features of the received image. The decoder system generates a first output defining values representing the determined features of the received image. The decoder system decodes the received image using a message decoding neural network implemented on the server computing device to extract a signature embedded in the received image. The embedded signature represents recovered features of the received image. The decoder system generates a second output defining values representing the recovered features of the received image. The decoder system provides the first and second outputs to a manipulation detection neural network implemented on the server computing device. The manipulation detection neural network generates an estimate of the authenticity of the received image using at least the first and second outputs.

[0124] Machine Learning Models

[0125] Figure 8 A block diagram of an example machine learning model 800 is depicted according to an example embodiment of the present disclosure. Figure 8 As shown, the machine learning model 800 is trained to receive one or more types of input data and, in response, provide one or more types of output data. Figure 8 A machine learning model 800 is shown performing inference.

[0126] Input data may include one or more features associated with an instance or example. In some embodiments, the one or more features associated with an instance or example may be organized into a feature vector. In some embodiments, output data may include one or more predictions. Predictions may also be referred to as inferences. Thus, given features associated with a particular instance, a machine learning model may output a prediction for that instance based on those features.

[0127] The machine learning model can be or include one or more of a variety of different types of machine learning models. In particular, in some embodiments, the machine learning model can perform classification, regression, clustering, association, anomaly detection, recommendation generation, and / or other tasks.

[0128] Machine learning models can perform various types of classification based on input data. For example, a machine learning model can perform binary classification or multi-class classification. In binary classification, the output data may include the classification of the input data into one of two different categories. In multi-class classification, the output data may include the classification of the input data into one (or more) of two or more categories. Classification can be single-label or multi-label.

[0129] Machine learning models can perform discrete classification, where input data is simply categorized into one or more classes or categories.

[0130] The machine learning model can perform classification, wherein the machine learning model provides a numerical value for each of one or more categories that describes the degree to which the input data is believed to be classified into the corresponding category. In some cases, the numerical value provided by the machine learning model can be referred to as a "confidence score," which indicates the corresponding confidence associated with the classification of the input into the corresponding category. In some embodiments, the confidence score can be compared to one or more thresholds to provide a discrete classification prediction. In some embodiments, only a certain number of categories (e.g., one) with the relatively largest confidence scores can be selected to present a discrete classification prediction.

[0131] The machine learning model can provide probabilistic classification. For example, given a sample input, the machine learning model can predict a probability distribution of a set of categories. Therefore, the machine learning model can output the probability that the sample input belongs to the category for each category, rather than just outputting the most likely category to which the sample input should belong. In some embodiments, the sum of the probability distributions of all possible categories can be 1. In some embodiments, a softmax function or layer can be used to squeeze a set of real values associated with each possible category into a set of real values in the range (0, 1) whose sum is 1.

[0132] The probability provided by the probability distribution can be compared with one or more thresholds to provide a discrete classification prediction. In some embodiments, only a certain number of categories (e.g., one) with relatively large predicted probabilities can be selected to present a discrete classification prediction.

[0133] In implementations where a machine learning model performs classification, supervised learning techniques can be used to train the machine learning model. For example, a machine learning model can be trained on a training dataset that includes training examples that are labeled as belonging to (or not belonging to) one or more classes. Further details on supervised training techniques are provided below.

[0134] The machine learning model can perform regression to provide output data in the form of continuous numerical values. The continuous numerical values can correspond to any number of different metrics or numerical representations, including, for example, monetary values, scores, or other numerical representations. For example, the machine learning model can perform linear regression, polynomial regression, or nonlinear regression. For example, the machine learning model can perform simple regression or multiple regression. As described above, in some embodiments, a softmax function or layer can be used to squeeze a set of real values associated with two or more possible categories into a set of real values in the range (0, 1) that sum to 1.

[0135] The machine learning model can perform various types of clustering. For example, the machine learning model can identify one or more previously defined clusters to which the input data most likely corresponds. As another example, the machine learning model can identify one or more clusters within the input data. That is, where the input data includes multiple objects, documents, or other entities, the machine learning model can classify the multiple entities included in the input data into multiple clusters. In some embodiments where the machine learning model performs clustering, the machine learning model can be trained using unsupervised learning techniques.

[0136] Machine learning models can perform anomaly detection or outlier detection. For example, a machine learning model can identify input data that does not conform to an expected pattern or other characteristics (e.g., as previously observed from previous input data). For example, anomaly detection can be used for fraud detection or system failure detection.

[0137] The machine learning model can provide output data in the form of one or more recommendations. For example, the machine learning model can be included in a recommendation system or engine. As an example, given input data describing previous results for certain entities (e.g., scores, rankings, or ratings indicating success or enjoyment), the machine learning model can output suggestions or recommendations for one or more additional entities that are expected to have the desired results based on the previous results (e.g., eliciting scores, rankings, or ratings indicating success or enjoyment). As an example, given input data describing the number of products that a user has purchased or highly rated, a recommendation system can output suggestions or recommendations for additional products that the user may like or wish to purchase.

[0138] A machine learning model can act as an agent in an environment. For example, a machine learning model can be trained using reinforcement learning, which is discussed in further detail below.

[0139] The machine learning model can be a parametric model, while in other embodiments, the machine learning model can be a non-parametric model. In some embodiments, the machine learning model can be a linear model, while in other embodiments, the machine learning model can be a non-linear model.

[0140] As described above, the machine learning model can be or include one or more of a variety of different types of machine learning models. Examples of such different types of machine learning models are provided below for illustration. One or more of the example models described below can be used (e.g., combined) to provide output data in response to input data. In addition to the example models provided below, other models can also be used.

[0141] The machine learning model can be or include one or more classifier models, such as, for example, a linear classification model, a quadratic classification model, etc.

[0142] The machine learning model can be or include one or more regression models, such as, for example, a simple linear regression model, a multiple linear regression model, a logistic regression model, a stepwise regression model, a multiple adaptive regression spline; a locally estimated scatter plot smoothing model; and the like.

[0143] The machine learning model can be or include one or more decision tree-based models, such as, for example, classification and / or regression trees; ID3 (Iterative Dichotomy 3) decision trees; C4.5 decision trees; Chi-squared automatic interaction detection decision trees; decision barriers; conditional decision trees; and the like.

[0144] The machine learning model may be or include one or more core machines. In some embodiments, the machine learning model may be or include one or more support vector machines.

[0145] The machine learning model may be or include one or more instance-based learning models, such as, for example, a learning vector quantization model, a self-organizing map model, a locally weighted learning model, and the like.

[0146] The machine learning model may be or include one or more nearest neighbor models, such as, for example, a k-nearest neighbor classification model; a k-nearest neighbor regression model; and the like.

[0147] The machine learning model can be or include one or more Bayesian models, such as, for example, a naive Bayes model, a Gaussian naive Bayes model, a multinomial naive Bayes model; an averaged single dependency estimator; a Bayesian network; a Bayesian belief network; a hidden Markov model; and the like.

[0148] A machine learning model can be or include one or more artificial neural networks (also referred to as neural networks). A neural network can include a set of connected nodes, also called neurons or perceptrons. A neural network can be organized into one or more layers. A neural network that includes multiple layers can be called a "deep" network. A deep network can include an input layer, an output layer, and one or more hidden layers between the input and output layers. The nodes of a neural network can be connected or partially connected.

[0149] The machine learning model can be or include one or more feed-forward neural networks. In a feed-forward network, connections between nodes do not form loops. For example, each connection can connect a node in an earlier layer to a node in a later layer.

[0150] The machine learning model can be or include one or more recurrent neural networks. In some cases, at least some nodes of the recurrent neural network can form loops. Recurrent neural networks are particularly useful for processing input data that is continuous in nature. In particular, in some cases, the recurrent neural network can transfer or retain information from an earlier portion of an input data sequence to a later portion of the input data sequence by using recursive or directed recurrent node connections.

[0151] As an example, sequential input data may include time series data (e.g., sensor data versus time or images captured at different times). For example, a recurrent neural network can analyze the relationship between sensor data and time to detect or predict the direction of a swipe, perform handwriting recognition, etc. As another example, sequential input data may include words in a sentence (e.g., for natural language processing, speech detection or processing, etc.); notes in a musical composition; sequential actions taken by a user (e.g., detecting or predicting sequential application usage); sequential object states; etc.

[0152] Example recurrent neural networks include long short-term (LSTM) recurrent neural networks; gated recurrent units; bidirectional recurrent neural networks; continuous-time recurrent neural networks; neural history compressors; echo state networks; Elman networks; Jordan networks; recurrent neural networks; Hopfield networks; fully recurrent networks; inter-sequence configurations; and more.

[0153] The machine learning model can be or include one or more convolutional neural networks. In some cases, a convolutional neural network can include one or more convolutional layers that perform convolution on the input data using learned filters. Filters can also be referred to as kernels. Convolutional neural networks are particularly useful for vision problems, such as when the input data includes images such as still images or videos. However, convolutional neural networks can also be applied to natural language processing.

[0154] The machine learning model can be or include one or more generative networks, such as, for example, generative adversarial networks. The generative networks can be used to generate new data, such as new images or other content.

[0155] A machine learning model can be or include an autoencoder. In some cases, the purpose of an autoencoder is to learn a representation (e.g., a low-dimensional encoding) of a set of data, typically for the purpose of dimensionality reduction. For example, in some cases, an autoencoder may attempt to encode input data and then provide output data that reconstructs the input data from the encoding. More recently, the concept of autoencoders has been more widely used to learn generative models of data. In some cases, autoencoders include additional losses in addition to reconstructing the input data.

[0156] The machine learning model can be or include one or more other forms of artificial neural networks, such as, for example, deep Boltzmann machines, deep belief networks, stacked autoencoders, etc. Any of the neural networks described herein can be combined (e.g., stacked) to form more complex networks.

[0157] One or more neural networks can be used to provide embeddings based on the input data. For example, an embedding can be a representation of knowledge abstracted from the input data into one or more learned dimensions. In some cases, embeddings can be a useful source for identifying related entities. In some cases, embeddings can be extracted from the output of the network, while in other cases, embeddings can be extracted from any hidden node or layer of the network (e.g., near the final layer but not the final layer of the network). Embeddings can be used to perform tasks such as automatically suggesting the next video, product recommendations, entity or object recognition, etc. In some cases, embeddings are useful inputs to downstream models. For example, embeddings can be used to summarize input data (e.g., a search query) for downstream models or processing systems.

[0158] The machine learning model may include one or more clustering models, such as, for example, a k-means clustering model, a k-medians clustering model, an expectation-maximization model, a hierarchical clustering model; and the like.

[0159] The machine learning model can perform one or more dimensionality reduction techniques, such as, for example, principal component analysis; kernel principal component analysis; graph-based kernel principal component analysis; principal component regression; partial least squares regression; Sammon mapping; multidimensional scaling; projection pursuit; linear discriminant analysis; hybrid discriminant analysis; quadratic discriminant analysis; generalized discriminant analysis; flexible discriminant analysis; autoencoding; and the like.

[0160] The machine learning model can implement or be subject to one or more reinforcement learning techniques such as, for example, Markov decision processes, dynamic programming; Q-functions or Q-learning; value function methods; deep Q-networks; differentiable neural computers; asynchronous advantage actor-critic; deterministic policy gradients; etc.

[0161] A machine learning model can be an autoregressive model. In some cases, an autoregressive model can specify that the output data depends linearly on its own previous values and a random term. In some cases, an autoregressive model can take the form of a stochastic difference equation. An example autoregressive model is WaveNet, which is a generative model of raw audio.

[0162] A machine learning model may include or form part of a multi-model ensemble. As an example, bootstrap aggregation, which may also be referred to as "packing," may be performed. In bootstrap aggregation, a training dataset is divided into multiple subsets (e.g., by random sampling with replacement), and multiple models are trained on the multiple subsets, respectively. At inference time, the respective outputs of the multiple models may be combined (e.g., by averaging, voting, or other techniques) and used as the output of the ensemble.

[0163] An example model ensemble is the random forest, which can also be called a random decision forest. A random forest is an ensemble learning method for classification, regression, and other tasks. A random forest is generated by growing multiple decision trees during training. In some cases, at inference time, the class, which is the mode of the individual trees' classes (classification) or average predictions (regression), can be used as the forest's output. Randomized decision trees can correct for the tendency of decision trees to overfit to the training set.

[0164] Another example ensemble technique is stacking, which in some cases can be referred to as stacked generalization. Stacking involves training a combiner model to mix or otherwise combine the predictions of several other machine learning models. Thus, multiple machine learning models (e.g., of the same or different types) can be trained based on training data. Furthermore, the combiner model can be trained to take predictions from other machine learning models as input and, in response, generate a final inference or prediction. In some cases, a single-layer logistic regression model can be used as the combination model.

[0165] Another example ensemble technique is boosting. Boosting can include incrementally building an ensemble by iteratively training weak models and then adding to a final strong model. For example, in some cases, each new model can be trained to emphasize training examples that were misinterpreted (e.g., misclassified) by the previous model. For example, the weight associated with each such misinterpreted example can be increased. A common implementation of boosting is AdaBoost, which can also be called adaptive boosting. Other example boosting techniques include linear programming boosting (LPBoost); TotalBoost; BrownBoost; XGBoost; MadaBoost, LogitBoost, gradient boosting; and so on.

[0166] In addition, any of the models described above (e.g., regression models and artificial neural networks) can be combined to form an ensemble. For example, an ensemble can include top-level machine learning models or heuristic functions to combine and / or weight the outputs of the models that form the ensemble.

[0167] Multiple machine learning models (e.g., models forming an ensemble) can be linked and trained together (e.g., via sequential error backpropagation in the model ensemble). However, in some embodiments, only a subset (e.g., one) of the jointly trained models is used for inference.

[0168] Machine learning models can be used to preprocess input data for subsequent input into another model. For example, machine learning models can perform dimensionality reduction techniques and embeddings (e.g., matrix factorization, principal component analysis, singular value decomposition, Word2vec / GloVe, and / or related methods); clustering; and even classification and regression for downstream consumption. Many of these techniques have been discussed above and will be discussed further below.

[0169] Reference again Figure 8 As described above, a machine learning model can be trained or otherwise configured to receive input data and, in response, provide output data. The input data can include input data of different types, forms, or variations. As an example, in various embodiments, the input data can include determined image features and / or user-provided image features.

[0170] The machine learning model can receive and use input data in its raw form. In some embodiments, the raw input data can be pre-processed. Thus, in addition to or in lieu of the raw input data, the machine learning model can receive and use pre-processed input data.

[0171] Preprocessing the input data may include extracting one or more additional features from the original input data. For example, a feature extraction technique may be applied to the input data to generate one or more new additional features. Example feature extraction techniques include edge detection; corner detection; blob detection; ridge detection; scale-invariant feature transforms; motion detection; optical flow; Hough transforms; and the like.

[0172] The extracted features may include, or be derived from, a transformation of the input data into other domains and / or dimensions. For example, the extracted features may include, or be derived from, a transformation of the input data into the frequency domain. For example, a wavelet transform and / or a fast Fourier transform may be performed on the input data to generate additional features.

[0173] The extracted features may include statistics calculated from the input data or certain portions or dimensions of the input data. Example statistics include the mode, mean, maximum, minimum, or other measure of the input data or portions thereof.

[0174] As described above, the input data may be sequential in nature. In some cases, sequential input data may be generated by sampling or segmenting the input data stream. As an example, frames may be extracted from a video. In some embodiments, sequential data may be rendered non-sequential by aggregation.

[0175] As another example of a pre-processing technique, portions of the input data may be estimated, for example, by interpolation and / or extrapolation to generate additional synthetic input data.

[0176] As another example preprocessing technique, some or all of the input data may be scaled, standardized, normalized, generalized, and / or regularized. Example regularization techniques include ridge regression; least absolute shrinkage and selection operator (LASSO); elastic net; least angle regression; cross validation; L1 regularization; L2 regularization; etc. As an example, some or all of the input data may be normalized by subtracting the mean of the eigenvalues for a given dimension from each individual eigenvalue and then dividing by the standard deviation or another metric.

[0177] As another example preprocessing technique, some or all of the input data can be quantized or discretized. As yet another example, qualitative features or variables included in the input data can be converted into quantitative features or variables. For example, one-hot encoding can be performed.

[0178] Before feeding into a machine learning model, dimensionality reduction techniques can be applied to the input data. Several examples of dimensionality reduction techniques are provided above, including, for example, principal component analysis; kernel principal component analysis; graph-based kernel principal component analysis; principal component regression; partial least squares regression; Sammon mapping; multidimensional scaling; projection pursuit; linear discriminant analysis; mixture discriminant analysis; quadratic discriminant analysis; generalized discriminant analysis; flexible discriminant analysis; autoencoding; and so on.

[0179] During training, the input data can be intentionally deformed in a variety of ways to improve the robustness, generalization, or other qualities of the model. Example techniques for deforming the input data include adding noise; changing color, shade, or hue; magnification; segmentation; upscaling; and so on.

[0180] Reference again Figure 8 In response to receiving input data, the machine learning model 800 can provide output data. The output data can include output data of different types, forms, or variations. As an example, in various implementations, the output data can include values representing features of an image (e.g., input image, output image, received image), values representing image features embedded in the received image, and / or a prediction (e.g., an estimate) of image realism.

[0181] As described above, the output data may include various types of classification data (e.g., binary classification, multi-class classification, single label, multi-label, discrete classification, regression classification, probability classification, etc.) or may include various types of regression data (e.g., linear regression, polynomial regression, nonlinear regression, simple regression, multivariate regression, etc.). In other cases, the output data may include clustering data, anomaly detection data, recommendation data, or any other form of output data described above.

[0182] The output data may affect downstream processes or decisions. As an example, in some embodiments, the output data may be interpreted and / or acted upon by a rule-based regulator.

[0183] In various aspects, when a machine learning model is stored on a computing system (e.g., user computing system 302), software cryptographic rules (e.g., a secure hash algorithm) can be used to protect the integrity of the model and prevent third parties from tampering with the model (e.g., replacing a portion of the machine learning model with another model). In some aspects, an alarm signal can be generated in response to detecting an attempted tampering.

[0184] Thus, the present disclosure provides systems and methods that include or otherwise utilize one or more machine learning models to generate predictions (e.g., estimates) of image authenticity based on determined features embedded in an image and / or recovered image features. Any of the various types or forms of input data described above can be combined with any of the various types or forms of machine learning models described above to provide any of the various types or forms of output data described above.

[0185] The machine learning model 800 may be stored on a computing system (e.g., Figure 3 The machine learning model 800 may be stored on a user computing device or an embedded computing device and / or implemented locally on the user computing device or the embedded computing device. Output data obtained by implementing the machine learning model locally on the computing system may be used to improve the performance of the computing system (e.g., an application implemented by the computing system). As an example, Figure 3 A block diagram of a user computing system 302 (e.g., a mobile computing device) that locally stores and implements a machine learning model 320 is shown.

[0186] Machine learning models can be stored on server computing devices such as Figure 3In some cases, the output data obtained by implementing the machine learning model at the server computing device may be used to improve other server tasks, or may be used by other non-user devices to improve services performed by or for such other non-user devices. For example, the output data may improve other downstream processes performed by the server computing device for the user computing device or embedded computing device. In other cases, the output data obtained by implementing the machine learning model at the server computing device may be sent to and used by the user computing device, embedded computing device, or some other client device. For example, the server computing device may be said to perform machine learning as a service. As an example, Figure 3 A block diagram is shown of an example user computing system 302 that can communicate with an example server computing system 330 including a machine learning model 340 via a network 380.

[0187] Different respective portions of the machine learning model may be stored in and / or implemented by some combination of a user computing device, an embedded computing device, a server computing device, and the like.

[0188] The computing device may perform graph processing techniques or other machine learning techniques using one or more machine learning platforms, frameworks, and / or libraries, such as, for example, TensorFlow, Caffe / Caffe2, Theano, Torch / PyTorch, MXNet, Cognitive Toolkit (CNTK), and the like.

[0189] The computing devices may be distributed across different physical locations and connected via one or more networks. The distributed computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. In one example, the distributed computing devices may be controlled or directed using a parameter server.

[0190] Multiple instances of a machine learning model can be parallelized to provide increased processing throughput. For example, multiple instances of a machine learning model can be parallelized on a single processing device or computing device, or across multiple processing devices or computing devices.

[0191] As described above, the machine learning models described herein can be trained at a training computing system and then provided for storage and / or implementation at one or more computing devices. For example, the model trainer 360 can be located at the training computing system 350, such as Figure 3 As shown. The training computing system 350 can be included in one or more computing devices that implement the machine learning model or separated therefrom. As an example, Figure 3A block diagram of an example user computing system 302 in communication with an example training computing system 350 including a model trainer 360 is shown.

[0192] Machine learning models can be trained in an offline or online manner. In offline training (also known as batch learning), the model is trained on a static set of training data in its entirety. In online learning, the model is continuously trained (or retrained) as new training data becomes available (e.g., when the model is used to perform inference).

[0193] The model trainer can perform centralized training of the machine learning model (e.g., based on a centrally stored data set). In other embodiments, decentralized training techniques such as distributed training, federated learning, etc. can be used to train, update, or personalize the machine learning model.

[0194] The machine learning models described herein can be trained according to one or more of a variety of different training types or techniques. For example, in some embodiments, a machine learning model can be trained using supervised learning, where the machine learning model is trained on a training data set that includes instances or examples with labels. Labels can be applied manually by experts, generated through crowdsourcing, or provided through other techniques (e.g., through physics-based or complex mathematical models). In some embodiments, if the user has provided consent, the training examples can be provided by the user's computing device. In some embodiments, this process can be referred to as a personalized model.

[0195] A machine learning model can be trained by optimizing an objective function. For example, in some embodiments, the objective function can be or include a loss function that compares output data generated by the model from training data and labels associated with the training data (e.g., ground truth labels) (e.g., determining the difference between them). For example, the loss function can calculate the sum or mean of the squared differences between the output data and the annotations. As another example, the objective function can be or include a cost function that describes the cost of a particular result or output data. Other objective functions can include margin-based techniques, such as triplet loss or maximum margin training.

[0196] One or more of a variety of optimization techniques can be performed to optimize the objective function. For example, the optimization technique can minimize or maximize the objective function. Example optimization techniques include Hessian-based techniques and gradient-based techniques, such as, for example, coordinate descent, gradient descent (e.g., stochastic gradient descent), subgradient methods, etc. Other optimization techniques include black-box optimization techniques and heuristics.

[0197] Backpropagation of errors can be used in conjunction with optimization techniques (e.g., gradient-based techniques) to train a model (e.g., a multi-layer model such as an artificial neural network). For example, an iterative cycle of propagation and model parameter (e.g., weight) updates can be performed to train the model. Example backpropagation techniques include time-truncated backpropagation, Levenberg-Marquardt backpropagation, and the like.

[0198] The machine learning models described here can be trained using unsupervised learning techniques. Unsupervised learning can involve inferring functions that describe hidden structure from unlabeled data. For example, classification or categorization may not be included in the data. Unsupervised learning techniques can be used to generate machine learning models capable of performing clustering, anomaly detection, learning latent variable models, or other tasks.

[0199] The machine learning models described here can be trained using semi-supervised techniques that combine supervised and unsupervised learning.

[0200] The machine learning models described here can be trained or generated using evolutionary techniques or genetic algorithms.

[0201] The machine learning models described here can be trained using reinforcement learning. In reinforcement learning, an agent (e.g., a model) can take actions in an environment and learn to maximize rewards and / or minimize penalties resulting from those actions. Reinforcement learning differs from supervised learning problems in that there are no correct input / output pairs presented, nor are there explicit corrections for suboptimal actions.

[0202] One or more generalization techniques can be performed during training to improve the generalization of a machine learning model. Generalization techniques can help reduce overfitting of a machine learning model to the training data. Example generalization techniques include dropout, weight decay, batch normalization, early stopping, subset selection, stepwise selection, and others.

[0203] The machine learning models described herein may include or be affected by multiple hyperparameters, such as, for example, the learning rate, the number of layers, the number of nodes in each layer, the number of leaves in a tree, the number of clusters, and the like. Hyperparameters can affect model performance. Hyperparameters can be selected manually or automatically by applying techniques such as grid search; black-box optimization techniques (e.g., Bayesian optimization, random search, etc.); gradient-based optimization; and the like. Example techniques and / or tools for performing automatic hyperparameter optimization include Hyperopt; Auto-WEKA; Spearmint; Metric Optimization Engine (MOE); and the like.

[0204] When training a model, various techniques can be used to optimize and / or adjust the learning rate. Example techniques and / or tools for performing learning rate optimization or adaptation include AdaGrad, Adaptive Moment Estimation (ADAM), ADADELTA, RMSprop, etc.

[0205] Transfer learning techniques can be used to provide an initial model from which to train the machine learning models described herein.

[0206] The machine learning models described herein can be included in different portions of computer-readable code on a computing device. In one example, the machine learning model can be included in a specific application or program and used (e.g., exclusively) by such specific application or program. Thus, in one example, a computing device can include multiple applications, and one or more such applications can include their own corresponding machine learning libraries and machine learning models.

[0207] The machine learning models described herein can be included in an operating system of a computing device (e.g., in a central intelligence layer of the operating system) and can be called or otherwise used by one or more applications that interact with the operating system. In some embodiments, each application can communicate with the central intelligence layer (and the models stored therein) using an application programming interface (API) (e.g., a public API across all applications).

[0208] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized data repository for a computing device. The central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0209] The technology discussed here refers to servers, databases, software applications, and other computer-based systems, as well as the actions taken and information sent to and received from these systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes discussed here can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0210] Furthermore, the machine learning techniques described herein are readily interchangeable and combinable. Although certain example techniques have been described, many other techniques exist and can be used in conjunction with aspects of this disclosure.

[0211] Thus, although the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of explanation, not limitation, of the present disclosure. Those of ordinary skill in the art may readily make changes, variations, and equivalents to such embodiments. Therefore, the subject disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, which would be apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used with another embodiment to generate a further embodiment.

[0212] Example

[0213] In the next section, some examples are described:

[0214] Example 1: A method comprising: receiving, by a decoder system (220), an image to be verified (210); performing feature recognition on the received image (210) to determine a plurality of determined features (238) of the received image (210); generating a first output (236) defining a value representing the determined features (238) of the received image (210); decoding the received image (210) via a message decoding neural network (252) of the decoder system (220) to extract a signature (254) embedded in the received image (210), the signature a manipulation detection neural network (272) of a decoder system (220) that generates an estimate of the authenticity of the received image (210) using at least the first output (236) and the second output (256).

[0215] Example 2: The method of Example 1, wherein performing feature recognition on the received image further comprises at least one of: processing the received image using a first feature extraction neural network to select multiple features of the received image; or receiving a first user input to select multiple features of the received image.

[0216] Example 3: The method of Example 1 or Example 2 also includes: receiving an input image by an encoder system; performing feature recognition on the input image to determine multiple determined features of the input image; and generating a third output defining values representing the determined features of the input image.

[0217] Example 4: The method of Example 3, wherein performing feature recognition on a received image comprises processing the received image using a first feature extraction neural network to select multiple features of the received image; wherein performing feature recognition on an input image comprises processing the input image using a second feature extraction neural network to select multiple features of the input image; the method further comprises: providing a second output and a third output to an identification feature loss calculation engine; calculating, by the identification feature loss calculation engine, a first loss based on a first loss function based on at least the second output and the third output; calculating a second loss based on a second loss function based on at least the input image and the received image; calculating a total loss based on a total loss function based on at least the first loss and the second loss; and training at least one of the first feature extraction neural network or the second feature extraction neural network together with a message decoding neural network based at least on the calculated total loss.

[0218] Example 5: The method of Example 3 or Example 4, further comprising: providing a third output to a message encoding neural network; generating a second signature from the third output; and embedding the second signature into the input image to generate an output image.

[0219] Example 6: The method of Example 3, wherein performing feature recognition on the input image further comprises at least one of: processing the input image using a second feature extraction neural network to select multiple features of the input image; or receiving a second user input to select multiple features of the input image.

[0220] Example 7: The method of any one of Examples 4-6 also includes: executing a feature extraction neural network training program, including: calculating a third loss based on a third loss function based on at least the second output and the third output; and training at least one of the first feature extraction neural network or the second feature extraction neural network based on at least the calculated third loss.

[0221] Example 8: The method of any of Examples 3-7, wherein receiving an input image comprises at least one of: receiving the input image from an image repository; or receiving the input image in response to user selection; and wherein receiving a received image to be verified comprises at least one of: receiving the received image from an image repository; or receiving the received image in response to user selection.

[0222] Example 9: The method of any of Example 3, Example 5, or Example 8, wherein performing feature recognition on a received image comprises processing the received image using a first feature extraction neural network to select multiple features of the received image; wherein performing feature recognition on an input image comprises processing the input image using a second feature extraction neural network to select multiple features of the input image; the method further comprises: fixing a value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image comprising at least one image manipulation; sending the training image to a manipulation detection neural network; determining an image manipulation applied to the training image; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and training the manipulation detection neural network based at least on the calculated normal loss.

[0223] Example 10: The method of any one of Example 3, Example 5, or Example 8, wherein performing feature recognition on a received image comprises processing the received image using a first feature extraction neural network to select multiple features of the received image; wherein performing feature recognition on an input image comprises processing the input image using a second feature extraction neural network to select multiple features of the input image; the method further comprises: calculating an image loss based on an image loss function based on the input image; calculating a feature loss based on a feature loss function based on the input image; calculating a total loss based on the image loss and the feature loss; and training at least one of a message encoding neural network or a message decoding neural network based on at least the calculated total loss.

[0224] Example 11: The method of any of Examples 3-8 or 10, further comprising: applying at least one image manipulation to the input image.

[0225] Example 12: The method of Example 10 also includes: fixing a value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image including at least one image manipulation; sending the training image to the manipulation detection neural network; determining the image manipulation applied to the training image; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and training the manipulation detection neural network based at least on the calculated normal loss.

[0226] Example 13: The method of Example 5 or Example 8 also includes: executing a feature extraction neural network training program, including: calculating a third loss based on a third loss function based on at least the second output and the third output; training at least one of the first feature extraction neural network or the second feature extraction neural network based on at least the calculated third loss; and executing a manipulation detection neural network training program, including: fixing the value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image including at least one image manipulation; sending the training image to the manipulation detection neural network; determining the image manipulation applied to the training image; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; training the manipulation detection neural network based on at least the calculated normal loss; and jointly training the feature extraction neural network and the manipulation detection neural network.

[0227] Example 14: The method of Example 3 or Example 8, wherein performing feature recognition on a received image comprises processing the received image using a first feature extraction neural network; wherein performing feature recognition on an input image comprises processing the input image using a second feature extraction neural network; the method further comprises: providing a third output to a message encoding neural network; generating a second signature from the third output; embedding the second signature into the input image to generate an output image; providing the second output and the third output to an identification feature loss calculation engine; executing a message encoding and decoding neural network training program, comprising: calculating, by the identification feature loss calculation engine, a first loss based on a first loss function based at least on the second output and the third output; calculating a second loss based on a second loss function based at least on the input image and the received image; and calculating a second loss based at least on the first loss function. loss and a second loss, calculating a total loss based on the total loss function; training at least one of the first feature extraction neural network, the second feature extraction neural network, or the message decoding neural network based on at least the calculated total loss; performing a manipulation detection neural network training procedure, including: fixing a value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image including at least one image manipulation; sending the training image to the manipulation detection neural network; calculating a normal loss for the image manipulation applied to the training image based on a normal loss function, wherein the normal loss is a cross entropy loss; and training the manipulation detection neural network based on at least the calculated normal loss; and jointly training the message encoding neural network and the manipulation detection neural network.

[0228] Example 15: The method of Example 3 or Example 8, wherein performing feature recognition on a received image comprises processing the received image using a first feature extraction neural network; wherein performing feature recognition on an input image comprises processing the input image using a second feature extraction neural network; the method further comprises: providing a third output to a message encoding neural network; generating a second signature from the third output; embedding the second signature into the input image to generate an output image; providing the second output and the third output to an identification feature loss calculation engine; executing a message encoding and decoding neural network training program, comprising: calculating, by the identification feature loss calculation engine, a first loss based on a first loss function based on at least the second output and the third output; calculating a second loss based on a second loss function based on at least the input image and the received image; calculating a total loss based on a total loss function based on at least the first loss and the second loss; and training the first feature extraction neural network, the second feature extraction neural network, and the second feature extraction neural network based on at least the calculated total loss. at least one of the first feature extraction neural network or the second feature extraction neural network; performing a feature extraction neural network training procedure, including: calculating a third loss based on a third loss function based on at least the second output and the third output; training at least one of the first feature extraction neural network or the second feature extraction neural network based on at least the calculated third loss; and performing a manipulation detection neural network training procedure, including: fixing a value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image including at least one image manipulation; sending the training image to the manipulation detection neural network; determining the image manipulation applied to the training image; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and training the manipulation detection neural network based on at least the calculated normal loss; and jointly training the feature extraction neural network, the message encoding neural network, and the manipulation detection neural network.

[0229] Example 16: The method of any of Examples 4-15, wherein the first feature extraction neural network and the second feature extraction neural network are the same feature extraction neural network.

[0230] Embodiment 17: The method of any one of embodiments 9, 12, or 13-15, wherein the normal loss is a cross entropy loss.

[0231] Example 18: A computing device comprising: a processor; and a computer-readable storage medium having instructions stored thereon, the instructions, in response to execution by the processor, causing the processor to perform the method of any one of Examples 1 to 17.

[0232] Example 19: A system comprising one or more computers and one or more memory devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations including the method of any one of Examples 1 to 17.

[0233] Example 20: One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations including the method of any one of Examples 1 to 19.

[0234] in conclusion

[0235] Although techniques and apparatus for verifying image authenticity have been described using language specific to features and / or methods, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, these specific features and methods are disclosed as example embodiments of techniques and apparatus for verifying image authenticity.

Claims

1. A method performed by a computing device, comprising: receiving, by a decoder system, an image to be verified; performing feature recognition on a received image using a feature extraction neural network to determine a plurality of determined features of the received image; generating a first output defining a value representative of a determined feature of the received image; decoding the received image via a message decoding neural network of the decoder system to extract a signature embedded in the received image, the signature representing a recovered feature of the received image; generating a second output defining a value representative of the recovered feature of the received image; providing the first output and the second output to a manipulation detection neural network of the decoder system; generating, by the manipulation detection neural network, an estimate of the authenticity of the received image using at least the first output and the second output; receiving an input image through an encoder system; performing feature recognition on the input image to determine a plurality of determined features of the input image; generating a third output, the third output defining a value representative of the determined feature of the input image; as well as The feature extraction neural network and the manipulation detection neural network are jointly trained, wherein the feature extraction neural network is trained based on a third loss calculated based on the second output and the third output, and wherein the manipulation detection neural network is trained based on a normal loss trained based on an image manipulation determined based on processing a training image with the manipulation detection neural network.

2. The method according to claim 1, wherein Performing feature recognition on the received image further comprises at least one of the following: processing the received image using a first feature extraction neural network to select the plurality of determined features of the received image; or A first user input is received to select the plurality of determined features of the received image.

3. The method according to claim 1, in, performing feature recognition on the received image includes processing the received image using a first feature extraction neural network to select the plurality of determined features of the received image; wherein performing feature recognition on the input image comprises processing the input image using a second feature extraction neural network to select the plurality of determined features of the input image; The method further comprises: providing the second output and the third output to a discriminative feature loss calculation engine; calculating, by the discriminative feature loss calculation engine, a first loss based on a first loss function based on at least the second output and the third output; calculating a second loss based on a second loss function based on at least the input image and the received image; calculating a total loss based on a total loss function based on at least the first loss and the second loss; and At least one of the first feature extraction neural network or the second feature extraction neural network is trained together with the message decoding neural network based on at least the calculated total loss.

4. The method according to claim 1, further comprising: providing the third output to a message encoding neural network; generating a second signature from the third output; as well as The second signature is embedded in the input image to generate an output image.

5. The method according to claim 1, wherein Performing feature recognition on the input image further includes at least one of the following: Processing the input image using a second feature extraction neural network to select the plurality of determined features of the input image; or A second user input is received to select the plurality of determined features of the input image.

6. The method according to claim 3, further comprising: Execute the feature extraction neural network training procedure, including: calculating a third loss based on a third loss function based on at least the second output and the third output; and At least one of the first feature extraction neural network or the second feature extraction neural network is trained based on at least the calculated third loss.

7. The method according to claim 1, in, Receiving the input image includes at least one of the following: receiving the input image from an image repository; or receiving the input image in response to a user selection; as well as Wherein, receiving the received image to be verified includes at least one of the following: receiving the received image from an image library; or The received image is received in response to a user selection.

8. The method according to claim 1, in, performing feature recognition on the received image includes processing the received image using a first feature extraction neural network to select the plurality of determined features of the received image; wherein performing feature recognition on the input image comprises processing the input image using a second feature extraction neural network to select the plurality of determined features of the input image; The method further comprises: fixing a value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image comprising at least one image manipulation; sending the training image to the manipulation detection neural network; determining the image manipulation to apply to the training image; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and The manipulation detection neural network is trained based on at least the calculated normal loss.

9. The method according to claim 1, in, performing feature recognition on the received image includes processing the received image using a first feature extraction neural network to select the plurality of determined features of the received image; wherein performing feature recognition on the input image comprises processing the input image using a second feature extraction neural network to select the plurality of determined features of the input image; The method further comprises: Based on the input image, calculating an image loss based on an image loss function; Based on the input image, calculating a feature loss based on a feature loss function; Calculating a total loss based on the image loss and the feature loss; and At least one of the message encoding neural network or the message decoding neural network is trained based on at least the calculated total loss.

10. The method according to claim 1, further comprising: At least one image manipulation is applied to the input image.

11. The method according to claim 9, further comprising: fixing a value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image comprising at least one image manipulation; sending the training image to the manipulation detection neural network; as well as The image manipulation applied to the training image is determined.

12. The method according to claim 1, in, Performing the feature recognition on the received image includes processing the received image using a first feature extraction neural network; wherein performing the feature recognition on the input image comprises processing the input image using a second feature extraction neural network; The method further comprises: providing the third output to a message encoding neural network; generating a second signature from the third output; embedding the second signature in the input image to generate an output image; providing the second output and the third output to a discriminative feature loss calculation engine; Execute the message encoding and decoding neural network training program; Perform the manipulation detection neural network training procedure, including: fixing a value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image comprising at least one image manipulation; sending the training image to the manipulation detection neural network; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and training the manipulation detection neural network based at least on the calculated normal loss; and The message encoding and decoding neural network and the manipulation detection neural network are trained jointly.

13. The method according to claim 1, in, Performing the feature recognition on the received image includes processing the received image using a first feature extraction neural network; wherein performing the feature recognition on the input image comprises processing the input image using a second feature extraction neural network; The method further comprises: providing the third output to a message encoding neural network; generating a second signature from the third output; embedding the second signature in the input image to generate an output image; providing the second output and the third output to a discriminative feature loss calculation engine; Execute the message encoding and decoding neural network training program, including: calculating, by the discriminative feature loss calculation engine, a first loss based on a first loss function based on at least the second output and the third output; calculating a second loss based on a second loss function based on at least the input image and the received image; calculating a total loss based on a total loss function based on at least the first loss and the second loss; and training at least one of the first feature extraction neural network, the second feature extraction neural network, or the message decoding neural network based on at least the calculated total loss; Execute the feature extraction neural network training procedure, including: Calculating a third loss based on a third loss function based on at least the second output and the third output; training at least one of the first feature extraction neural network or the second feature extraction neural network based on at least the calculated third loss; and Perform the manipulation detection neural network training procedure, including: fixing a value of at least one of the first feature extraction neural network, the second feature extraction neural network, the message encoding neural network, or the message decoding neural network; generating a training image comprising at least one image manipulation; sending the training image to the manipulation detection neural network; determining the image manipulation to apply to the training image; calculating a normal loss based on a normal loss function using the image manipulation applied to the training image; and The manipulation detection neural network is trained based on at least the calculated normal loss; and the feature extraction neural network, the message encoding and decoding neural network, and the manipulation detection neural network are jointly trained.

14. A computing device comprising: processor; as well as A computer-readable storage medium having instructions stored thereon, wherein the instructions, in response to being executed by the processor, cause the processor to perform the method according to any one of claims 1 to 13.

Images