Method and system for training generative adversarial networks with heterogeneous data

Abstract

The present invention includes methods and systems for training a neural network to perform processing of digital data. The input data may be heterogeneous, and the method or system obtains a plurality of loss signals. The input data may be selected so as to balance the loss signal, and several conditions of the output data may be satisfied. When running a trained neural network on a digital frame image, an intermediate result of processing a frame may be used for a subsequent frame, and in this way, processing latency may be reduced.

Description

[0001] This application is a divisional application of the international application number PCT / EP2019 / 062880, the international application date May 17, 2019, the national phase application number 201980098514.6, and the application named “Method and system for training generative adversarial networks with heterogeneous data”.

[0002] The present invention relates to a method and system for training a generative adversarial network with structured data and a computer program product for performing the method. The present invention also relates to a method and system for maintaining the impression of eye contact in a video conference. The present invention also relates to a method and system for distorting pixels in an image. Background Art

[0003] Video conferencing has been around for more than a decade and is now a common form of communication. However, the technology has drawbacks that make people shy away (such as the inability to maintain eye contact) and prefer face-to-face conversations.

[0004] Ganin (Ganin et al., European Conference on Computer Vision, 2016, pp. 311-326) describes training a neural network to offset pixels in a photograph of an eye, thereby changing the gaze direction by a given angle. For training, they collected their own data set by looking at an object in front of a camera and letting them look at the point that appears. Such data collection procedures make it feasible to sample pairs of images of the same eye with different gaze directions. Kononenko (Kononenko et al., 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018), 2018, 535-539) continues this work and allows the use of only arbitrary pairs of images of the same eye without knowing the exact gaze direction of these images. Ganin and Kononenko both use the same dataset. In order to train such a distortion method, it is necessary to collect sequences of images of the eye, in which the individual and his / her head posture remain fixed, while his / her gaze direction changes. Even for Kononenko, not all of these images need to be labeled with their absolute gaze, which is also an expensive process that leads to a relatively small dataset relative to the current benchmark set in deep learning.

[0005] Giger (Giger et al., 2014 IEEE International Conference on Multimedia and Expo (ICME), Chengdu, 2014, 1-6) tilts the entire face so that the person appears to be facing his / her camera. This approach requires mapping pixels as a texture onto a 3D head model, or it only affects the face without affecting the head contour to change the proportions of the person. In any case, these methods require more complex detection of the head and provide more obvious errors in the final image.

[0006] Dolhansky (Dolhansky et al., "Eye In-Painting with Exemplar Generative Adversarial Networks". 2017) and Wood (Wood et al., "Gaze Director: Fully Articulated Eye Gaze Redirection in Video" 2017) disclosed the generation of new eyes in place of the original input eyes. To open the new eyes, example images are used with a generative adversarial network (GAN).

[0007] Wood's system matches the eyes and the area around them to a 3D model. The gaze direction of that model can then be modified before being fused back to the original face.

[0008] The downside of discarding the original input eyes is that it may reduce the realism of the video and / or lose the effect of eye movements. Trying to train sequentially on different datasets can avoid dealing with competing goals. But with regard to gaze distortions, first training on a synthetic dataset, and then training on real images such as photographs, would mean first learning to make a wide range of corrections, and only subsequently learning to keep the fidelity of these corrections, which may forget how to make the original corrections. Summary of the invention

[0009] Embodiments of the present invention relate to methods and systems for training a generative adversarial network with heterogeneous data and computer program products for performing the method.

[0010] In another aspect, embodiments of the present invention relate to methods and systems for maintaining the impression of eye contact in a video conference.

[0011] In another aspect, embodiments of the present invention relate to methods and systems for warping pixels in an image.

[0012] In one aspect of the invention, an embodiment of the invention provides a method for training a neural network by adjusting weight values ​​of nodes of the neural network, the neural network having output nodes.

[0013] The method may include storing the following types of digital images: a first data type as input and target digital real image pairs, a second data type as input and target digital synthetic image pairs, and a third data type as input unlabeled digital real images,

[0014] Each image is thus represented by pixels, which have pixel properties.

[0015] The method may include processing digital images of a first data type, a second data type, and a third data type;

[0016] The process includes the steps of creating one or more loss signals by, for example, the following methods:

[0017] - modifying each input image of the first data type, the second data type and the third data type to form a modified digital image of the first data type, the second data type and the third data type,

[0018] - processing pixels of the modified digital image by means of a neural network to obtain output values ​​in output nodes,

[0019] - Obtain loss signals through:

[0020] - For the first data type, use all output nodes,

[0021] - for a second data type, using at least one of the output nodes, and

[0022] - For the third data type, at least one of the output nodes is used, such that the second and third data types are using all of the output nodes together.

[0023] The processing may include the steps of combining the loss signals from the digital image of each of the first data type, the second data type, and the third data type into a combined loss signal:

[0024] The processing includes the steps of updating weight values ​​of the neural network by back-propagating the combined loss signal through the neural network; and

[0025] The process includes the step of storing the updated weight values.

[0026] The step of creating one or more loss signals may additionally include:

[0027] For each modified digital image of the first data type and the second data type residing in the output node,

[0028] A loss signal is created using the pixel values ​​of all output nodes and the target digital image.

[0029] The step of creating one or more loss signals may additionally include:

[0030] For each modified digital image,

[0031] processing the modified digital image by using the inversely modified network to obtain an inversely modified digital image in an output node;

[0032] For each data type, a loss signal is created using the pixel values ​​of the output node and the unmodified digital image.

[0033] The step of creating the one or more loss signals may additionally include:

[0034] For each modified and reference digital image of the first data type and the third data type,

[0035] For each output digital image, the trained adversarial network is made to estimate each modified and reference digital image by:

[0036] assigning first and second numerical output values, respectively, in the output node,

[0037] and creating a loss signal as a difference between the first output value and the second output value.

[0038] For each unmodified and modified digital image of the third data type, preferably

[0039] causing the trained adversarial network to estimate for each unmodified and modified digital image,

[0040] assigning first and second numerical output values, respectively, in the output node,

[0041] and the loss signal is created as the difference between:

[0042] The first numerical output value is added to the input correction value, and the second output value.

[0043] The method may include any of pre-processing, interleaving or shuffling all input data sets before constructing input data batches by sampling.

[0044] The method may include creating a mapping between a value in the output node and a target digital image.

[0045] In another aspect, embodiments of the present invention provide a method for warping a digital image for use with a system having means for processing and non-volatile memory,

[0046] The system is adapted to define facial landmarks of a digital image.

[0047] The method may include:

[0048] receiving a digital image;

[0049] The device for processing receives the frame image number x

[0050] detecting image pixel coordinates of facial landmarks on the image and sending the image pixel coordinates to a neural network,

[0051] The neural network is adapted to generate a distortion field;

[0052] Apply the warp field to frame image number x;

[0053] Outputting the distorted frame image number x on the display;

[0054] And for at least one frame image number x+n, n>=1;

[0055] Use the warp field obtained from frame image number x;

[0056] And for at least one frame image number x+m, m>=n+l;

[0057] Use the facial landmarks obtained from frame image number x+n.

[0058] The method may include reading an input dataset, wherein the input dataset is a heterogeneous dataset. The heterogeneous dataset may include any or all of labeled photo pairs, labeled CGI pairs, and unlabeled photos.

[0059] The method may include either interleaving or shuffling all input data sets before constructing input data batches by sampling.

[0060] When there are multiple types of datasets, each dataset can have an equal impact on the gradient, in which case they appear equally frequently in the batch, regardless of the dataset size.

[0061] To achieve equal occurrence, all input datasets before shuffling can be interpreted as an interleaved union of repeated datasets, where samples from each dataset alternately follow each other and can be reused when a dataset runs out of samples.

[0062] The method may include performing a first modification pass on labeled data pairs comprising input and target images.

[0063] A supervised loss signal can be calculated.

[0064] The first modification pass may also be performed on unlabeled data, but no supervised loss signal needs to be computed for it.

[0065] The network can be adapted to receive input data and correct for angles.

[0066] The means for processing may provide, for example, different types of data to the modified network.

[0067] The modification network can be adapted to distort the pixels of the input image with a correction angle.

[0068] The method may comprise representing the loss signal by an L distance calculated between pixel values ​​of the warped image and the target image.

[0069] The method may include storing the intermediate loss signal in a memory.

[0070] The method may include performing an inverse modification pass, wherein an input data set may be provided by output data from a first modification pass of labeled data and unlabeled data.

[0071] The target data may be original labeled input data, as well as unlabeled input data.

[0072] The correction angle may be a reverse correction angle.

[0073] As in the first modification pass, the L distance loss signal may be calculated in the reverse modification pass.

[0074] The obtained loss signal may be a cyclic loss signal of the training signal.

[0075] The inverse modification can stimulate the neural network to model a reversible function from eye image to eye image.

[0076] The intermediate loss signal may be stored in a memory.

[0077] In the method, the adversarial evaluation pass may include the adversarial network receiving the output from the first modification pass.

[0078] The modified image can be sent through an adversarial or discriminative network for estimation, and depending on the data type, different loss signals can be calculated.

[0079] An adversarial loss signal may be computed for the modified labeled photos.

[0080] Based on this result, it can decide whether the adversarial network estimates these images to be real or fake.

[0081] A gaze angle regression loss signal may be calculated and used to determine whether the adversarial network estimates the gaze angle as a target angle.

[0082] For example, a gaze angle regression loss signal may be calculated for the modified labeled CGT.

[0083] This gaze angle regression loss signal can be used to decide whether the adversarial network estimates the gaze angle in the modified image as the target angle.

[0084] An adversarial loss signal can be computed for unlabeled photos, and this result can be used to decide whether the adversarial network estimates these images to be real or fake.

[0085] The method may include sending an original unlabeled photograph through an adversarial network to be estimated.

[0086] The distorted image can be sent to the adversarial network for estimation.

[0087] The loss signal may be calculated as the difference between the estimated angle of the original, unlabeled photo plus the corrected angle and the estimated angle of the distorted photo.

[0088] The calculated loss signal may be given a numerical score or alphanumeric score, for example, on the realism of the photo output. The score for real images may be trained to be greater than or have a higher value than the score for modified images to create distinctions between different image types.

[0089] In the method, loss signal aggregation and refinement may include:

[0090] All previously calculated loss signals are weighted and added to a resultant or combined loss signal.

[0091] The gradient of the loss signal can be back-propagated to the modified network.

[0092] For example, fully training a generative adversarial network can be achieved when all included loss signals have converged.

[0093] The adversarial network is preferably trained while keeping the parameters of the modified network fixed.

[0094] The image evaluation pass may include: all input images are sent through the modified network, and the outputs are differentiated depending on their original type.

[0095] For unlabeled photos, only the adversarial loss signal needs to be computed, which gives a measure of whether the image is real or fake.

[0096] The modified network has preferably already been trained and used in operation.

[0097] Intermediate results may be used in order to reduce processing delays.

[0098] Any of the methods of the embodiments of the present invention may include unoptimized sequential processing, where a camera frame image number x is sent from a local device for processing to a cloud server or other device for processing, and facial landmark coordinates are detected and added to the image.

[0099] A frame image with facial landmarks may be sent to a trained modified neural network, which computes a warp field and applies it to the eye region of the frame image.

[0100] A cloud server or other device for processing can host the neural network.

[0101] The warped image may be sent from the cloud server or device for processing back to the local device for processing, which may then output a final image frame.

[0102] Several frames in each processing row (such as four frames in four processing rows) may be processed in parallel, for example, so that subsequent processing rows may use the results of one or more previous processing rows.

[0103] The processing of the first row may begin by sending a first camera frame image from a local device for processing to a cloud server or device for processing.

[0104] Facial landmark detection may be performed and facial landmark coordinates may be defined.

[0105] The next line (such as the second line) can be processed simultaneously on the second frame image, which is the image of frame number x+n, where n>=1. The second frame image can be sent from the local device for processing to the cloud server or the device for processing.

[0106] The facial landmark coordinates of the first frame image may be used instead of defining the facial landmark coordinates of the second frame image.

[0107] The modified neural network may receive the facial landmark coordinates of the second frame image and the first frame image, and may calculate a distortion field that is sent back from the cloud server or the device for processing to the local device for processing.

[0108] The processing of the next row, such as the third row, may be started simultaneously by sending the third frame image number x+m, m>=n+l to a cloud server or a device for processing.

[0109] Facial landmark detection may be performed, and facial landmark coordinates of the third frame image may be transmitted from the cloud server or the device for processing to the local device for processing.

[0110] At the same time, the next row, such as the fourth row, can be processed using the fourth frame image number x+q, q>=m+l on the local device for processing.

[0111] The warping field may have been received based on the second frame and the first frame, and the facial landmarks may have been received based on the third frame, which may be used to warp the next frame (such as the fourth frame) into the fifth frame.

[0112] Settings can be made that determine how far apart the frames used to calculate and display marker coordinates and distortions can be.

[0113] The method may receive labeled and unlabeled data that has been modified by a modification network.

[0114] The parameters of the modified network can be kept fixed while the adversarial network is trained to estimate each data.

[0115] Evaluating each data may include calculating a score for the data without knowing whether it is real or modified.

[0116] A higher score or a score with a higher value may be calculated for the real data, and a lower score or a lower value may be calculated for the modified data.

[0117] When training the modified network, the adversarial network may have been trained to give high scores or higher values ​​to the unmodified data, and to give low scores to the modified data. The parameters of the adversarial network are now kept fixed, whereby the modified network can then modify the data and send it to the adversarial network. This scores the data, whereby the parameters of the modified neural network can then be adjusted (in several iterations) until the score reaches the desired value.

[0118] The scores for all real data and modified data respectively may be aggregated. Aggregation may include calculating an average value for the corresponding set of data types.

[0119] A loss signal may be provided to the adversarial network based on the gap between the aggregated scores of the real data and the aggregated scores of the modified data.

[0120] In a further aspect, embodiments of the present invention may provide a system for training a neural network by adjusting weight values ​​of nodes of the neural network, the neural network having output nodes.

[0121] The system may include:

[0122] for storing digital images of a first data type as input and target digital real image pairs, a second data type as input and target digital synthetic image pairs, and a third data type as input unlabeled digital real images,

[0123] Each image is thus represented by pixels, which have pixel properties.

[0124] The system may include a processor or device for processing digital images of the first data type, the second data type, and the third data type.

[0125] The processor may be adapted to create one or more loss signals by:

[0126] - modifying each input image of the first data type, the second data type and the third data type to form a modified digital image of the first data type, the second data type and the third data type,

[0127] - obtaining an output value in an output node by processing pixels of the modified digital image through a neural network,

[0128] - Obtain loss signals through:

[0129] - For the first data type, use all output nodes,

[0130] - for a second data type, using at least one of said output nodes, and

[0131] - for the third data type, using at least one of the output nodes such that the second data type and the third data type are using all of said output nodes together;

[0132] The processor is adapted to combine the loss signals from the digital image of each of the first data type, the second data type, and the third data type into a combined loss signal:

[0133] The processor is adapted to update weight values ​​of the neural network by back-propagating the combined loss signal through the neural network; and

[0134] The processor is adapted to store the updated weight value.

[0135] The processor may be adapted to create a loss signal for each modified digital image of the first data type and the second data type residing in the output nodes using pixel values ​​of all output nodes and the target digital image.

[0136] The processor may additionally be adapted to:

[0137] For each modified digital image,

[0138] processing the modified digital image by using the inverse modified network to obtain an inverse modified digital image in an output node; and

[0139] For each data type, a loss signal is created using the pixel values ​​of the output node and the unmodified digital image.

[0140] The processor may additionally be adapted to:

[0141] For each modified and reference digital image of the first data type and the third data type,

[0142] For each output digital image, the trained adversarial network is made to estimate each modified and reference digital image by:

[0143] assigning first and second numerical output values, respectively, in the output node,

[0144] and creating a loss signal as a difference between the first output value and the second output value.

[0145] The system may include:

[0146] For each unmodified and modified digital image of the third data type,

[0147] a trained adversarial network for estimating each unmodified and modified digital image,

[0148] assigning first and second numerical output values, respectively, in the output node,

[0149] and the loss signal is created as the difference between:

[0150] The first numerical output value is added to the input correction value and the second output value.

[0151] The system may be adapted to any of pre-process, interleave or shuffle all input data sets before constructing input data batches by sampling.

[0152] The system may be adapted to create a mapping between a value in the output node and a target digital image.

[0153] In another aspect, embodiments of the present invention may provide a system for warping a digital image, wherein the system may include a processor or means for processing and a non-volatile memory.

[0154] The system may be adapted to define facial landmarks for a digital image.

[0155] The system may be adapted to receive a digital image.

[0156] The processor or means for processing is adapted to receive a frame image number x, to detect image pixel coordinates of facial landmarks on the image and to send the image pixel coordinates to a neural network, thereby

[0157] The neural network may be adapted to generate a distortion field;

[0158] and applying the warp field to frame image number x;

[0159] To output the distorted frame image number x on the display;

[0160] And for at least one frame image number x+n, n>=1;

[0161] Use the warp field obtained from frame image number x;

[0162] And for at least one frame image number x+m, m>=n+l;

[0163] Use the facial landmarks obtained from frame image number x+n.

[0164] The system may be adapted to read an input data set, wherein the input data set is a heterogeneous data set.

[0165] The heterogeneous dataset may include labeled photo pairs, labeled CGI pairs, and unlabeled photos.

[0166] The system may be further adapted such that before constructing input data batches by sampling, there is an interleaving or shuffling of all input data sets.

[0167] When there are multiple types of datasets, each dataset can have an equal impact on the gradient, in which case they appear equally frequently in the batch, regardless of the dataset size.

[0168] The system can be adapted to achieve equal appearance in the following way: all input data sets before shuffling are interpreted as an interleaved union of repeated data sets, where samples from each data set alternately follow each other and can be reused when a data set runs out of samples.

[0169] The system may be further adapted to perform a first modification pass on labeled data pairs comprising input and target images.

[0170] The system may be adapted to calculate a supervised loss signal.

[0171] The system may be adapted such that the first modification pass may also be performed on unlabeled data, but for which a supervised loss signal is not calculated.

[0172] The system may include a modification network adapted to receive input data and correct the angle.

[0173] The means for processing or the processor may provide different types of data to the modification network.

[0174] The modification network can be adapted to distort the pixels of the input image with a correction angle.

[0175] The system may be further adapted to represent the loss signal by an L distance calculated between pixel values ​​of the warped image and the target image.

[0176] The system may further include a memory for storing the intermediate loss signal.

[0177] The system may be adapted to perform a reverse modification pass comprising providing an input data set from output data from a first modification pass of labeled data and unlabeled data.

[0178] The target data may be original labeled input data, as well as unlabeled input data.

[0179] The correction angle may be a reverse correction angle.

[0180] As in the first modification pass, the L distance loss signal may be calculated.

[0181] The obtained loss signal may be a cyclic loss signal of the training signal.

[0182] The inverse modification pass can be adapted to stimulate the neural network to model a reversible function from eye image to eye image.

[0183] The system may further include a memory in which the intermediate loss signal is stored.

[0184] The adversarial evaluation pass may include the adversarial network receiving the output from the first modification pass.

[0185] The system can be adapted so that the modified image is sent through an adversarial or discriminative network for estimation, and depending on the data type, different loss signals are calculated.

[0186] The system may be adapted to compute an adversarial loss signal for the modified tagged photograph.

[0187] Based on this result, an adversarial network can be adapted to estimate whether these images are real or fake.

[0188] A gaze angle regression loss signal may be calculated.

[0189] The system can be adapted to use the gaze angle regression loss signal to decide whether the adversarial network estimates the gaze angle as a target angle.

[0190] A gaze angle regression loss signal may be calculated for the modified labeled CGT.

[0191] The system can be adapted to use the gaze angle regression loss signal to decide whether the adversarial network estimates the gaze angle in the modified image as the target angle.

[0192] The system can be adapted to compute an adversarial loss signal for unlabeled photos.

[0193] The system can be further adapted to use the results to decide whether the adversarial network estimates the images to be real or fake.

[0194] The system can be further adapted to send original unlabeled photos through the adversarial network to be estimated.

[0195] The system may be further adapted to send the distorted image to an adversarial network to be estimated.

[0196] The system may be configured to calculate the loss signal as the difference between the estimated angle of the original unlabeled photograph plus the correction angle, and the estimated angle of the distorted photograph.

[0197] The system may be configured to output the calculated loss signal as a numerical score for the realism of the photographic output.

[0198] The system may be configured such that the scores for real images are trained to be larger than the scores for modified images to create differentiation.

[0199] The system may be adapted so that for loss signal aggregation and refinement:

[0200] All previously calculated loss signals are weighted and added to a resultant or combined loss signal.

[0201] The system can be adapted to back-propagate the gradient of the loss signal to the modified network.

[0202] The system can be adapted to fully train the generative adversarial network when all included loss signals have converged.

[0203] The system can be adapted to train the adversarial network while keeping the parameters of the modified network fixed.

[0204] The image evaluation pass may include that all input images may be sent through the modified network and the outputs may be differentiated depending on their original type.

[0205] The system can be adapted to compute, for unlabeled photographs, only a loss-resistant signal that gives a measure of whether the image is real or fake.

[0206] The system may include a modified network that may be trained and used in operation.

[0207] The system may be adapted to use intermediate results in order to reduce processing delays.

[0208] The system may be adapted for non-optimized sequential processing, where a camera frame image number x is sent from a local device for processing to a cloud server or other device for processing, and facial landmark coordinates are detected and added to the image.

[0209] A frame image with facial landmarks may be sent to a trained modified neural network, which may calculate a warp field and may apply it to the eye region of the frame image.

[0210] The system can be adapted to host a cloud server or other device for processing the neural network.

[0211] The system may be adapted to send the warped image from the cloud server or device for processing back to the local device for processing, which is adapted to output the final image frame.

[0212] The system may be adapted to process several frames in several processing rows (such as four frames in four processing rows) in parallel so that subsequent processing rows can use the results of previous processing rows.

[0213] The system may be adapted to begin processing a row, such as a first row, by sending a first camera frame image from a local means for processing to a cloud server or means for processing.

[0214] The system may be adapted to perform facial landmark detection and define facial landmark coordinates.

[0215] The system may be adapted to start processing the next row (such as processing the second row) simultaneously on a second frame image, where the second frame image is an image of frame number x+n, where n>=1.

[0216] The system may be adapted to send the second frame image from the local device for processing to a cloud server or device for processing.

[0217] The facial landmark coordinates of the first frame image may be used instead of defining the facial landmark coordinates of the second frame image.

[0218] The modified neural network may be adapted to receive facial landmark coordinates of a next frame of image (such as a second frame of image) and a first frame of image or other previous images, and adapted to calculate a distortion field that is sent back from a cloud server or device for processing to a local device for processing.

[0219] The system may be adapted to start processing another row (such as the third row) by sending a third frame image number x+m, m>=n+l to the cloud server or device for processing.

[0220] The system may be adapted to perform facial landmark detection and transmit facial landmark coordinates of the third frame image from the cloud server or the device for processing to the local device for processing.

[0221] The system may be adapted to start processing the next row, such as the fourth row, with the fourth frame image number x+q, q>=m+l, locally on the device for processing.

[0222] The system may be adapted such that having received a warp field based on the second frame and the first frame, and having received facial landmarks based on the third frame, these may be used to warp the fourth frame into a fifth frame.

[0223] The system may be adapted to make settings that determine how far apart the frames used to calculate and display marker coordinates and distortions may be.

[0224] The system may be adapted to receive marked and unmarked data that has been modified by the modification network.

[0225] The system can be adapted to keep the parameters of the modified network fixed while training the adversarial network to estimate each datum.

[0226] The system may be adapted to estimate each data by calculating a score for that data without knowing whether it is real or modified.

[0227] The system may be adapted to calculate a higher score or a higher value for the real data and a lower score or a lower value for the modified data.

[0228] The system may be adapted so that when training the modified network, the adversarial network may have been trained to give high scores or values ​​to the unmodified data and to give low scores or values ​​to the modified data.

[0229] The system may be adapted so that the parameters of the adversarial network may now be kept fixed, and the modification network may be adapted to then modify the data and send it to the adversarial network, which scores the data.

[0230] The system may be adapted so that the parameters of the neural network may then be adjusted (in several iterations) until the score reaches the desired value.

[0231] The system may be adapted to aggregate scores of all real data and modified data separately.

[0232] The system may be adapted such that aggregating comprises calculating an average value for a corresponding set of data types.

[0233] The system may be further adapted to provide a loss signal to the adversarial network based on a gap between the aggregated scores of the real data and the aggregated scores of the modified data.

[0234] Embodiments of the present invention may provide a computer program product which, when executed on a processor, performs any method of the present invention.

[0235] A non-transitory machine-readable signal storage medium storing, for example, a computer program product may be provided. This may be an optical disk such as a CDROM or DVD-ROM, a hard disk, a flash memory, a solid-state non-volatile memory, a magnetic tape, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0237] Figures 1 to 6 A flow chart of an embodiment of the present invention is shown, which includes the steps of training a modified neural network.

[0238] Figures 7 and 8 A flow chart of an embodiment of the present invention is shown, which includes the steps of training an adversarial neural network.

[0239] Figure 9a ) and b) illustrate embodiments of the present invention including the operation of a trained modified neural network.

[0240] Fig.10 A block diagram of an embodiment of the present invention comprising a computerized system is shown.

[0241] definition

[0242] A "neural network" is a computer architecture in which several network processing nodes are implemented on at least one computer processor and interconnected in a manner that can be described as mimicking the neurons of the human brain. Each processor can host a computational node of the neural network. Each node can have an input with a weight value that defines how to process digital information sent to that node. A node can have several inputs, but usually only one output. These weights can be adjusted when training the neural network. Initially, these weight values ​​can be random. The nodes can be viewed as reference points connected by weights, and digital data can be sent through each node following a certain route or sequence or graph or network architecture and processed accordingly. Thus, a neural network provides a graph describing the calculations performed by a computer processor. Depending on the hardware, processing in each parallel branch of the node network can be performed simultaneously.

[0243] These nodes, weights, and node sequences can be stored on non-volatile memory. When the neural network processes each digital image, it can process the signals of these digital images, such as color points or brightness, which in turn can be represented by digital drive levels (such as drive current). In a fully connected neural network, all nodes are connected to each other.

[0244] A "generative adversarial network" (GAN) may contain two neural networks, one of which is generative (or modified) and one of which is adversarial (or discriminative). The generative neural network (or "generative network" or "modifying network") may generate or modify data, which may then be used as input to the adversarial neural network (or "adversarial network" or "discriminative network"). The adversarial network may then discriminate between the generated (or modified) data and the real or unmodified data. This may sometimes be referred to as "fake" or "real" data, respectively, and the separation may be viewed as a classification problem.

[0245] The results of the adversarial network can then be used to adjust the parameters of the generative network. The networks are trained alternately to keep the parameters fixed for the untrained network. Both the generative and adversarial neural networks can be convolutional neural networks.

[0246] For example, when training the modification network, the adversarial network may have been trained to give high scores to unmodified data and low scores to modified data. The parameters of the adversarial network are now held fixed. The modification network can then modify the data and send it to the adversarial network, which scores the data. The parameters of the modification neural network can then be adjusted (in several iterations) until the score reaches the desired value. For example, if realistic images are desired, the parameters of the modification network are adjusted until the adversarial network scores the modified images as real images.

[0247] The training of the adversarial network may include it receiving labeled and unlabeled data that has been modified by the modification network. The parameters of the modification network are now kept fixed. The adversarial network estimates each data, i.e., calculates the score of the data (without knowing whether it is real or modified), for example, it may give a higher score to the real data and a lower score to the modified data. Then, the scores of all the real and modified data may be aggregated separately (e.g., by calculating the average of the corresponding data type set).

[0248] The loss signal (or training signal) for the adversarial network can then be the difference between the aggregated scores for the real data and the aggregated scores for the modified data. The adversarial network can be trained / encouraged to output a higher score for the real data than for the modified data. Optionally, it is feasible to have the adversarial network classify images with a confidence in the interval [0, 1.0].

[0249] Thus, the two networks can train each other. In fact, these networks can be said to compete in a zero-sum game, where the generative (or modifying) network tries to "fool" the adversarial (or discriminative) network: the modifying network will try to produce modified data that the adversarial network will score as real data. These networks can provide feedback to each other, and when given unlabeled input data, they can learn to mimic the data distribution.

[0250] The dataset can be generalized to

[0251] oLabeled: Perfect but small dataset

[0252] oSynthesis: Sufficient controlled datasets, different inputs / outputs

[0253] o Unlabeled: sufficient uncontrolled dataset, similar input / output

[0254] An example of an embodiment not involving pixels could be speech to text:

[0255] ·Tagged: audio recording

[0256] Synthesis: Robotic reading of text

[0257] Unlabeled: Unlabeled speech + irrelevant conversations (the discriminative network checks whether the output text resembles actual conversations).

[0258] All three datasets can be used for training at the same time in the same batch.

[0259] "Gradient descent" is an iterative optimization algorithm in which the task is to minimize an objective function that can describe, for example, the difference between an obtained value and a target value or a reference value. At a certain point on the function, a step is taken that is proportional to the negative derivative of the gradient of the curve in that point. This will lead to the minimum of the function.

[0260] "Stochastic Gradient Descent" or "SGD" uses an approximate gradient, which reduces the risk of ending up in a local minimum and prevents strong zigzag behavior. Therefore, SGD speeds up convergence and is applicable to large datasets. Neural networks can be trained by regression using SGD.

[0261] Alternative methods may use alternating direction methods such as multipliers, evolution strategies, or synthetic gradients.

[0262] The above mentioned difference may also be referred to as a "loss signal".

[0263] A "loss signal" may be the difference between an estimated data value (or modified data value, or output data value) of a neural network and a reference or target data value.

[0264] The loss signal may be calculated by a processor managing the neural network.

[0265] A neural network can be represented as a function of all weights, where each weight aji can be multiplied by the input xj, such as

[0266] zi=a1ix1+a2ix2+…+ajixj+bi, (1)

[0267] There are i nodes, j node inputs and an offset or bias bi. Expression (1) is valid per layer in most cases (the index for the layer can be added to the expression).

[0268] An output node is a node that retrieves an output value, which may be, for example, a node in the last layer of a neural network.

[0269] When input data is sent through a neural network, it can be processed by all weights according to the node connections to obtain the relevant output data. The output data of a node can be a nonlinear function of the sum of all its inputs, z=f(zi). For example, in supervised training of a neural network, there may be reference output data available. The weights can be adjusted iteratively until the known input gives a usable output, and thus the neural network is trained. By using derivative-based optimization (such as gradient descent), training in a certain direction can be encouraged by using a preferred direction for weight adjustment. For example, in the gradient descent method, the next step length can be proportional to the negative gradient of the function at the current position.

[0270] The difference between the obtained output and the reference value or data or image can be called a "loss signal". The target can be a reference. For example, for a reference value R, the loss signal can be the difference ||RZ||, where Z is a vector containing (f(zl1),…,f(zlp)), where l is the last layer with p nodes. These nodes can be called output nodes.

[0271] The loss signal may also include outputs of at least one but fewer than p nodes.

[0272] Therefore, the loss signal can be an expression with the same number of terms as there are weights in the neural network. A deep neural network has multiple layers of nodes, and all weights of one layer are adjusted / processed first before any weights in the next layer are adjusted / processed. All layers between the first or input layer and the last or output layer can be called hidden layers. The weights of a neural network can be adjusted by "back propagating" the loss signal through the network, one layer at a time. This means that the weights are increased in the negative direction of the gradient of the loss signal or loss function expression.

[0273] For each iteration, the loss signal value may also be calculated as a floating point number. Predefined limits may be set for the variation of this value and may be used to define the convergence of the optimization.

[0274] For example, if the input data is an image, then xj may comprise pixel values, and the image may be represented as X being an NxMxP matrix, where N and M are the height and width of the image (e.g., in pixels), and P may be a variable number that describes the pixels by their attributes (e.g., color).

[0275] For example, X may then be MxNx3, where each xj may include red, green, and blue color values. The reference image R may be represented as an NxMxP matrix. The last layer of the neural network may be configured to have NxMx3 nodes so that a loss signal may be created. The nodes of the last layer may be referred to as output nodes. Alternatively, if the output from the neural network has another format, the output may be converted to the format of the reference data or image. The neural network may then be trained so that the outputs from the last neural network nodes (or their transformations) converge toward R, for example by minimizing the loss signal.

[0276] The neural network may be trained to obtain continuous values, for example the reference data may include a range of decimal numbers. In this case, the loss signal may be considered an error signal. If the neural network is trained to provide a realistic output, the output may be a real number, and predefined conditions may be used to consider the output as "unmodified" or "modified". Therefore, the loss signal is not always an error signal.

[0277] When the input data has traversed the neural network, the output data resides in the nodes of the last network layer, or output layer. If the format of the output data is different from the format of the reference data, a mapping can be performed between the output data and the reference data so that for each output data value there is a reference data value available.

[0278] In the field of image processing, a data value may be a pixel attribute, such as color setting or brightness. It may also be a specific attribute of a group of pixels within an image, such as the gaze angle of an eye of an imaged face. The difference may then be expressed in a chosen measure, such as the L-norm, and may be referred to as the L-gap. Thus, while the attribute to be evaluated is physical and measurable, the actual loss signal associated with it may be normalized and lack units.

[0279] The loss signal can be a measure of the closeness to the target solution.

[0280] Training of a neural network may be performed by placing a limit on the loss signal, such that when the limit is reached the neural network is considered trained / optimized, and the training / optimization process is then stopped.

[0281] Alternatively, the training / optimization process may be stopped when the loss signal converges or a predetermined number of steps have been taken.

[0282] Machine learning and training of neural networks is an optimization problem. There is no guarantee that the loss signal will eventually reach its theoretical minimum. When a machine learning practitioner adds a loss signal, he / she specifies the direction to improve and its importance, without knowing how much the system will learn to improve.

[0283] Thus, a "gaze angle regression loss" or "gaze angle regression loss signal" can be a loss signal for gaze angle (in an image) as a result of a regression procedure. During this procedure or training phase, labels for real and synthetic data can be used to train an adversarial network to estimate images with gaze angles. For example, if the adversarial network believes that the image has / does not have the desired gaze angle, it can return a small / large loss signal.

[0284] Labeled data or images can be used to train a neural network. For example, a photo of a cat can have the label "cat". Alternatively, the label of an input image can be another image, which provides the desired reference information. For example, the label of a labeled input image for gaze angle can be an image of a face of a person whose eyes have adopted the desired gaze angle α. Preferably, such labels have the same format and type as the output of the neural network. For example, if the final output of the adversarial network is an estimated angle, the format of the label can be a string or a number (the actual representation in a computer system), and the type can be, for example, gaze angle.

[0285] Inputting a labeled image or inputting a reference digital truth image may include:

[0286] - an input image (e.g., a digital image in pixel format), - an image label (which may be the digital image itself representing the image features of the final result),

[0287] - the feature value in the label (float number),

[0288] - The corrected value of the feature (i.e. the difference between the feature value in the input image and the feature value in the label).

[0289] For example, if the dataset includes labeled images for gaze angles, the labeled samples may include:

[0290] - Input digital image showing a gaze angle of 3 degrees

[0291] - is a label or target showing a digital image with a gaze angle of 10 degrees

[0292] - The value of the gaze angle 10 degrees (float)

[0293] -Correction value (float) for gaze angle (i.e. 7 degrees).

[0294] There may be different labels for training the modified network and the adversarial network.

[0295] "Heterogeneous data" may be a data set with different data types and data formats. For example, labeled or unlabeled photos or labeled computer-generated images (CGI) are examples of different data types. Labeled and unlabeled photos may be referred to as labeled and unlabeled real images.

[0296] "Real data" is data that has not been modified, such as a photograph. Real data has been captured in the real world.

[0297] "Synthetic data" can be real data that has been digitally modified by a computer or completely generated by a computer. The purpose of synthetic data is to mimic real data.

[0298] “Modified data” is input data that has been changed by a neural network, such as an image that has been distorted by a neural network.

[0299] "Complementary data sets" include data of different data types. For example, one data type may include images showing realistic (to the viewer) content. Another data type may be images showing specific continuous values ​​(such as angles, lengths, volumes, colors, shapes, patterns, etc.).

[0300] The advantage of using complementary data sets is that the loss signal for each data type can be optimized for different (but desired) properties of the output. If only one loss signal were to be used, the result would not meet all requirements, but could be weighted towards one type of data. For example, a desired output could be a change in angle within an image. An undesirable weighted result could then be, for example, that the angle has changed, but the image is no longer realistic (or vice versa).

[0301] "Target data" represents a desired result. For example, it can be an image containing a correct representation of the object as a whole, or a desired value of a detail in the image. Target data is reference data. Examples of target data can be, for example, photographs, man-made or computer-generated drawings showing physical objects, people or animals, text, diagrams, maps, etc.

[0302] The corrected data is provided to the modification network as input for modification.

[0303] An image may be "distorted" so that the extension of a feature in the image may be, for example, lengthened, enlarged, shortened, reduced, or any change of shape that increases or decreases in size or modifies the outline. Additionally or alternatively, an image may be distorted by resampling the colors of the image to new pixel locations that are offset (not necessarily adjacent or continuous) from their previous locations. For example, for a pixelated image, this may mean that a feature extending over a first set of pixels may be distorted to extend over a second set of pixels. The second set of pixels may, for example, be smaller, larger, or differently shaped than the first set of pixels, and the second set of pixels may overlap or be adjacent to the first set of pixels in order to avoid discontinuities.

[0304] A "warp field" can be applied to any data collection (e.g., an image) to obtain a warped data collection. In the case of a pixelated image, the warp field can store coordinate offsets (e.g., color coordinate offsets) for each pixel instead of their RGB color values. For example, pixels with coordinate offsets (0,0) can be unwarped, i.e., retain their original color settings. Other pixels can be replaced with the color settings of pixels immediately adjacent to them (possibly at a distance beyond many pixel spacings (the distance between two adjacent pixels) steps). The distance can be expressed in steps of full pixel spacings, or as a fraction of them (i.e., a floating point number). In the latter case, the final position can be located between the pixels. The pixel can then take the average of the color settings of the two or more pixels closest to the final position. Applying the warp field can include copying and moving pixels from the original image to the warped image. The neural network can output the warp field instead of the warped image. Applying the warp field is computationally cheaper than computing the warp field with a neural network.

[0305] "Supervised training" of a neural network is done using only labeled data, so there is always a "ground truth" (known reference values) available. "Unsupervised training" of a neural network is when this data has no labels, so there are no reference values ​​available.

[0306] "Gaze correction" may be described as the process of changing the apparent viewing direction of the imaging eye. Optionally, gaze correction may be obtained through the use of distortions.

[0307] "Facial landmarks" or "key points" can be coordinates on a facial image, indicating points of interest. These points can make it easier to compare facial images and treat them uniformly, for example when performing image warping. For example, the eyes of a person are always included in the set of points of interest. This allows subsequent processing blocks to easily take the positions of these eyes as input or use these eye positions to calculate the best image crop for only the eyes. In this way, the processing block only needs to be designed to handle the pixels of the eyes, rather than any combination of pixel values.

[0308] Information extraction about facial landmarks can be provided by Bulat et al. (https: / / www.adrianbulat.com / downloads / FaceAlignment / FaceAlignment.pdf), Zadeh et al. (https: / / arxiv.org / pdf / 1611.08657.pdf), Cao et al. (https: / / arxiv.org / pdf / 1812.08008.pdf) or Faceplusplus (https: / / www.faceplusplus.com / landmarks / ).

[0309] A "pixel value" may be any value associated with a pixel in a digitally displayed image. For example, a color or tristimulus value or a brightness value. Indirectly, it may also be expressed as a drive current or drive voltage of a physical unit that displays the pixel. DETAILED DESCRIPTION

[0310] Past proposals for alleviating the problem of maintaining eye contact in interactive video sessions include, for example, specialized hardware such as camera arrays, depth cameras, or infrared cameras. Embodiments of the present invention include training methods or systems applicable to many conventional color cameras or to any conventional color camera (such as a web camera). Single channel cameras (e.g., black / white or grayscale) may also be used.

[0311] Ganin investigated using GANs to obtain gaze correction, where they only used labeled image pairs. But since they did not obtain any significant improvement, this approach was discarded. However, the inventors have found that it is indeed possible to train GANs to obtain gaze correction if heterogeneous datasets and multiple loss signals are used in training.

[0312] The amount of available training data can limit the size or depth of a neural network, i.e., the number of layers of the network. Since collecting labeled data pairs is a relatively slow and costly process, the size of a dataset consisting only of these kinds of samples will be limited. On the other hand, datasets consisting of unlabeled data can be large because they are available in large quantities in many different types of datasets (e.g., unlabeled images on the Internet). Similarly, large datasets of computer-generated images can become sufficiently diverse and quickly generated.

[0313] On the other hand, deep neural networks with many layers and parameters can easily overfit. Instead of learning the desired function, the network starts to memorize all the images in the training set and stores the desired output in the parameters.

[0314] The paper “Semi-supervised” (“SEMI-SUPERVISED REGRESSION WITH GENERATIVE ADVERSARIAL NETWORKS FOR END TO END LEARNING IN AUTONOMOUSDRIVING”, published anonymously for review, ICLR 2018) suggests using both labeled and unlabeled input data. Patent application WO2018184187A1 uses real unlabeled data together with synthetic data.

[0315] Embodiments of the present invention use heterogeneous datasets containing labeled and unlabeled real data together with synthetic data (e.g., computer-generated images), where each dataset type alone or two datasets containing two dataset types will not be enough to train the system, even if it contains a large number of training examples. The labels of all datasets and the corresponding training signals are made different and complementary together. This is a way to prevent the training of GAN from converging to a solution that is overfitting or does not meet all requirements (e.g., if the network outputs realistic images but there is no change in the gaze angle). Therefore, the above defects can be avoided. The supplementary dataset may include a smaller and fully labeled dataset, a combination of unlabeled photos, and computer-generated images (CGI). CGI can have perfect real conditions, i.e., exact information about the specific parameters sought. This type of dataset implementation uses a larger dataset and a deeper architecture (i.e., more layers in the neural network) than conventional solutions provide. The "semi-supervised" paper does not point out the addition of a synthetic dataset. WO2018184187A1 does not mention the addition of real labeled images. On the contrary, WO2018184187A1 suggests saving time and cost by replacing real labeled data with synthetic data.

[0316] Therefore, the prior art suggests supplementing the real labeled data set with unlabeled data (to increase the size of the data set) or replacing the real labeled data set with synthetic data (to get rid of expensive real labeled data and increase the data set). The prior art does not increase the number of data types in the input data set. However, the present invention can use at least real labeled data, synthetic labeled data and unlabeled data simultaneously when training a neural network. This enables the network to consider all types of given input data, thereby balancing the output.

[0317] An additional advantage of using a heterogeneous dataset in embodiments of the present invention is that it can be implemented by using any picture or photo with a face. Therefore, a large number of photos available on the Internet can be used to create a dataset.

[0318] Additionally, each modified image can be modified a second time with an inverse correction angle, where the unmodified input data is used as the target data. The modified neural network can then be encouraged to limit the error between the second output signal and the original input signal. Using such forward and inverse modifications makes unsupervised training feasible by allowing collection of any photo of the eye.

[0319] The system can create a reverse modification of the input data by applying the forward modification in the opposite direction. For example, if the input angle in the forward modification is x degrees, then in the reverse modification the input angle will be -x degrees. Such correction values ​​in the reverse direction may be referred to as reverse correction values.

[0320] Embodiments of the invention include GANs where an adversarial network can estimate a loss signal for a continuous attribute of a data set, such as the gaze angle of an eye image. Another loss signal can be used to stimulate at the same time, obtaining a realistic image. The combined loss signal can include a weighted sum of the loss signals from each pass in the process. The loss signal can be used, for example, to train a modified (or generative) network of the GAN using SGD regression.

[0321] The neural network can be evaluated on a fixed number of inputs before calculating the loss signal, and deriving the regression gradients and back-propagating the regression gradients to modify the network parameters. The modified (or generative) network and the adversarial (or discriminative) network can be trained alternately so that when one network is trained, the parameters of the other network are kept fixed. In a GAN configuration, the adversarial network can be used alone to train the modified network. When these networks have been trained, only the trained modified network is used for operation.

[0322] Figure 1 The process of an embodiment of the present invention is shown Fig.10 , which comprises the different steps of a method for training a modified network (while the parameters of the adversarial network are kept fixed). Steps 11 to 13 are performed by the modified network and the output is given as input to the adversarial network, which in steps 14 to 15 classifies or estimates the data set and calculates the gradient of the loss signal. Step 11 comprises reading the input data set; step 12 comprises the data passing through a first modification pass; step 13 comprises the data passing through a reverse modification pass; step 14 comprises the data passing through an adversarial evaluation pass; step 15 comprises loss signal aggregation and refinement. This step thus combines the gradients of the loss signal with respect to the weights of the neural network nodes from each step and creates a weighted sum, which in step 16 is back-propagated to the modified network to adjust its parameters.

[0323] Now I will explain in more detail Figure 1In these exemplary embodiments, the loss signal is derived from the gaze angle correction in the facial image. However, the present invention is not limited thereto.

[0324] Figure 2 Details of step 11, i.e., reading a dataset, are shown. The input may be a heterogeneous dataset, including, for example, labeled photo pairs or data pairs 21, labeled CGI pairs 22, and unlabeled photos 23. The labeled photo pairs may be examples of input and reference digital real images. The labeled CGI pairs may be examples of input and reference digital synthetic images. The unlabeled photos may be examples of input digital real images. In order to add diversity to the input order and gradients, the system may interleave 24 and shuffle 25 all the data before constructing an input batch 26, for example, by sampling. In the case of multiple types of datasets, each dataset has an equal impact on the gradient, in which case they appear equally frequently in the batch, regardless of the size of the dataset. To achieve this, all data before shuffling may be interpreted as an interleaved union of repeated datasets, where samples from each dataset follow each other alternately and can be reused when a dataset runs out of samples.

[0325] Figure 3 Details of a first modification pass 12 are shown, which is performed on labeled data pairs including input and target images and a supervised loss signal is calculated. The first modification pass is also performed on unlabeled data, but a supervised loss signal cannot be calculated for it. The modification network receives input data and correction angles in step 31. The training or running operation can be managed by a device for processing, which can provide different types of data to the modification network in step 32.

[0326] The modification network distorts the pixels of the input image with a correction angle. The correction angle can be a portion of the input sample or image. The loss signal can be represented by an L distance, for example, L2 (mean square error) can be calculated between the pixel values ​​of the distorted image and the target image. This can be performed in step 33 for the labeled photo pair 21 and in step 34 for the labeled CGI pair 22.

[0327] The intermediate loss signal may be stored in a memory.

[0328] Figure 4Details of the inverse modification pass 13 are shown. The input data is now the output data from the first modification pass of the labeled data and the unlabeled data. The target data is now the original labeled input data 21, 22 and the unlabeled input data 23. The correction angle is now opposite to the above correction angle. Therefore, the neural network uses the output data of the first modification pass or the distorted output to distort or modify the pixel 41 at the opposite correction angle. The method for processing management training can provide different types of data to the modification network in step 42. As in the first modification pass 12, the L distance loss 43, 44, 45 is calculated. The resulting loss signal can be referred to as a cyclic loss signal of the training signal. The inverse modification pass can stimulate the neural network to model a reversible function from the eye image to the eye image. Therefore, the output can be more likely to resemble a realistic eye. The intermediate loss signal can be stored in a memory.

[0329] Figure 5 Details of the inverse modification pass 14 are shown. The adversarial network receives the output from the first modification pass 12. The modified image may be sent 51 through an adversarial (or discriminant) network for evaluation, and depending on the type of data set, the apparatus for processing management training may provide different types of data to the adversarial network in step 52.

[0330] In step 53, an adversarial loss signal is calculated for the modified labeled photos. The input of the adversarial network here is the output modified image or photo and the target data. The adversarial loss signal can be obtained by studying the output digital image with an adversarially trained neural network. If the image is considered to be false / real, a large / small loss signal value can be defined. The result can be used to decide whether the adversarial network estimates these images to be real or false. In step 54, a gaze angle regression loss signal is calculated, and the result can be used to decide whether the adversarial network estimates the gaze angle as the target angle.

[0331] In step 55, a gaze angle regression loss signal is calculated for the modified labeled CGT. The result can be used to decide whether the adversarial network estimates the gaze angle as the target angle.

[0332] In step 56, an adversarial loss signal is calculated for the modified unlabeled photo. Optionally, in step 57, the original or unmodified unlabeled photo 23 may then be sent through the adversarial network for estimation. The warped image may then be sent to the adversarial network for estimation. The loss signal 58 is then the difference between the estimated angle of the original unlabeled photo plus the corrected angle and the estimated angle of the warped photo. (This loss signal controls whether angle correction has actually been achieved.)

[0333] The intermediate loss signal may be stored in a memory.

[0334] The calculated loss signal can give a numerical score for the realism of the photo output. The score for the real image can be trained to be larger and as far away from the score for the modified image as possible to create a distinction (although the number itself is not used).

[0335] The object of the present invention is to provide heterogeneous training data to a neural network so that training is performed on all provided data types at each iteration. This is obtained by providing a combined loss signal comprising loss signals from all different data types. Each loss signal depends on the value of one or more output nodes when an input value of a certain data type has passed through the network. For example, in the case of angle correction, this can be a single node comprising a floating point number. Or in the case of image modification, this can be a large number of nodes that each contain pixel values. Therefore, the use of one or more loss signals is associated with the use of one or more output nodes, and the use of all loss signals is associated with the use of all output nodes. In order to obtain a desired balance in the training of the neural network, it is therefore desirable to have a first loss signal that uses all output nodes, and at least two other loss signals for all output nodes that each use at least one node and use the first loss signal together.

[0336] Figure 6 Details of step 15 of loss signal aggregation and refinement are shown.

[0337] In step 61, all previously calculated loss signals may be weighted and added to a composite or combined loss signal. In step 62, the gradients are back-propagated to the modified network. In step 63, the means for processing management training checks whether the combined loss signal has sufficiently converged (e.g., the network may have been trained on 3,000,000 or 30,000,000 images). If this is not the case, the means for processing management training may check in step 64 whether the modified network (or modifier or generator) has been sufficiently trained (e.g., 50,000 images). If this is true, the system may switch to training the adversarial network (or discriminator) in step 65. In step 63, the system may conclude that all loss signals have converged, so the GAN has been fully trained and ends 66.

[0338] Figure 7 A flowchart 70 of an embodiment of the present invention is shown, which includes the different steps of a method for training an adversarial network (while the parameters of the modified network are kept fixed). The process is similar to the process of training the modified network, so steps 71, 72 and 73 correspond mutatis mutandis to Figure 1 Middle process Fig.10Steps 11, 14 and 15. The input to the adversarial network is the modified data and the target data (e.g., image labels), or unmodified unlabeled photos.

[0339] Figure 8 The details of the image evaluation pass 72 are shown. In step 81, all input images are sent through the modified network and in step 82, the outputs are differentiated by the means for processing management training depending on their original type. Steps 83, 84 and 85 correspond mutatis mutandis to Figure 5 Steps 53, 54 and 55 of the algorithm are described in detail. For unlabeled photos, only the adversarial loss signal is computed in step 86. This loss signal gives a measure of whether the image is real or fake. For example, the average of the loss signals for fake and real images, respectively, of a well-trained adversarial network should be easily distinguishable (although the values ​​themselves may not be correlated).

[0340] The inputs to the adversarial network are the modified data and the target data (e.g., image labels). The loss signal aggregation and refinement step 73 corresponds mutatis mutandis to Figure 6 Steps 61, 62, 63 and 66 in FIG.

[0341] Figure 9a ) and b) illustrate another embodiment of the invention, in which the modified network has been trained (eg by the above method) and is used in operation. This embodiment includes using intermediate results in order to reduce processing delays.

[0342] Figure 9a ) shows an unoptimized processing delay or sequential process 213, where a camera frame image number x 191 is sent from a local device for processing (not shown) to a cloud server (or other device for processing) 192, and facial landmark coordinates are detected 193 and added 194 to the frame image. Parameter x is an integer representing the position of the frame in the frame sequence, so, for example, frame image number x+1 is displayed before frame image number x+2. Parameters x, m, n, and q are integers.

[0343] The frame image with facial landmarks is sent to a (trained) modification neural network 195, which calculates a warp field 196 and applies it 197 to the eye region of the frame image. The cloud server (or other means for processing) 192 hosts the neural network. The warped frame image is then sent from the cloud / means for processing 198 back to a local means for processing (not shown), which outputs a final frame image 199.

[0344] Figure 9b) shows how the four frames in processing rows R1-R4 are processed in parallel to some extent so that subsequent processing rows can use the results of previous processing rows. This multi-step process with intermediate results achieves a trade-off between processing delay and distortion error. The individual processing steps are Figure 9a ) are the same as those described in .

[0345] Processing row R1 begins with sending a camera frame image 200 from a local device for processing (not shown) to a cloud / device for processing 201. Facial landmark detection 202 is performed and facial landmark coordinates 203 are defined.

[0346] Processing of row R2 begins simultaneously on frame image 204, which is frame image number x+n, where n>=1. Frame image 204 is sent from a local device for processing (not shown) to cloud / device for processing 201. Instead of defining facial landmark coordinates of frame image 204, the system uses facial landmark coordinates of frame image 200. Thus, modified neural network 205 receives frame image 204 and facial landmark coordinates 203 (of frame image 200), and computes warp field 206 that is sent back from cloud / device for processing 201 to the local device for processing (not shown). The cloud server (or other device for processing) hosts the neural network.

[0347] Processing row R3 begins simultaneously by sending a frame image 207 (with number x+m, m>=n+1) to the cloud / apparatus for processing 201. Facial landmark detection 208 is performed and facial landmark coordinates 209 of the frame image 207 are sent from the cloud / apparatus for processing 201 to a local apparatus for processing (not shown).

[0348] At the same time, processing of row R4 begins with frame image 210 (with number x+q, q>=m+1) on a local device for processing (not shown). Since the system has received the warp field 206 (based on frame images 204 and 200) and the facial landmark 209 (based on frame image 207), these can be used in step 201 to warp frame image 210 into frame image 211. Step 201 can be performed in a local device for processing, so that undesirable cloud processing can be avoided.

[0349] By connecting R4 to Figure 9a ) to obtain a reduction in processing delay.

[0350] The setting may determine how far apart the frames used to calculate and display the marker coordinates and distortions may be. The setting may be modified at any time. The inventors have found that the described process produces sufficient accuracy and has the advantage of reduced processing time.

[0351] In another embodiment of the invention, the adversarial network can also be used to run the operation. For example, the adversarial network can be used to do a final check on the warped result before showing it to the user, for example to ensure that the output data is still acceptable. For example, it can mean that the system checks whether the warped image used for gaze correction still resembles an eye. Then, Figure 9b ) is placed just before the final frame image 211. Since older facial landmarks are used, discrepancies may occur, for example, if the user moves too much, or if the image or correction angle is difficult to use. Because this will increase processing latency, although still only adding a small amount of non-optimized latency, it may be preferred to perform this check only on selected frames. Alternatively, this check may be performed when the facial landmarks or warp fields have exceeded a certain age threshold.

[0352] Additionally or alternatively, an adversarial network can be used to calibrate the size of the user's image and its position within the display frame. In order to make the relevant corrections, the system needs to know the distance between the camera and the eyes of the conversation partner on the display. The distance from the user to the camera can be estimated by looking at the head size in the camera frame and considering typical head sizes. The user can be asked to look at reference points on the display and the relevant gaze angle can be estimated. Using this input, a good estimate of the display size can be completed for a specific display. Using the above data, it is feasible to create one or more virtual cameras that are placed to overlap with the currently displayed individual.

[0353] Additionally or alternatively, the adversarial network can be used to limit the number of simultaneous gaze corrections that must be calculated. For example, in a lecture room with 60 students, it may be expensive to run gaze correction for everyone in the room simultaneously. This can be limited by only running gaze correction for people who are actually trying to make eye contact. Head pose may be sufficient to determine the display the teacher is currently looking at.

[0354] The following is an exemplary embodiment of how to train the neural network of the present invention.

[0355] Both the adversarial network and the modified network can be initialized using random weights. 5008 samples can be randomly extracted from the data set and sent through the untrained modified network. This can be the first data set for training the adversarial network. The adversarial network can process batches of 16 images at a time. After each batch, the gradient can be calculated and applied to modify these weights. After 500 batches, the training of the adversarial network can be stopped, and the modified network can start training batches of 16 samples. The input and output images are sent through the adversarial network using frozen weights. The truth score and the estimated angle can be used for loss signals. After, for example, every 100 batches, 40 batches can be sampled from the validation data set. If the loss signal value on these batches is lower than any previous loss signal value, these weights can be saved as "checkpoints", hereinafter referred to as minimum verification loss checkpoints. After 750 batches, the training of the modified network can be stopped. A new set containing 5008 samples can be collected and modified by the modified network. For batches of, say, 500, both the old and new modified sample sets are shuffled together and used to train the adversarial network.

[0356] The above loop can be iterated during, for example, two weeks (depending on the hardware). In one iterative session, the first validation loss signal value is 5.24 and the last validation loss signal value is 3.89. The lowest validation loss signal value is 2.40 (this may be due to the adversarial network having improved or due to overfitting). So frequently each time during training, the user can check the output of the modified network in terms of training and validation inputs for several angles. Both the minimum validation loss checkpoint and the last checkpoint are estimated. Depending on the image quality perceived by the observer, the training can be stopped.

[0357] Fig.10 Another embodiment of the present invention including a system of the present invention is shown. System 100 includes a CPU 101 that can host a computing node of one or more neural networks. CPU 101 can be local, for example, close to a user operating / using the system. Memory 102, display 104, and means 103 for additional processors can be connected to the local CPU via a digital network. The additional processor or CPU 103 can be local or remote (for example, on a cloud server) or a combination thereof. Display 104 can have pixels so that it can display a pixelated representation of an image. Each pixel can have various parameters, such as color point value and brightness, which can be represented by a digital drive level (DDL) (usually a current). The DDL can be controlled by CPU 101.

[0358] Additional processor 103 may be, for example, a remote CPU. CPU 101 may host the nodes of the neural network. Memory 102 may store these nodes and their interconnections, upload them to the RAM of CPU 101 and pass them to the remote CPU 103 if necessary. The training or training operations are managed by CPU 101. Alternatively, the neural networks may be stored in the local CPU 101. Processor 101 or 103 may include a GPU.

[0359] In another embodiment of the present invention, an apparatus and method for performing predictive analysis (e.g., implementing and using classification or regression methods) on the output data of the aforementioned embodiments in order to improve their performance (e.g., convergence speed or convergence value, or improved output quality (such as image quality)) is provided. The predictive analysis can be expressed in processor executable instructions, which can be in Fig.10 On a system that includes Fig.10 Unsupervised training may be performed by having an implementation of the foregoing embodiments run during training, and using the accumulated output data as a reference value when executing classification or regression instructions. Additionally or alternatively, supervised training may be performed by providing known reference values ​​to the system.

[0360] The method according to the present invention can be performed by a processor or processing device as a stand-alone device or embedded in a subsystem or other device. The present invention can use a processing engine adapted to perform a function. The processing engine preferably has a processing capability such as provided by one or more microprocessors, FPGAs or central processing units (CPUs) and / or graphics processing units (GPUs), and it is adapted to perform the corresponding functions by programming with software, i.e., one or more computer programs. References to software may include any type of program written in any language (via compiled or interpreted languages) that can be directly or indirectly executed by a processor. The implementation of any method of the present invention can be performed by logic circuits, electronic hardware, processors or circuits, which can include any kind of logic or analog circuits, integrated to any degree, and not limited to general-purpose processors, digital signal processors, ASICs, FPGAs, discrete components or transistor logic gates, etc.

[0361] The processing device or processor may have memory (such as non-transitory computer readable media, RAM and / or ROM), an operating system, optionally a display (such as a fixed format display), a port for a data input device (such as a keyboard), a pointing device (such as a "mouse"), a serial or parallel port for communicating with other devices, a network card and connections to any network.

[0362] The software may be implemented in a computer program product adapted to perform the functionality of any of the methods of the present invention when the software is loaded onto a controller and executed on one or more processing engines such as a microprocessor, ASIC, FPGA, etc. Thus, a processing device or processor for use with any embodiment of the present invention may be incorporated into a computer system capable of running one or more computer applications in the form of computer software.

[0363] The method described above with reference to the embodiments of the present invention can be performed by one or more computer applications running on a computer system, and the computer application runs on the computer system by being loaded into a memory and running on an operating system such as WindowsTM, Linux, Android provided by Microsoft Corporation of the United States or in association therewith. The computer system may include a main memory, preferably a random access memory (RAM), and may also include a non-transient hard disk drive and / or a removable non-transient memory and / or a non-transient solid-state memory. Non-transient removable memory may be an optical disk, a magnetic tape such as a compressed disc (CD-ROM or DVD-ROM) read and written by a suitable reader. Removable non-transient memory may be a computer-readable medium having computer software and / or data stored therein. Non-volatile storage memory may be used to store persistent information that should not be lost even when the computer system is powered off. Applications may use information and store it in non-volatile memory.

[0364] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0365] Training a generative adversarial network with heterogeneous data and a computer program product for performing the method.

[0366] Maintain the impression of eye contact during video conferences.

[0367] Distorts the pixels in an image.

[0368] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0369] A neural network is trained by adjusting the weight values ​​of the neural network nodes, which have output nodes.

[0370] storing the following types of digital images: a first data type as pairs of input and target digital real images, a second data type as pairs of input and target digital synthetic images, and a third data type as input unlabeled digital real images,

[0371] Each image is thus represented by pixels, which have pixel properties.

[0372] processing digital images of a first data type, a second data type, and a third data type;

[0373] The process includes the steps of creating one or more loss signals by, for example, the following methods:

[0374] - modifying each input image of the first, second and third data types to form a modified digital image of the first, second and third data types,

[0375] - processing pixels of the modified digital image by means of a neural network to obtain output values ​​in output nodes,

[0376] - Obtain loss signals through the following methods

[0377] - For the first data type, use all output nodes,

[0378] - for a second data type, using at least one of the output nodes, and

[0379] - For the third data type, at least one of the output nodes is used, such that the second and third data types are using all of the output nodes together.

[0380] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0381] The loss signals from the digital image of each of the first, second, and third data types are combined into a combined loss signal:

[0382] The processing includes the steps of back-propagating the combined loss signal through the neural network to update the weight values ​​of the neural network; and

[0383] Stores updated weight values.

[0384] Creating one or more loss signals may additionally include:

[0385] For each modified digital image of the first and second data types residing in the output node,

[0386] A loss signal is created using the pixel values ​​of all output nodes and the target digital image.

[0387] Creating one or more loss signals may additionally include:

[0388] For each modified digital image,

[0389] processing the modified digital image by using the inversely modified network to obtain an inversely modified digital image in an output node;

[0390] For each data type, a loss signal is created using the pixel values ​​of the output node and the unmodified digital image.

[0391] Creating the one or more loss signals may additionally include:

[0392] For each modified and reference digital image of the first and third data types,

[0393] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0394] For each output digital image, the trained adversarial network is made to estimate each modified and reference digital image by:

[0395] By assigning first and second numerical output values ​​in the output nodes, respectively,

[0396] A loss signal is created as a difference between the first output value and the second output value.

[0397] For each unmodified and modified digital image of the third data type, the trained adversarial network estimates for each unmodified and modified digital image,

[0398] assigning first and second numerical output values, respectively, in the output node,

[0399] And creating a loss signal as the difference between the first numerical output value plus the input correction value and the second output value.

[0400] All input datasets are preprocessed, interleaved, or shuffled before constructing input data batches through sampling.

[0401] Create a mapping between the values ​​in the output node and the target digital image.

[0402] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0403] A digital image is warped for use in a system having means for processing and non-volatile memory, the system being adapted to define facial landmarks of the digital image.

[0404] receiving a digital image;

[0405] Means for processing received frame image number x

[0406] Detecting the image pixel coordinates of facial landmarks on the image and sending the image pixel coordinates to the neural network,

[0407] The neural network is adapted to generate the distortion field;

[0408] Apply the warp field to frame image number x;

[0409] Outputting the distorted frame image number x on the display;

[0410] And for at least one frame image number x+n, n>=1;

[0411] Use the warp field obtained from frame image number x;

[0412] And for at least one frame image number x+m, m>=n+l;

[0413] Use the facial landmarks obtained from frame image number x+n.

[0414] An input dataset is read, wherein the input dataset is a heterogeneous dataset, whereby the heterogeneous dataset may include any or all of labeled photo pairs, labeled CGI pairs, and unlabeled photos.

[0415] All input data sets are interleaved or shuffled before constructing input data batches through sampling.

[0416] When there are multiple types of datasets, each dataset has an equal impact on the gradient, in which case they appear equally frequently in the batch, regardless of the dataset size.

[0417] Equal appearance is provided in the following way: all input data sets before shuffling are interpreted as an interleaved union of repeated data sets, where samples from each data set alternately follow each other and can be reused when a data set runs out of samples.

[0418] A first modification pass is performed on labeled data pairs comprising input and target images.

[0419] Compute the supervised loss signal.

[0420] The first modification pass is performed on unlabeled data, but no supervised loss signal needs to be computed for it.

[0421] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0422] Adaptation modifies the network to receive input data and correct angles.

[0423] Different types of data are provided to the modification network by the processor or means for processing.

[0424] Adaptation modifies the network to distort the pixels of the input image with the correction angle.

[0425] The loss signal is represented by the L distance calculated between the pixel values ​​of the warped image and the target image.

[0426] The intermediate loss signal is stored in a memory.

[0427] An inverse modification pass is performed, wherein an input data set may be provided by output data from a first modification pass of labeled data and unlabeled data.

[0428] The target data may be original labeled input data, as well as unlabeled input data.

[0429] The correction angle may be a reverse correction angle.

[0430] As in the first modification pass, the L distance loss signal is calculated in the reverse modification pass.

[0431] The obtained loss signal may be a cyclic loss signal of the training signal.

[0432] The inverse modification can stimulate the neural network to model a reversible function from eye image to eye image.

[0433] The intermediate loss signal may be stored in a memory.

[0434] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0435] The adversarial evaluation pass may include the adversarial network receiving the output from the first modification pass.

[0436] The modified image is sent through an adversarial or discriminative network for estimation, and depending on the data type, different loss signals can be calculated.

[0437] An adversarial loss signal may be computed for the modified labeled photos.

[0438] Based on this result, it can decide whether the adversarial network estimates these images to be real or fake.

[0439] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0440] Compute the gaze angle regression loss signal.

[0441] The gaze angle regression loss signal is used to decide whether the adversarial network estimates the gaze angle as the target angle.

[0442] A gaze angle regression loss signal is calculated for the modified labeled CGI.

[0443] This gaze angle regression loss signal is used to decide whether the adversarial network estimates the gaze angle in the modified image as the target angle.

[0444] An adversarial loss signal is computed for unlabeled photos, which can then be used to decide whether the adversarial network estimates these images to be real or fake.

[0445] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0446] Send the original unlabeled photos through the adversarial network to be estimated.

[0447] Send the distorted image to the adversarial network to be estimated.

[0448] The loss signal is calculated as the difference between the estimated angle of the original unlabeled photo plus the corrected angle, and the estimated angle of the distorted photo.

[0449] The calculated loss signal is given a numerical or alphanumeric score on the realism of the photo output.

[0450] The training score for real images must be greater than or have a higher value than the score for modified images to create a distinction between different image types.

[0451] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0452] Loss signal aggregation and refinement include:

[0453] All previously calculated loss signals are weighted and added to a resultant or combined loss signal.

[0454] Back-propagate the gradient of the loss signal to the modified network.

[0455] The generative adversarial network is fully trained when all included loss signals have converged.

[0456] The adversarial network is trained while keeping the parameters of the modified network fixed.

[0457] The image evaluation pass may include: all input images are sent through the modified network, and the outputs are differentiated depending on their original type.

[0458] For unlabeled photos, only the adversarial loss signal needs to be computed, which gives a measure of whether the image is real or fake.

[0459] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0460] Use the modified network that has been trained in the operation.

[0461] Use intermediate results to reduce processing delays.

[0462] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0463] An unoptimized sequential process is performed where a camera frame image number x is sent from a local device for processing to a cloud server or other device for processing, and facial landmark coordinates are detected and added to the image.

[0464] The frame image with facial landmarks is sent to the trained modified neural network, which calculates the warp field and applies it to the eye region of the frame image.

[0465] The neural network is hosted on a cloud server or other device for processing.

[0466] The distorted image is sent from the cloud server or device for processing back to the local device for processing, and then the local device for processing can output the final image frame.

[0467] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0468] Several frames in each processing row (such as four frames in four processing rows) are processed in parallel so that subsequent processing rows can use the results of one or more previous processing rows.

[0469] The processing of the first row may begin by sending a first camera frame image from a local device for processing to a cloud server or device for processing.

[0470] Perform facial landmark detection and define facial landmark coordinates.

[0471] The next row (such as the second row) can be processed starting from the second frame image at the same time, where the second frame image is the image with frame number x+n, where n>=1.

[0472] The second frame image is sent from the local device for processing to the cloud server or the device for processing.

[0473] The facial landmark coordinates of the first frame image are used instead of defining the facial landmark coordinates of the second frame image.

[0474] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0475] The modified neural network receives the facial landmark coordinates of the second frame image and the first frame image, and calculates a distortion field that is sent back from the cloud server or the device for processing to the local device for processing.

[0476] The next row, such as the third row, is processed by sending the third frame image number x+m, m>=n+l to a cloud server or a device for processing.

[0477] Facial landmark detection is performed, and the facial landmark coordinates of the third frame image are sent from the cloud server or the device for processing to the local device for processing.

[0478] The next row, such as the fourth row, is processed locally on the device for processing using the fourth frame image number x+q, q>=m+1.

[0479] A warp field based on the second frame image and the first frame image, and a facial landmark based on the third frame image are received.

[0480] The next frame image (such as the fourth frame image) is warped into the fifth frame image.

[0481] Makes settings that determine how far apart the frames used to calculate and display marker coordinates and distortions can be.

[0482] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0483] Labeled and unlabeled data that has been modified by a modified network is received.

[0484] The parameters of the network are fixed and the adversarial network is trained to estimate each data.

[0485] The data is estimated by calculating a score for each data without knowing whether it is real or modified.

[0486] A higher score or a score with a higher value is calculated for the real data, and a lower score or a lower value is calculated for the modified data.

[0487] The software embodied in the computer program product is adapted to perform the following functions when the software is loaded onto the corresponding device or devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0488] When training the modified network, the adversarial network may have been trained to give high scores or higher values ​​to the unmodified data and low scores to the modified data.

[0489] The parameters of the adversarial network are fixed, and the modified network can then modify the data and send it to the adversarial network.

[0490] This data is scored, whereby the parameters of the neural network are adjusted (in several iterations) until the score reaches the desired value.

[0491] Aggregation is performed on the scores of all real and modified data respectively, whereby aggregation may include calculating an average value for the corresponding set of data types.

[0492] Any of the above software may be implemented as a computer program product that has been compiled for a processing engine in any server or node in the network. The computer program product may be stored on a non-transient signal storage medium, such as an optical disk (CD-ROM or DVD-ROM), a digital tape, a disk, a solid-state memory (such as a USB flash memory), a ROM, etc.

[0493] For the purpose of explanation, specific examples of systems, methods and devices have been described herein. These are only examples. The technology provided herein can be applied to systems other than the above-described example systems. Within the practice of the present invention, many changes, modifications, additions, omissions and substitutions are feasible. The present invention includes changes in the described embodiments that are obvious to those skilled in the art, including changes obtained by: replacing features, elements and / or behaviors with equivalent features, elements and / or behaviors; mixing and matching features, elements and / or actions from different embodiments; combining features, elements and / or actions from the embodiments described herein with features, elements and / or actions of other technologies; and / or omitting combined features, elements and / or actions from the described embodiments. Therefore, it is intended that the following appended claims and the claims introduced below be interpreted as including all such modifications, arrangements, additions, omissions and sub-combinations that can be reasonably inferred. The scope of the claims should not be limited to the preferred embodiments set forth in these examples, but should be given the broadest interpretation consistent with the entire specification.

[0494] Although the present invention has been described above with reference to specific embodiments, this is done to illustrate rather than limit the present invention. It will be appreciated by those skilled in the art that various modifications and different combinations of the disclosed features may be made without departing from the scope of the present invention.

Claims

1. A method for warping a digital image for use with a system having means for processing and a non-volatile memory, the system being adapted to define a landmark of the digital image; The method comprises the following steps: receiving a digital image; Receiving, by the means for processing, a frame image number x; detecting image pixel coordinates of the mark on the image and sending the image pixel coordinates to a neural network; The neural network is adapted to generate a distortion field; Applying the warp field to the frame image number x; Outputting the distorted frame image number x on the display; And for at least one frame image number x+n, n>=1; Using the warp field obtained from frame image number x; And for at least one frame image number x+m, m>=n+l;, The flag obtained from the frame image number x+n is used.

2. The method of claim 1, wherein four frames in four processing rows are processed in parallel so that subsequent processing rows can use results of previous processing rows. The method according to claim 2 , wherein landmark detection is performed and landmark coordinates are defined.

4. The method according to claim 3, wherein the modified neural network receives the landmark coordinates of the second frame image and the first frame image, and calculates the distortion field returned from the cloud server or the device for processing to the local device for processing.

5. The method of claim 4, wherein processing of the third row is started simultaneously by sending the third frame image number x+m, m>=n+l to the cloud server or the device for processing. 6 . The method according to claim 3 , wherein the marker detection is performed and the marker coordinates of the third frame image are sent from the cloud server or the device for processing to the local device for processing.

7. The method of claim 3, wherein the warping field has been received based on the second frame image and the first frame image, and a flag has been received based on the third frame image, so that these can be used to warp the fourth frame image into a fifth frame image.

8. The method according to claim 2, wherein processing the first row begins by sending a first camera frame image from a local device for processing to the cloud server or the device for processing, and wherein processing the second row is simultaneously started on a second frame image, the second frame image being an image of frame image number x+n, where n>=l.

9. The method of claim 2, wherein processing the first row begins by sending a first camera frame image from a local device for processing to the cloud server or device for processing, and wherein processing of the fourth row is simultaneously started on the local device for processing using the fourth frame image number x+q, q>=m+l. 10 . The method of claim 1 , wherein the marker coordinates of the first frame image having the frame image number 1 are used instead of defining the marker coordinates of the second frame image having the frame image number 2 .

Images