Generating gaze-corrected images using bi-directionally trained network

Abstract

The invention relates to generating gaze-corrected images using a bi-directionally trained network. An example apparatus for correcting gaze in an image includes an image receiver to receive an image including an eye and a target angle set as a center. The apparatus also includes a bi-directionally trained convolutional neural network (CNN) to receive the image and the target angle from the image receiver and generate a vector field and a luminance map based on the image and the target angle. The apparatus also includes an image corrector to generate a gaze-corrected image based on the vector field and the luminance map.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 818,255, entitled “GENERATING GAZE CORRECTED IMAGESUSING BIDIRECTIONALLY TRAINED NETWORK”, filed March 14, 2019, the disclosure of which is incorporated herein by reference as if fully set forth herein. Technical Field

[0003] This application relates generally to the field of image processing, and more specifically to the use of bidirectionally trained networks to generate gaze correction images. Background Technology

[0004] Video conferencing systems enable users to interact face-to-face remotely. For example, one user might be using a laptop while another user is using a different computing device, such as a smartphone. Summary of the Invention

[0005] According to one aspect of this application, an apparatus for correcting gaze in an image is provided, comprising: an image receiver for receiving an image including an eye and a target angle set to a center; a bidirectionally trained convolutional neural network (CNN) for receiving the image and the target angle from the image receiver and generating a vector field and a luminance map based on the image and the target angle; and an image corrector for generating a gaze-corrected image based on the vector field and the luminance map.

[0006] According to another aspect of this application, a method for correcting gaze in an image is provided, comprising: receiving, via a processor, an image including an eye and a target angle set to a center; sending, via the processor, the target angle and the input image to a trained network; receiving, via the processor, a vector field and a luminance map from the trained network; and via the processor, warping the input image based on the vector field to generate a gaze-corrected image.

[0007] According to another aspect of this application, a system for correcting gaze in an image is provided, comprising: means for receiving an image including an eye and a target angle set to a center; means for generating a vector field and a luminance map based on the image and the target angle; and means for generating a gaze-corrected image based on the vector field and the luminance map.

[0008] According to another aspect of this application, a machine-readable medium including code is provided, which, when executed, causes a machine to perform the above-described method for correcting gaze in an image. Attached Figure Description

[0009] Figure 1 This is a block diagram illustrating an example system for generating gaze-corrected images using a bidirectionally trained network;

[0010] Figure 2 This is a schematic diagram illustrating an example process for bidirectional training of a model to generate gaze-corrected images;

[0011] Figure 3 This is a flowchart illustrating an example method for generating gaze correction images;

[0012] Figure 4 This is a flowchart illustrating an example method for predicting input gaze using a bidirectionally trained network;

[0013] Figure 5 This is a flowchart illustrating an example method for training a network to perform image adjustment and gaze prediction;

[0014] Figure 6 This is a block diagram illustrating an example computing device capable of performing image adjustment and gaze prediction; and

[0015] Figure 7 This is a block diagram illustrating a computer-readable medium storing code used to train a network to perform image adjustment and gaze prediction.

[0016] The same reference numerals are used throughout this disclosure and the accompanying drawings to refer to similar components and features. The reference numerals in the 100 series refer to those that initially appeared in [the original text]. Figure 1 The characteristics of the 200 series. The designations in the 200 series initially appeared in Figure 2 The characteristics of [the text]; and so on. Detailed Implementation

[0017] As discussed above, video conferencing systems can be used for remote, face-to-face interactions. For example, one user can face a camera on a first device, while another user faces a second camera on a second device to conduct a conversation or meeting. However, maintaining eye contact during such interactions can be difficult. For instance, the first user might be looking at something on their laptop monitor instead of looking directly at the camera. Consequently, the second user might see the first user's eyes look away from them.

[0018] This disclosure generally relates to techniques for correcting gaze in images. Specifically, the techniques described herein include apparatus, methods, and systems for correcting gaze in images using a bidirectionally trained network. Example apparatuses include an image receiver for receiving an image including eyes and a target angle set to center. For example, a face landmark detector can locate and crop eyes in an image and feed them into the image receiver. The apparatus includes a bidirectionally trained convolutional neural network (CNN) for receiving the image and target angle from the image receiver and generating a vector field and a brightness map based on the image and target angle. For example, the CNN can be a deep neural network. The CNN can be trained to receive the image and target angle from the image receiver and generate a vector field and brightness map based on the image and target angle. The apparatus further includes an image corrector for warping the input image based on the vector field to generate a gaze-corrected image. The image corrector further adjusts the local brightness of the warped image based on the brightness map to generate an enhanced gaze-corrected image. In various examples, the image corrector uses a set of control mechanisms to output frames with a smooth and naturally corrected gaze. For example, control mechanisms can regulate the intensity of correction and prevent adverse side effects from overcorrected eye contact, as well as ensure temporal consistency in live applications. In some examples, the CNN further predicts the input gaze direction and magnitude. The predicted input gaze direction and magnitude can, for example, allow an application to dim the device's screen backlight when the user is not actively engaged, significantly increasing battery life. In some examples, this gaze information can also enable handless interaction.

[0019] The techniques described in this paper thus enable a first user to perform a task and look at various parts of the display while still appearing to be looking directly at a second user watching the first user via a video stream. For example, the first user might be searching for documents or browsing the web. Specifically, the techniques described in this paper support the redirection of gaze from any direction to the center, regardless of the relative positions of the camera and the display. Bidirectionally trained CNNs learn to redirect any gaze to the center without additional information such as the redirection angle. Furthermore, these techniques can provide real-time gaze correction capabilities. Therefore, the techniques described in this paper can improve the quality of the video conferencing experience. In some examples, these techniques also enable the prediction of the direction and magnitude of the input gaze without additional processing costs. Moreover, these techniques can be used in live applications running in real time on a CPU, making them a viable solution for a wide variety of devices. In some examples, the predicted input gaze can be used to save power by dimming the screen of a computing device in response to detecting that the predicted input gaze is outside a certain boundary.

[0020] In addition, such as Figure 2 and Figure 5 As described, training a model to output a vector field and a brightness map offers several advantages over training a general encoder-decoder model that produces pixel-wise dense predictions. First, the vector field generated by the model can be easily modified in a meaningful way using external signals. For example, the magnitude of the vector field can be scaled before warping to control the correction intensity, and the vector field can be averaged over time for temporal smoothing. Second, the prediction of motion vectors presupposes that pixels move when the gaze changes, rather than changing in an unconstrained manner. Finally, training a model to directly output pixel values ​​can result in an output lacking detail and biased towards the average image from the training set. This problem occurs when the model is trained to minimize the mean squared error between the input and output pixel values.

[0021] Furthermore, training a model using adversarial loss can actually help generate images with high detail, as generative adversarial networks (GANs) learn what is important in the output. However, while GANs are better at reconstructing detail, the detail produced by a GAN may not originate from either the input or the ground truth. For example, when the output includes unrestricted pixels, a model trained with adversarial loss may hallucinate detail. For many applications, this behavior may be acceptable or even preferred. However, in a video conferencing setting, this flexibility of redirected gaze may not be desired. For example, adding eyelashes or any other hallucinated features could lead to undesirable results. Therefore, the techniques described in this paper can include models that manipulate the position and brightness of existing pixels. This approach ensures that any detail in the output originates from the input.

[0022] Figure 1 This is a block diagram illustrating an example system for generating gaze correction images using a bidirectionally trained network. This example system is typically referred to by reference numeral 100 and can be used... Figure 3 Method 300 is implemented in Figure 6 In the computing device 600.

[0023] Example system 100 includes a received target angle 102 and an input image 104. System 100 includes a tiler 106 for tiling the received target angle. System 100 includes a stitcher 108 communicatively coupled to the tiler 106. System 100 also includes four pairs of convolutional blocks 110 and pooling blocks 112 communicatively coupled to each other. System 100 also includes a plurality of upper convolutional blocks 114 paired with the convolutional blocks 110. System 100 includes a convolutional layer 116 coupled to two upper convolutional blocks 114 and a sigmoid block 118. The sigmoid block 118 can apply a sigmoid activation function to the received input. Convolutional layer 116 is shown as an output vector field 120. The sigmoid block 118 is shown in the output brightness map 122. System 100 includes a warper 124 for receiving the input image 104 and the vector field 120. System 100 includes a local brightness adjuster 126 communicatively coupled to a warp 124. The local brightness adjuster 126 is shown as an output image 128. Each of the convolutional blocks 108 in system 100 includes an additional block as indicated by the dashed lines. Each convolutional block 110 includes three sets of convolutional layers 116 coupled to a batch normalizer 130 and a corrected linear unit (ReLU) activation 132. Each convolutional block 110 also includes residual connections 134 communicatively coupled to these three sets of convolutional layers 116 coupled to the batch normalizer 130 and the ReLU activation 132. The convolutional blocks 108 having pooling blocks 110 and upper convolutional blocks 112 may be collectively referred to herein as neural networks or convolutional neural networks.

[0024] like Figure 1 As shown, system 100 (also referred to herein as Eye Contact Correction Network (ECC)-Net) receives an input image 104 and a target angle 102, whereby the input image 104 may be an image patch containing a single eye, and the target angle 102 may be a target gaze vector. In various examples, the image patches are resized to 64x32 before being fed into the model. In some examples, the target angle 102 may be a target gaze vector represented in the Cartesian domain by its horizontal and vertical components. Tiler 106 may tile the target angle 102 to have the same spatial dimensions as the input image. Once training is complete, the target angle 102 may be set to zero to redirect the gaze to the center. For example, the center may be the center of the image patch and may correspond to an eye that appears to be looking directly at the center of the camera lens. Stitcher 108 may stitch the tiled target angle 102 and the input image together. For example, a tiled target corner of size 64x32x2 can be stitched with an input image 104 of size 64x32x3 with three color channels to produce an input of size 64x32x5, which is then sent to the first convolution block 110 and the first pooling block 112.

[0025] The core of the ECC-Net system 100 is a fully convolutional encoder-decoder network that uses U-Net-style skip connections and channel-by-channel stitching to recover details lost at pooling layer 112. This network is also referred to as the model in this paper. In various examples, this model performs extensive processing at low resolution to both reduce computational cost and improve spatial consistency of the results. Figure 1 In the example, the convolutional block 110 of the model includes three depthwise-separable convolutional layers 116 with residual connections that skip intermediate layers (as indicated by the arrows). Figure 1 The convolutional layer 116 also uses batch normalization 130 and ReLU activation 132.

[0026] exist Figure 1 In the example, the first convolutional block 110 and the first pooling block 112 generate an output with a size of 32x16x32, which is fed to the second convolutional block 110 and the second pooling block 112. The second convolutional block 110 and the second pooling block 112 generate an output with a size of 16x8x64, which is fed to the third convolutional block 110 and the third pooling block 112. The third convolutional block 110 and the third pooling block 112 generate an output with a size of 8x4x128, which is sent to the fourth convolutional block 110 and the fourth pooling block 112. The fourth convolutional block 110 and the fourth pooling block 112 generate an output with a size of 4x2x256, which is fed to the first upper convolutional block 114 and the fifth convolutional block 110. The first upper convolutional block 114 and the fifth convolutional block 110 generate an output with a size of 8x4x128, which is fed to the second upper convolutional block 114, the sixth convolutional block 110, and the second splicer 108. The second upper convolutional block 114, the sixth convolutional block 110, and the second splicer 108 generate an output with dimensions of 16x8x64, which is fed into the seventh convolutional block 110 and the third splicer 108. The seventh convolutional block 110 and the third splicer 108 generate an output with dimensions of 16x8x32, which is fed into the S-shaped block 118, the convolutional layer 116, and the two upper convolutional layers 114.

[0027] The final output layer of the model consists of two upper convolutional layers 114, followed by a convolutional layer 116 with a 3-channel output. The model thus produces a vector field and a brightness map. As an example, the vector field 120 can be a flow field with dimensions of 64x32x2 and the brightness map 122 can have dimensions of 64x32x1. The upper convolutional layers 114 can be 2x2 convolutions with a stride of 1 / 2. Two of these channels are directly used to predict the horizontal and vertical components of the vector field used to warp the input image. The third channel is passed through a sigmoid function and used as a map to adjust local brightness. In some examples, the brightness map 122 can be used as a mask to improve the appearance of the whites of the eyes after warping the input image to redirect the gaze. The brightness map 122 can be used to enhance eye clarity and correct artifacts caused by horizontal warping when there are insufficient white pixels to recover the whites of the eyes.

[0028] In various examples, a set of control mechanisms or control blocks (not shown) may also be included to increase temporal stability and may include smooth control of the correction strength to ensure a natural video conferencing experience. Control blocks can be used to reduce the correction strength by scaling the ECC-Net output when appropriate. For example, a control block may be used to smoothly disable eye contact correction when the user is too far from the center, too far from the screen, too close to the screen, or blinks. In some examples, correction may also be disabled when the user looks at something other than the camera and display. In some examples, the control block monitors face size, distance from the center, head pose, eye opening ratio, and average and maximum motion vector magnitudes to determine the overall correction strength. Head pose may include monitoring pitch, roll, and yaw. In various examples, these measurements may be inferred from the output of the same face landmark detector used for eye alignment and cropping. In addition to face landmarks, the control block may also take into account the average and maximum motion vector magnitudes to limit correction for extreme gazes. In some examples, a scaling factor between 0 and 1 is generated based on the signals from both the landmarks and motion vectors. In some examples, the overall correction strength is calculated by multiplying the weights calculated for each trigger signal.

[0029] In various examples, this set of control mechanisms may include a time stabilization block (not shown) that can be included to apply a time filter to the output. For example, the stabilization block uses an α-β filter to filter motion vectors that temporarily warp the input image. The α-β filter used herein is a Kalman filter-related filter. This process eliminates flicker and aberrant motion vectors in the input video stream. When used in conjunction with the control block, the time stabilization block ensures that eye contact correction operates smoothly in video conferencing settings.

[0030] Overall, this set of control mechanisms prevents sudden changes and ensures that the eye contact corrector avoids making any adjustments when the user deviates from typical video conferencing use cases. Therefore, the eye contact corrector operates more smoothly and avoids clumsy or unnecessary adjustments.

[0031] Figure 1 The diagram is not intended to indicate that the example system 100 will include Figure 1 All components shown. Alternatively, fewer components or... Figure 1 Example system 100 is implemented using additional components not shown in the figure (e.g., additional target angle, input image, output image, vector field, brightness map, convolutional block, convolutional layer, dimension, etc.).

[0032] Figure 2 This is a schematic diagram illustrating an example process for bidirectional training of a model to generate gaze-corrected images. This process 200 can be implemented in the remote device 638 below. For example, it can be used... Figure 5 Method 500 or Figure 7 The process 200 is implemented by the trainer module 706 of the computer-readable medium 700.

[0033] Figure 2 The diagram shows the target angle 202 and input image 204 being received at ECC-Net 206A. For example, ECC-Net 206A and 206B can be any convolutional neural network (CNN), such as... Figure 1 The networks described herein. In some examples, ECC-Net 206A and 206B are deep neural networks. ECC-Net 206A is outputting an output image 208 for comparison with a target image 210 to calculate a correction loss 212. The output image and input angle 214 are input into ECC-Net 206B. ECC-Net 206B outputs a reconstructed image 216. The reconstruction loss is calculated based on the reconstructed image 216 and the input image 204. A set of shared weights 220 is shared between ECC-Net 206A and ECC-Net 206B. For example, the shared weights 220 can be updated at each epoch based on the correction loss 212 and the reconstruction loss 218.

[0034] like Figure 2As shown, ECC-Net 206A and ECC-Net 206B are trained bidirectionally to achieve mapping invertibility. Bidirectional training has proven valuable and visually improves results. The model is first fed an input image and a target angle to redirect the gaze at ECC-Net 206A. In this direction, the model is expected to minimize the correction loss. For example, the correction loss could be the mean square error between the gaze-corrected image and the standard ground truth. In the other direction, the model is fed a gaze-corrected output image and an input angle at ECC-Net 206B to redirect the gaze back to its original state. In various examples, warping artifacts in the output can make it difficult to recover the original image. To address this issue, the reconstruction loss 218 between the reconstructed image 216 and the original input image 204 is simultaneously optimized along with the correction loss 212.

[0035] Training the model in a periodically consistent manner reduces artifacts and produces more natural gaze redirection results. However, training with equal weights for the correction loss 212 and reconstruction loss 216 during training can lead to mode collapse, where the model rapidly converges to the identity transformation to minimize the reconstruction loss. Therefore, in some examples, the weights of the reconstruction loss are reduced to help the optimizer maintain a good balance between the loss functions in both directions. As an example, the weights of the loss in the total loss function are readjusted to L. total =0:8L c +0:2L r This helps the optimizer maintain a good balance between the loss function in both directions.

[0036] The target angle 202 is used during training and can be set to zero (0,0) during inference because the model aims to redirect the gaze to the center to correct eye contact. Training with the target orientation invariant can be used to improve the model's robustness and is permitted for post-training calibration. For example, if the gaze remains offset after correction on a particular device, the target angle 202 can be tuned to compensate for the offset. Using paired input images 204 with arbitrary gazes also increases the number of possible image pairs. For example, a set of 40 images can be used to generate [image pairs] for a given subject. Each image pair is unique, compared to only 39 image pairs generated using a single target. This method can be used to effectively augment data and reduce the risk of overfitting.

[0037] In some examples, the model is trained on both the left and right eyes, where the left eye on the synthetic dataset is generated by flipping the right eye. However, this can produce poor horizontal correction because the model may have to allocate consideration resources to determine whether the input is from the left or right eye in order to correct the gaze horizontally to the correct amount. In some examples, to better utilize the model's capabilities for correction, the model is trained only on the right eye and flipped to the left eye during inference. In this way, the model can learn to correct the gaze better in both the horizontal and vertical directions.

[0038] As an example, two distinct datasets were prepared for training and validating the system: a synthetic dataset and a natural dataset. The synthetic dataset consisted of images in which all variations except gaze remained constant. Similarly, the natural dataset consisted of image pairs in which the subject observed random points on a display and then the camera in each pair. ECC-Net 206A and ECC-Net 206B were trained using only the synthetic dataset, and the natural dataset was used as a validation set to select the best-performing model configuration. The use of the synthetic dataset avoids privacy and legal issues and can be an important component of the dataset used.

[0039] In this example, realistic images of eyes viewed from different directions are generated programmatically. For example, the synthetic data generator uses the UnityEyes platform, first released by Wood et al. in 2016, to render and rasterize the eye images, which were later improved by generative adversarial networks. In some examples, sets of eye images can be created by programmatically moving a cursor to move the gaze in random directions. For example, cursor movement can be modeled as a zero-mean Gaussian random variable, where zero signifies a centered gaze directly at the camera. To increase the diversity of samples in the dataset, subject features, lighting, and head pose are randomized across different image sets. Randomizing subject traits alters the color, shape, and texture of the face, skin, and eyes. Using this process, 3200 artificial subjects with randomized features were generated, resulting in 128,000 images and nearly 2.5 million image pairs.

[0040] In each example, the range of movement in head pose randomization is limited because eye contact correction may not be enabled if the user is clearly looking at something other than the camera and display. Therefore, head pose can be kept within the limits of a typical use case where eye contact correction would be practical.

[0041] In some examples, the rendering quality of the synthesized images is randomized to further increase randomness. For instance, using higher rendering quality may be suitable for many applications. However, depending on the type of camera used in a given use case, the amount of detail (such as reflections of the outside world on the surface of the eye) in an image rendered at the highest quality may be impractical.

[0042] After capturing raster images from the UnityEyes platform, glasses of different sizes and shapes can be overlaid on image sets to further increase dataset diversity. In some examples, glasses of different sizes and shapes are overlaid on image sets. For example, 25 different designs can be used as templates to generate glasses, where size, color, and relative position are randomized within a visually realistic range. Once the glasses are overlaid, the eyes can be aligned and cropped. For example, facial landmarks provided by the UnityEyes platform can be used to align and crop the eyes. These landmarks can be equivalent to those used for natural images. Since the images are generated synthetically, they can be accurately aligned before the eyes are cropped. However, simply using bounded boxes surrounding the eye landmarks to crop each image can result in small offsets between images within the same set, as the landmarks shift with gaze. Therefore, in some examples, bounded boxes are created to fit all images in a given set, and each set can use a single bounded box. In various examples, the bounded boxes have a fixed aspect ratio of 2:1 and are padded to have a width twice the average width in the given set.

[0043] In each example, all processing steps up to the cropping discussed above can be performed only once as preprocessing steps. In some examples, image pairs are instantaneously distorted during training due to additive noise, brightness and contrast shifts, and Gaussian blur in terms of random order and magnitude. These distortions not only simulate imperfect imaging conditions but also further amplify the diversity of samples in the dataset.

[0044] In some examples, generative adversarial networks (GANs) can be used to enhance the realism of photographs. For instance, GANs can learn a mapping between synthetic and real samples, and make the distribution of synthetically generated data more closely resemble the data captured by the camera. Using trained GANs, all images in a synthetic dataset can be improved to create a large dataset of photorealistic images with virtually perfect labels.

[0045] In each example, a natural dataset comprising image pairs can be collected, where participants move their gaze between a camera and random points on a display. The participants' gaze is guided by displaying points on a screen. In each example, a laptop or tablet is used as the data collection device, allowing subjects to participate in data collection at their convenience without being invited into a controlled environment. Therefore, the collected data can represent typical use cases for the proposed application.

[0046] To improve data quality, a routine can be used to automatically remove frames that are likely to be erroneous. First, the cleanup procedure removes the first frame of each sequence to compensate for hysteresis from the subject. Second, the routine removes frames where no face was detected. Finally, the routine removes frames where the subject blinks, inferred from the distance between eye landmarks. These cleanup steps may result in some incomplete image pairs, where either the input or standard ground truth image is completely removed. Incomplete image pairs can be removed to ensure that all pairs in the dataset are complete. As an example, a clean dataset could include 3,125 gaze pair sequences collected from over 200 participants.

[0047] In various examples, natural datasets can be used both to evaluate models and to make synthetic datasets more photorealistic. For instance, generative adversarial networks can be used to transform synthetic images into those that look natural. The ability to generate photorealistic synthetic datasets allows for the generation of large amounts of data with pixel-perfect labels at minimal cost.

[0048] As an example, ECC-Net was trained using only synthetic datasets, and natural datasets were used as validation sets to select the best-performing model configuration. Once training was complete, the frozen model was tested on the Columbia Gaze dataset, released in 2013, a public benchmark dataset originally used for eye contact detection. Using data from completely different sources for the training, validation, and test sets minimized the risk of overfitting, including its implicit forms such as information leakage from the validation set due to over-tuning of hyperparameters or dataset bias.

[0049] In this example, the relative reduction in mean squared error is used as a performance metric. The error metric is modified to be more robust to small amounts of misalignment. This misalignment-tolerant error uses the minimum error between image pairs shifted within a relaxed 3x3 pixel range. Compared to rigid pixel-to-pixel squared error, misalignment-tolerant error is likely to be more consistent with the visual quality of the results.

[0050] In this example, the model can be trained for approximately 3 million iterations using the Adam solver with default momentum parameters of β1 = 0:9, β2 = 0:999, ∈ = 0:1, and a cyclic learning rate between 0.01 and 0.004. In some examples, the error may reach its minimum at approximately 2 million iterations. In some examples, the model iterated in this way can reduce the error by 63% compared to the identity transformation. The model also produces visually appealing results. In some examples, the overall error can be further reduced by using a portion of the natural dataset for fine-tuning and the remainder for validation, but fine-tuning the entire network on natural data can lead to overfitting. In some deep learning applications, the first layers can be frozen and the last layers can be fine-tuned to prevent overfitting. This is because these models transfer weights from other models that use similar data to perform different tasks. However, in this case, the task is the same for both the natural and synthetic datasets, but the input data distributions may differ. Therefore, only the first layers are tuned as domain adaptations, while the rest of the network remains frozen. Using a portion of the natural data for domain adaptation further reduced the relative error by 2%.

[0051] While domain adaptation helps reduce error, it can also significantly reduce the strength of correction and degrade the quality of results. Regardless of the measure of misalignment tolerance error, some of the residual error on natural datasets may be due to variations other than gaze (such as shading and reflection). A significant reduction in error may result in better gaze correction, while a smaller "improvement" may be a result closer to the mean after eliminating other varying factors. Therefore, natural datasets can be used as a development set, and the error can be computed as a sanity check rather than a benchmark, while continuously and qualitatively monitoring the results. Overall, training the model solely on synthetic data yields visually better results. This is likely because the influence of perfect labels on the synthetic set outweighs the influence of a data distribution closer to the real-world use cases on the natural set in terms of importance.

[0052] In some examples, to make the distribution of synthetic data closer to real data without sacrificing label quality, generative adversarial networks can be built based on CycleGAN, first released in 2017. CycleGAN uses periodic consistent training to learn a mapping between two image sets without a one-to-one correspondence. The generator in the trained model is used to generate photorealistic eye images given synthetic samples. To further encourage similarity between the synthetic input and the natural-looking output, two additional mean absolute error (L1) losses defined between the generator's input and output are used. These additional loss functions are defined only on the luminosity channel to give the model greater flexibility to modify color while preserving the gaze direction and the overall structure of the eye and its surroundings. The default hyperparameters for CycleGAN are used for training, treating the additional L1 losses as the same as the reconstruction loss.

[0053] The trained generator can produce photorealistic images without altering the gaze in the input. For typical use cases, the results may have virtually no artifacts. Even in challenging situations with glare, glasses frames too close to the eyes, or scenes that are too dark or blurry, artifacts may be minimal.

[0054] Furthermore, using the training process described above, the model can learn to implicitly predict input gaze. For example, by calculating the average motion vector based on the output vector field, inverting its direction, and scaling its magnitude to fit the screen, such as... Figure 5 As described, this generates an estimate of the input gaze. Therefore, unlike typical multi-task learning settings where the model is trained to perform multiple tasks simultaneously, the model learns to perform two tasks and is simultaneously trained to perform only one of them. In this respect, the model may not be a fully-fledged gaze predictor, but rather an eye contact corrector that can learn the input gaze to function better. This behavior may be a byproduct of training the model to redirect the gaze without providing a redirection angle, where the input gaze angle is inferred. The inferred gaze does not involve head posture or distance from the screen and relies solely on information extracted from the eyes in isolation. Therefore, it may not be as accurate as systems using dedicated sensors or models specifically designed for gaze prediction. However, the predicted gaze may still be practical in various use cases where computational costs are a concern, as the additional costs (i.e., average computation and inversion) are negligible. For example, video conferencing applications using eye contact correction would be able to compute gaze statistics with minimal overhead. Those gaze statistics can reveal information about the effectiveness of the conversation and provide feedback to the user. In some examples, real-time gaze information will also enable hands-free interaction, such as dimming the backlight when the user is not engaged. Therefore, the gaze prediction feature of our eye contact corrector has the potential to reduce battery consumption while providing additional functionality.

[0055] Figure 2 The diagram is not intended to indicate that example process 200 will include Figure 2 All components shown. Alternatively, fewer components or... Figure 2 Example process 200 is implemented using additional components not shown in the figure (e.g., additional input image, target angle, input angle, loss, reconstructed image, output image, target image, etc.).

[0056] Figure 3 This is a flowchart illustrating an example method for generating gaze correction images. (The remaining text appears to be incomplete and possibly contains errors.) Figure 1 System 100 Figure 6 The processor 602 of the computing device 600 or Figure 7 Method 300 is implemented in a computer-readable medium 700.

[0057] In block 302, an input image including an eye is received, and the target angle is set as the center. For example, the center can be represented using coordinates (0,0). For example, the input image can be an image patch of an eye. The center can correspond to the center of the image patch.

[0058] At block 304, the target angle and the input image are sent to the trained network. The trained network can be a bidirectionally trained convolutional neural network. For example, the trained network can be a convolutional neural network trained using method 500.

[0059] At block 306, a vector field and a brightness map are received from the trained network. For example, the vector field may include motion vectors indicating where pixels will be moved to. The brightness map may include a brightness mask comprising a brightness multiplier for each pixel in the input image.

[0060] At block 308, the input image is warped based on the vector field to generate a gaze-corrected image. In some examples, the size of the input image can also be adjusted. For example, the input image can be resized to a predetermined size. As an example, the predetermined size could be 64x32 pixels.

[0061] At block 310, the local brightness of the warped image is adjusted based on the luminance map to generate an enhanced gaze correction image. For example, luminance map 122 can be used as a mask to improve the appearance of the whites of the eyes after warping the input image to redirect the gaze.

[0062] This flowchart is not intended to indicate that the blocks of example method 300 will be executed in any particular order, or that all blocks will be included in every case. Furthermore, depending on the details of the specific implementation, any number of additional blocks, not shown, may be included within example method 300. For example, method 300 may include: receiving predicted input gaze direction and magnitude from a trained network; and dimming the backlight of the device's screen in response to detecting user inactivity based on the predicted input gaze direction and magnitude. For example, the predicted user input gaze and magnitude may be outside a predetermined threshold range. Method 300 may also include: receiving predicted input gaze direction and magnitude from a trained network; and using the predicted input gaze direction and magnitude as input to control an application. For example, the predicted input gaze direction may be used in place of or as a supplement to other user inputs, such as keyboard or mouse input and other inputs.

[0063] Figure 4 This is a flowchart illustrating an example method for predicting input gaze using a bidirectionally trained network. (The remaining text appears to be a fragment and doesn't translate directly.) Figure 1 System 100 Figure 6 The computing device 600 includes a processor 602 and a gaze predictor 636 or Figure 7 Method 400 is implemented in the gaze predictor module 714 of the computer-readable medium 700.

[0064] At block 402, an input image including the eye is received and the target angle is set as the center. For example, the center can be indicated by coordinates (0,0). The center can correspond to the center of the image patch of the eye.

[0065] At block 404, the target angle and the input image are sent to the trained network. For example, the trained network could be a bidirectionally trained convolutional neural network. As an example, the network could be bidirectionally trained based on a simulated input image to generate a vector field and a brightness map, such as... Figure 5 As described in [the text].

[0066] At block 406, a vector field is received from the trained network. For example, the vector field may include motion vectors indicating where pixels will be moved to.

[0067] At block 408, the average motion vector is calculated based on the vector field. For example, the average motion vector can be calculated by averaging the motion vectors of the vector field.

[0068] At block 410, the direction of the motion vector is inverted and the magnitude of the average motion vector is scaled to fit the screen. For example, the screen could be the screen of a computing device capturing video frames. The computing device could be a laptop, tablet, or any device that supports video conferencing.

[0069] At block 412, an estimated input gaze is generated based on a scaled average motion vector. For example, the estimated input gaze can approximate the gaze of the user's eyes captured in the input image.

[0070] This process flowchart is not intended to indicate that the blocks of Example Method 400 will be executed in any particular order, or that all blocks will be included in every case. Furthermore, depending on the details of the specific implementation, any number of additional blocks, not shown, may be included within Example Method 400.

[0071] Figure 5 This is a flowchart illustrating an example method for training a network to perform image adjustment and gaze prediction. (The remaining text appears to be a fragment and doesn't translate directly.) Figure 6 Remote device 638 or Figure 7 Method 500 is implemented in a computer-readable medium 700, and method 500 can be used for training. Figure 1 System 100 or Figure 6 A bidirectionally trained CNN 632. Method 500 can be used to implement the above. Figure 2 The process 200.

[0072] At block 502, the target angle and input image are received at the first network. For example, the network can be an artificial neural network, such as a convolutional neural network. In various examples, the input image can be one of any number of simulated images generated based on sample input images. In some examples, the simulated images can come from a synthetic dataset that includes overlaid glasses of different sizes and shapes. For example, the glasses may have been overlaid on some images in the simulated dataset to increase dataset diversity.

[0073] At block 504, the output image is generated based on the target angle and the input image. For example, the target angle can be set as the center, which can be represented by coordinates (0,0).

[0074] At block 506, the weights of the model are calculated to minimize the correction loss between the output image and the target image. For example, the weights can be parameters of the model, which can be modified to change the output of one or more layers of the model.

[0075] At block 508, a reconstructed image is generated in the second network based on the input angle, the output image, and the adjusted weights. For example, the second network could be based on a model used in the first network but with the weights computed at block 506.

[0076] At block 510, the model weights are calculated to minimize the reconstruction loss between the reconstructed image and the input image. For example, the reconstruction loss could be the difference in pixel values ​​between the reconstructed image and the input image.

[0077] At block 512, the shared weights of the model are adjusted based on the calculated weights. For example, the weights of the model calculated at block 506 can be adjusted based on the weights calculated at block 510. Thus, the model to be used in the trained network can be trained bidirectionally.

[0078] This flowchart is not intended to indicate that the blocks of example method 500 will be executed in any particular order, or that all blocks will be included in every case. Furthermore, depending on the details of the specific implementation, any number of additional blocks, not shown, may be included within example method 500. For example, additional iterations of bidirectional training via blocks 502-512 may be performed until a predetermined number of iterations are detected or convergence of parameter values ​​is detected. For example, convergence of parameter values ​​may be detected when a parameter change is detected as falling below a threshold difference of a value.

[0079] Now for reference Figure 6 The block diagram is shown as an example computing device capable of performing image adjustment and gaze prediction. Computing device 600 may be, for example, a laptop computer, desktop computer, server, tablet computer, mobile device, or wearable device. In some examples, computing device 600 may be a smartwatch. Computing device 600 may include a central processing unit (CPU) 602 configured to execute stored instructions, and a memory device 604 storing instructions executable by CPU 602. CPU 602 may be coupled to memory device 604 via bus 606. Additionally, CPU 602 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Furthermore, computing device 600 may include more than one CPU 602. In some examples, CPU 602 may be a system-on-a-chip (SoC) with a multi-core processor architecture. In some examples, CPU 602 may be a dedicated digital signal processor (DSP) for image processing. Memory device 604 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. For example, memory device 604 may include dynamic random access memory (DRAM).

[0080] Memory device 604 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. For example, memory device 604 may include dynamic random access memory (DRAM).

[0081] The computing device 600 may also include a graphics processing unit (GPU) 608. As shown, the CPU 602 can be coupled to the GPU 608 via a bus 606. The GPU 608 can be configured to perform any number of graphics operations within the computing device 600. For example, the GPU 608 can be configured to render or manipulate graphical images, graphics frames, videos, etc., to be displayed to a user of the computing device 600.

[0082] Memory device 604 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. For example, memory device 604 may include dynamic random access memory (DRAM). Memory device 604 may include device driver 610, which is configured to execute instructions for generating a gaze correction image and predicting an input gaze. Device driver 610 may be software, application programs, application code, etc.

[0083] CPU 602 can also be connected via bus 606 to input / output (I / O) device interface 612, which is configured to connect computing device 600 to one or more I / O devices 614. I / O devices 614 may include, for example, a keyboard and pointing devices, wherein the pointing device may include a touchpad or a touchscreen. I / O devices 614 may be built into computing device 600 or may be devices externally connected to computing device 600. In some examples, memory 604 may be communicatively coupled to I / O devices 614 via direct memory access (DMA).

[0084] CPU 602 can also be linked to display interface 616 via bus 606, which is configured to connect computing device 600 to display device 618. Display device 618 may include a display screen, which is a built-in component of computing device 600. Display device 618 may also include a computer monitor, television, or projector connected to computing device 600, either internally or externally.

[0085] The computing device 600 also includes a storage device 620. The storage device 620 is a physical memory such as a hard disk drive, optical drive, thumb drive, drive array, solid-state drive, or any combination thereof. The storage device 620 may also include a remote storage drive.

[0086] The computing device 600 may also include a network interface controller (NIC) 622. The NIC 622 can be configured to connect the computing device 600 to a network 624 via a bus 606. The network 624 can be a wide area network (WAN), a local area network (LAN), or the Internet, etc. In some examples, the device can communicate with other devices wirelessly. For example, the device can communicate with other devices via a wireless local area network connection. In some examples, the device can communicate via... Or similar technologies to connect and communicate with other devices.

[0087] The computing device 600 further includes a camera 626. For example, the camera 626 may include one or more image sensors. In some examples, the camera 626 may include a processor for generating image patches of the eye.

[0088] The computing device 600 further includes a gaze predictor and corrector 628. For example, the gaze predictor and image corrector 628 can be used to predict an input gaze and correct an input image of the eye. The gaze predictor and image corrector 628 may include an image receiver 630, a bidirectionally trained convolutional neural network (CNN) 632, an image corrector 634, and a gaze predictor 636. In some examples, each of components 630-636 of the gaze predictor and image corrector 628 may be a microcontroller, an embedded processor, or a software module. The image receiver 630 may receive an image including the eye and a target angle centered on it and transmit that image. The bidirectionally trained CNN 632 may receive the image and target angle from the image receiver 630 and generate a vector field and a brightness map based on the image and target angle. In some examples, the bidirectionally trained CNN 632 is trained on a synthetic dataset including glasses of different sizes and shapes superimposed on some image sets to increase dataset diversity. The image corrector 634 may generate a gaze-corrected image based on the vector field and brightness map. For example, image corrector 634 can warp the input image based on a vector field to generate a gaze-corrected image. Image corrector 634 can also adjust the local brightness of the warped image based on a brightness map to generate an enhanced gaze-corrected image. Gaze predictor 636 can predict the gaze direction and magnitude based on a vector field. For example, gaze predictor 636 can calculate an average motion vector based on a vector field. In some examples, gaze predictor 636 can invert the direction of the motion vector and scale the magnitude of the average motion vector to fit the screen. Gaze predictor 636 can then generate an estimated input gaze based on the scaled average motion vector.

[0089] Computing device 600 can be coupled to remote device 638 via network 624. Remote device 638 can be used to train a bidirectionally trained CNN 632. For example, remote device 638 can remotely train the CNN and provide the bidirectionally trained CNN 632 to computing device 600.

[0090] Figure 6 The block diagram is not intended to indicate that computing device 600 will include Figure 6 All the components shown. Conversely, computing device 600 may include fewer components or Figure 6 Additional components not shown in the diagram, such as additional buffers and additional processors, may be included. Depending on the specific implementation details, computing device 600 may include... Figure 6 Any number of additional components not shown. For example, computing device 600 may also include a power-saving module for conserving power by dimming the screen in response to detecting that a user is looking away from the screen. In some examples, computing device 600 may also include a vision controller for controlling the computing device using gaze information including gaze direction and magnitude. In various examples, computing device 600 may also include a temporal filter for filtering motion vectors in a vector field that warp the input image.

[0091] Furthermore, any one of the functionalities of the image receiver 630, the bidirectionally trained CNN 632, the image corrector 634, and the gaze predictor 636 can be implemented, partially or entirely, in hardware and / or in the processor 602. For example, the functionalities can be implemented using an application-specific integrated circuit (ASIC), logic implemented in the processor 602, or any other device. Additionally, any one of the functionalities of the CPU 602 can be implemented, partially or entirely, in hardware and / or in the processor. For example, the functionalities of the gaze predictor and the image corrector 628 can be implemented using an ASIC, logic implemented in the processor, logic implemented in a dedicated graphics processing unit such as the GPU 608, or any other device.

[0092] Figure 7 This is a block diagram illustrating a computer-readable medium 700 storing code for training a network to perform image adjustment and gaze prediction. The computer-readable medium 700 can be accessed by a processor 702 via a computer bus 704. Furthermore, the computer-readable medium 700 may include code configured to instruct the processor 702 to perform the methods described herein. In some embodiments, the computer-readable medium 700 may be a non-transitory computer-readable medium. In some examples, the computer-readable medium 700 may be a storage medium.

[0093] The various software components discussed in this article can be stored on one or more computer-readable media 700, such as Figure 7 As indicated in [the document]. For example, trainer module 706 can be configured to bidirectionally train the network to generate vector field and brightness maps. For example, trainer module 706 can be configured to bidirectionally train the network based on a simulated input image to generate vector field and brightness maps. For example, the simulated input image can include superimposed glasses of different sizes and shapes. As an example, trainer module 706 can be configured to perform [the following]. Figure 5 Method 500. Receiver module 708 can be configured to receive an image including the eye and a target angle set to center and send the image to a trained network. CNN module 710 can be configured to generate a vector field and a brightness map based on the image and the target angle. Image corrector module 712 is configured to generate a gaze-corrected image based on the vector field and brightness map. For example, image corrector module 712 can be configured to warp the input image based on the vector field to generate a gaze-corrected image. In some examples, image corrector module 712 can be configured to resize the input image to a predetermined size. For example, image corrector module 712 can be configured to adjust the local brightness of the warped image based on the brightness map to generate an enhanced gaze-corrected image. Gaze predictor module 714 can be configured to predict the gaze direction and magnitude based on the vector field. For example, gaze predictor module 714 can be configured to calculate an average motion vector based on the vector field. Gaze predictor module 714 can be configured to invert the direction of the motion vector and scale the magnitude of the average motion vector to fit the screen. The gaze predictor module 714 can be configured to generate an estimated input gaze based on a scaled average motion vector.

[0094] Figure 7 The block diagram is not intended to indicate that the computer-readable medium 700 will include Figure 7 All components shown. Further, depending on the details of the specific implementation, the computer-readable medium 700 may include... Figure 7 Any number of additional components not shown. For example, the computer-readable medium 700 may also include a power-saving module for conserving power by dimming the screen as the user's gaze is removed. In some examples, the computer-readable medium 700 may also include a vision controller for controlling the computing device using gaze information including gaze direction and magnitude.

[0095] Example

[0096] Example 1 is an apparatus for correcting gaze in an image. The apparatus includes an image receiver for receiving an image including an eye and a target angle set to a center. The apparatus also includes a bidirectionally trained convolutional neural network (CNN) for receiving the image and the target angle from the image receiver and generating a vector field and a luminance map based on the image and the target angle. The apparatus further includes an image corrector for generating a gaze-corrected image based on the vector field and the luminance map.

[0097] Example 2 includes the apparatus according to Example 1, including or excluding optional features. In this example, the image corrector warps the input image based on the vector field to generate the gaze-corrected image.

[0098] Example 3 includes the apparatus according to any one of Examples 1 to 2, including or excluding optional features. In this example, the image corrector adjusts the local brightness of the warped image based on the brightness map to generate an enhanced gaze-corrected image.

[0099] Example 4 includes the apparatus according to any one of Examples 1 to 3, including or excluding optional features. In this example, the apparatus includes a gaze predictor for predicting the gaze direction and magnitude based on the vector field.

[0100] Example 5 includes the apparatus according to any one of Examples 1 to 4, including or excluding optional features. In this example, the apparatus includes a gaze predictor for calculating an average motion vector based on the vector field.

[0101] Example 6 includes the apparatus according to any one of Examples 1 to 5, including or excluding optional features. In this example, the apparatus includes a gaze predictor for inverting the direction of a motion vector and scaling the magnitude of the average motion vector to fit the screen of the apparatus.

[0102] Example 7 includes an apparatus according to any one of Examples 1 to 6, including or excluding optional features. In this example, the apparatus includes a gaze predictor for generating an estimated input gaze based on a scaled average motion vector.

[0103] Example 8 includes the apparatus according to any one of Examples 1 to 7, including or excluding optional features. In this example, the apparatus includes a temporal filter for filtering the motion vectors in the vector field that warp the input image in time.

[0104] Example 9 includes the apparatus according to any one of Examples 1 to 8, including or excluding optional features. In this example, the apparatus includes a power-saving module for conserving power by dimming the screen in response to detecting that a user is looking away from the screen.

[0105] Example 10 includes an apparatus according to any one of Examples 1 to 9, including or excluding optional features. In this example, the bidirectionally trained CNN is trained on a synthetic dataset that includes glasses of different sizes and shapes superimposed on some image set to increase the diversity of the dataset.

[0106] Example 11 is a method for correcting gaze in an image. The method includes receiving, via a processor, an image including an eye and a target angle set to a center. The method further includes sending the target angle and the input image via the processor to a trained network. The method further includes receiving a vector field and a luminance map from the trained network via the processor. The method further includes warping the input image via the processor based on the vector field to generate a gaze-corrected image.

[0107] Example 12 includes the method according to Example 11, including or excluding optional features. In this example, the method includes bidirectionally training the network based on a simulated input image to generate a vector field and a brightness map.

[0108] Example 13 includes the method described in any of Examples 11 to 12, including or excluding optional features. In this example, the method includes bidirectionally training the network based on simulated input images of superimposed glasses of different sizes and shapes.

[0109] Example 14 includes the method according to any one of Examples 11 to 13, including or excluding optional features. In this example, the method includes adjusting the local brightness of the warped image based on a luminance map via the processor to generate an enhanced gaze-corrected image.

[0110] Example 15 includes the method according to any one of Examples 11 to 14, including or excluding optional features. In this example, the method includes predicting the gaze direction and magnitude based on the vector field.

[0111] Example 16 includes the method according to any one of Examples 11 to 15, including or excluding optional features. In this example, the method includes: calculating an average motion vector based on the vector field; inverting the direction of the motion vector; scaling the magnitude of the average motion vector to fit the screen; and generating an estimated input gaze based on the scaled average motion vector.

[0112] Example 17 includes the method according to any one of Examples 11 to 16, including or excluding optional features. In this example, warping the input image further includes resizing the input image to a predetermined size.

[0113] Example 18 includes the method according to any one of Examples 11 to 17, including or excluding optional features. In this example, the method includes applying a temporal filter to filter the motion vectors in the vector field that are used to warp the input image in time.

[0114] Example 19 includes the method according to any one of Examples 11 to 18, including or excluding optional features. In this example, the method includes: receiving predicted input gaze direction and magnitude from the trained network; and dimming the backlight of the device's screen in response to detecting user inactivity based on the predicted input gaze direction and magnitude.

[0115] Example 20 includes the method according to any one of Examples 11 to 19, including or excluding optional features. In this example, the method includes: receiving predicted input gaze direction and magnitude from the trained network; and using the predicted input gaze direction and magnitude as input to control an application.

[0116] Example 21 is at least one computer-readable medium for correcting gaze in an image, wherein instructions are stored to instruct a processor to receive an image including an eye and a target angle set to center. The computer-readable medium also includes instructions to instruct the processor to generate a vector field and a luminance map based on the image and the target angle. The computer-readable medium further includes instructions to instruct the processor to generate a gaze-corrected image based on the vector field and the luminance map.

[0117] Example 22 includes a computer-readable medium according to Example 21, including or excluding optional features. In this example, the computer-readable medium includes instructions for bidirectionally training the network based on a simulated input image to generate a vector field and a brightness map.

[0118] Example 23 includes a computer-readable medium according to any one of Examples 21 to 22, including or excluding optional features. In this example, the computer-readable medium includes instructions for warping the input image based on the vector field to generate the gaze-corrected image.

[0119] Example 24 includes a computer-readable medium according to any one of Examples 21 to 23, including or excluding optional features. In this example, the computer-readable medium includes instructions for adjusting the local brightness of the warped image based on the brightness map to generate an enhanced gaze-corrected image.

[0120] Example 25 includes a computer-readable medium according to any one of Examples 21 to 24, including or excluding optional features. In this example, the computer-readable medium includes instructions for predicting the direction and magnitude of the gaze based on the vector field.

[0121] Example 26 includes a computer-readable medium according to any one of Examples 21 to 25, including or excluding optional features. In this example, the computer-readable medium includes instructions for performing the following operations: calculating an average motion vector based on the vector field, inverting the direction of the motion vector, scaling the magnitude of the average motion vector to fit the screen, and generating an estimated input gaze based on the scaled average motion vector.

[0122] Example 27 includes a computer-readable medium according to any one of Examples 21 to 26, including or excluding optional features. In this example, the computer-readable medium includes instructions for resizing the input image to a predetermined size.

[0123] Example 28 includes a computer-readable medium according to any one of Examples 21 to 27, including or excluding optional features. In this example, the computer-readable medium includes instructions for applying a temporal filter to filter the motion vectors in the vector field that warp the input image in time.

[0124] Example 29 includes a computer-readable medium according to any one of Examples 21 to 28, including or excluding optional features. In this example, the computer-readable medium includes instructions for performing the following operations: receiving a predicted input gaze direction and magnitude from the trained network and dimming the backlight of the device's screen in response to detecting user inactivity based on the predicted input gaze direction and magnitude.

[0125] Example 30 includes a computer-readable medium according to any one of Examples 21 to 29, including or excluding optional features. In this example, the computer-readable medium includes instructions for receiving predicted input gaze direction and magnitude from the trained network and using the predicted input gaze direction and magnitude as input to control an application.

[0126] Example 31 is a system for correcting gaze in an image. The system includes an image receiver for receiving an image including the eye and a target angle set to center. The system also includes a bidirectionally trained convolutional neural network (CNN) for receiving the image and the target angle from the image receiver and generating a vector field and a brightness map based on the image and the target angle. The system further includes an image corrector for generating a gaze-corrected image based on the vector field and the brightness map.

[0127] Example 32 includes the system according to Example 31, including or excluding optional features. In this example, the image corrector warps the input image based on the vector field to generate the gaze-corrected image.

[0128] Example 33 includes the system according to any one of Examples 31 to 32, including or excluding optional features. In this example, the image corrector adjusts the local brightness of the warped image based on the brightness map to generate an enhanced gaze-corrected image.

[0129] Example 34 includes a system according to any one of Examples 31 to 33, including or excluding optional features. In this example, the system includes a gaze predictor for predicting gaze direction and magnitude based on the vector field.

[0130] Example 35 includes a system according to any one of Examples 31 to 34, including or excluding optional features. In this example, the system includes a gaze predictor for calculating an average motion vector based on the vector field.

[0131] Example 36 includes a system according to any one of Examples 31 to 35, including or excluding optional features. In this example, the system includes a gaze predictor for inverting the direction of a motion vector and scaling the magnitude of the average motion vector to fit the device's screen.

[0132] Example 37 includes a system according to any one of Examples 31 to 36, including or excluding optional features. In this example, the system includes a gaze predictor for generating an estimated input gaze based on a scaled average motion vector.

[0133] Example 38 includes a system according to any one of Examples 31 to 37, including or excluding optional features. In this example, the system includes a temporal filter for filtering the motion vectors in the vector field that warp the input image.

[0134] Example 39 includes a system according to any one of Examples 31 to 38, including or excluding optional features. In this example, the system includes a power-saving module for conserving power by dimming the screen in response to detecting that a user is looking away from the screen.

[0135] Example 40 includes the system described in any of Examples 31 to 39, including or excluding optional features. In this example, the bidirectionally trained CNN is trained on a synthetic dataset that includes glasses of different sizes and shapes superimposed on some image set to increase the diversity of the dataset.

[0136] Example 41 is a system for correcting gaze in an image. The system includes means for receiving an image including the eye and a target angle set as a center. The system also includes means for generating a vector field and a brightness map based on the image and the target angle. The system further includes means for generating a gaze-corrected image based on the vector field and the brightness map.

[0137] Example 42 includes the system according to Example 41, including or excluding optional features. In this example, the means for generating the gaze-corrected image is used to warp the input image based on the vector field to generate the gaze-corrected image.

[0138] Example 43 includes the system according to any one of Examples 41 to 42, including or excluding optional features. In this example, the means for generating the gaze-corrected image is used to adjust the local brightness of the warped image based on the brightness map to generate an enhanced gaze-corrected image.

[0139] Example 44 includes a system according to any one of Examples 41 to 43, including or excluding optional features. In this example, the system includes means for predicting the gaze direction and magnitude based on the vector field.

[0140] Example 45 includes a system according to any one of Examples 41 to 44, including or excluding optional features. In this example, the system includes means for calculating an average motion vector based on the vector field.

[0141] Example 46 includes a system according to any one of Examples 41 to 45, including or excluding optional features. In this example, the system includes means for inverting the direction of a motion vector and scaling the magnitude of the average motion vector to fit the device's screen.

[0142] Example 47 includes a system according to any one of Examples 41 to 46, including or excluding optional features. In this example, the system includes means for generating an estimated input gaze based on a scaled average motion vector.

[0143] Example 48 includes a system according to any one of Examples 41 to 47, including or excluding optional features. In this example, the system includes means for temporally filtering the motion vectors in the vector field that warp the input image.

[0144] Example 49 includes a system according to any one of Examples 41 to 48, including or excluding optional features. In this example, the system includes means for saving power by dimming the screen in response to detecting that a user is looking away from the screen.

[0145] Example 50 includes the system according to any one of Examples 41 to 49, including or excluding optional features. In this example, the means for generating the vector field and the brightness map includes a bidirectionally trained CNN trained on a synthetic dataset comprising glasses of different sizes and shapes superimposed on some image set to increase dataset diversity.

[0146] Not all components, features, structures, characteristics, etc., described and illustrated herein need to be included in one or more specific aspects. If the specification refers to a component, feature, structure, or characteristic as "may," "may," "may," or "can" be included, then, for example, inclusion of that particular component, feature, structure, or characteristic is not required. If the specification or claims refer to an element as "a" or "an," it does not mean that only one element exists. If the specification or claims refer to an "additional" element, it does not exclude the existence of more than one additional element.

[0147] It should be noted that although some aspects have been described with reference to specific embodiments, other embodiments are possible according to some aspects. Furthermore, the arrangement and / or sequence of the circuit elements or other features illustrated in the figures and / or described herein are not required to be in the specific manner shown and described. Many other arrangements are also possible according to some aspects.

[0148] In each system shown in the figures, elements may, in some cases, have the same or different reference numerals to indicate that the represented elements may be different and / or similar. However, the elements can be flexible enough to have different embodiments and work with some or all of the systems shown or described herein. The various elements shown in the figures may be the same or different. Which element is referred to as the first element and which as the second element is arbitrary.

[0149] It should be understood that the details in the foregoing examples can be used anywhere in one or more aspects. For example, all the optional features of the computing device described above can also be implemented with respect to any of the methods described herein or any of the computer-readable media. Furthermore, although flowcharts and / or state diagrams have been used herein to describe aspects, these techniques are not limited to those diagrams or their corresponding descriptions herein. For example, the process does not necessarily have to move through each illustrated box or state or in the exact same order as illustrated and described herein.

[0150] The technology of this application is not limited to the specific details listed herein. In fact, those skilled in the art who will benefit from this disclosure will appreciate that many other variations can be made within the scope of the technology of this application based on the foregoing description and drawings. Therefore, the scope of the technology of this application is defined by the appended claims (including any modifications thereof).

Claims

1. At least one memory including instructions that cause at least one programmable circuit to perform at least the following operations: The neural network is trained to estimate gaze based on multiple synthetic datasets, the synthetic datasets including synthetic images, some of which include eyes generated in a synthetic manner. The trained neural network is improved based on at least a first portion of a real image dataset, the real image dataset including real images, some of which include the eyes of human subjects; The modified neural network is validated based on at least a second portion of the real image dataset. as well as The neural network is provided after being trained, refined, and validated to estimate the gaze associated with an input image from a camera.

2. The at least one memory according to claim 1, wherein the neural network comprises a convolutional neural network.

3. The at least one memory according to claim 1, wherein one or more of the at least one programmable circuitry includes a graphics processing unit (GPU).

4. The at least one memory according to claim 1, wherein the instructions are configured to cause one or more of the at least one programmable circuit to: Perform blink detection on the input image; and The gaze associated with the input image is estimated based on the blink detection.

5. The at least one memory of claim 1, wherein the instructions are configured to cause one or more of the at least one programmable circuit to train the neural network based on the plurality of synthetic datasets to output an estimated gaze direction.

6. The at least one memory of claim 5, wherein the instructions are configured to cause one or more of the at least one programmable circuit to train the neural network based on the plurality of synthetic datasets to output an estimated gaze magnitude.

7. The at least one memory of claim 1, wherein the instructions are configured to cause one or more of the at least one programmable circuitry to execute the trained, refined, and verified neural network to output an estimated gaze direction associated with the input image.

8. A method comprising: The neural network is trained to estimate gaze based on multiple synthetic datasets, the synthetic datasets including synthetic images, some of which include eyes generated in a synthetic manner. The trained neural network is improved based on at least a first portion of a real image dataset by at least one programmable circuit programmed based on at least one instruction, the real image dataset comprising real images, some of which include the eyes of human subjects. The modified neural network is verified by one or more of the at least one programmable circuits based on at least a second portion of the real image dataset; as well as The neural network is provided after being trained, refined, and validated to estimate the gaze associated with an input image from a camera.

9. The method of claim 8, wherein the neural network comprises a convolutional neural network.

10. The method of claim 8, wherein one or more of the at least one programmable circuit includes a graphics processing unit (GPU).

11. The method of claim 8, comprising: Perform blink detection on the input image; as well as The gaze associated with the input image is estimated based on the blink detection.

12. The method of claim 8, wherein training the neural network comprises training the neural network based on the plurality of synthetic datasets to output an estimated gaze direction and an estimated gaze magnitude.

13. The method of claim 8, further comprising executing the trained, refined, and validated neural network to output an estimated gaze direction associated with the input image.

14. An apparatus comprising: Interface circuit; Computer-readable instructions; as well as At least one programmable circuit, which is programmed based on the computer-readable instructions to perform the method according to any one of claims 8 to 13.

15. A computing device comprising means for performing the method according to any one of claims 8 to 13.

16. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 8 to 13.

Images