Method for correcting eye images using machine learning and method for machine learning

Abstract

The present invention relates to a method for correcting eye images using machine learning and a method for machine learning. The present invention relates to automation and computing technology, i.e., to the field of processing image and video data, i.e., to correcting the eye images of interlocutors in video chats and video conferences for the purpose of redirecting gaze. A method for correcting eye images, wherein the method obtains at least one frame of a person's face, then determines the position of the person's eyes in the image, and forms two rectangular areas that tightly surround the eyes, and finally replaces the color components of each pixel in the eye area with the color components of the pixels shifted according to the prediction of a machine learning predictor. The technical effect of the present invention is that the correction accuracy of eye images for the purpose of redirecting gaze is improved, wherein the resources required for the process of processing video images are reduced.

Description

[0001] This application is a divisional application of the PCT international application PCT / RU2016 / 000118 filed on March 3, 2016, which entered the Chinese national phase on November 17, 2017, and whose Chinese patent application number is 201680028919.9 and whose invention name is "Method for correcting eye images using machine learning and machine learning method". Technical Field

[0002] The present group of inventions relates to automation and computing technology, in general to the field of processing image and video data, namely to correcting images of the eyes of interlocutors during video chats, video conferences. Background Art

[0003] Nowadays, video conferencing between business partners and video calls to relatives abroad have become a daily occurrence. The choice of specific software and hardware is hampered by the complexity associated with the lack of visual contact between the parties. This problem has long attracted researchers and engineers and is caused by the inevitable difference between the position of the camera capturing the video image and the image of the face on the screen.

[0004] The most successful currently known solutions require additional equipment besides the webcam, such as: a semi-transparent mirror / screen, a stereo camera or an RGB-D camera.

[0005] Patent application WO2011148366 “Method and system for correcting gaze offset” published on December 1, 2011 by the applicant Ramot At Tel-Aviv University Ltd. is known in the prior art. The method comprises the following steps: processing an image so as to extract the position of at least one eye on the image; processing the image so as to replace the image data associated with each position of each eye with replacement data; and sending the corrected image to a display device. The disadvantages of the given method are, firstly, the necessity to pre-record a set of image data of the gaze towards the camera for each participant of the video conference before the start of the video conference and, secondly, the unnatural fixation of the gaze direction during the video conference.

[0006] Also known from the prior art is the article “Gaze Correction with a Single Webcam” by authors D. Giger, JC Bazin, a C. Kuster, T. Popa, M. Gross, published in Proceedings of IEEE ICME 2014 (Chengdu, China, July 14-18, 2014). The specified method comprises: determining facial feature points and matching a geometric model of a human head according to the determined facial features, projecting the image onto the texture of the adjusted model, rotating the model to a certain angle and projecting the rotated model into the image, matching the model image with the source image. The limitations of the given method are: overall face scale deformation, the texture of the masked part of the head (e.g., the chin) must be recorded in advance, and a graphics accelerator is required to achieve the productivity necessary for real-time system operation. Summary of the invention

[0007] A given group of inventions has as its object the correction of the images of the eyes of the interlocutors during a video conference using a device comprising only one video recording tool (video camera).

[0008] A technical effect of the given invention is that the correction accuracy of the eye image is improved while reducing the resources required for the process of processing the video frames.

[0009] The technical effect is achieved due to a machine learning method of a predictor (for correcting gaze direction in an image), wherein the method first obtains a plurality of image pairs, each pair containing images of the same person, but the plurality of image pairs differ only in gaze direction, then determines the position of the eyes in each image pair, then learns a predictor, generates an adjustment displacement vector field, so that for each image pair, the color component in each pixel of the first image in the pair is replaced by the color component of another pixel of the first image in the pair that is shifted according to the prediction of the predictor, resulting in an image that is as similar as possible to the second image in the pair, and saves the predictor as a result.

[0010] The present invention provides a machine learning method for correcting a predictor of gaze orientation in an image, characterized in that:

[0011] Obtain a plurality of image pairs, each pair comprising an image of the same person, wherein the plurality of image pairs differ only in gaze direction;

[0012] Determine the position of the eye in each image pair;

[0013] learning the predictor to generate an adjustment displacement vector field such that, for each image pair, replacing the color component in each pixel of the first image of the pair with the color component of another pixel of the first image of the pair shifted according to the prediction of the predictor results in an image that is as similar as possible to the second image of the pair;

[0014] • Save the predictor.

[0015] The predictor of the displacement field can take the following form:

[0016] - Single-layer or multi-layer neural networks.

[0017] - At least one decision tree or an ensemble of decision trees.

[0018] - Generate a predictor that depends only on a fixed displacement vector of pixel positions of feature points related to the eye.

[0019] - Any other form of predictor known in machine learning.

[0020] Centers the eye area of ​​the image to a preset pixel size.

[0021] The predictor is stored in an information medium including a hard disk or solid state drive, or flash memory, or optical disk, or hybrid drive, or random access memory (RAM), or a remote computer system, or a remote data storage.

[0022] Furthermore, the specified technical effect is achieved using a method for correcting the position of eyes in an eye image, wherein the method first loads a predictor, then obtains at least one frame for a person's face, then determines the position of the person's eyes in the image and forms two rectangular regions that tightly surround the eyes, and as a result replaces the color component of each pixel in the eye region with the color component of the pixel shifted according to the prediction of the predictor.

[0023] The present invention provides a method for correcting an eye image, characterized in that:

[0024] Loading predictor;

[0025] Get at least one frame of the person’s face;

[0026] Determine the position of the person's eyes in the image and form two rectangular regions that closely surround the eyes;

[0027] - Replacing the color components of each pixel in the eye region with the color components of the pixel shifted according to the machine learning based prediction of the predictor.

[0028] The predictor is trained according to the above criteria (to minimize the difference between the actual image and the obtained image for the second image in each training pair).

[0029] The predictor is loaded from an information medium, which may include a hard disk or solid state drive, or a flash memory, or an optical disk, or a hybrid drive, or a random access memory (RAM), or a remote computer system, or a remote data storage.

[0030] The predictor is a single-layer or multi-layer neural network.

[0031] A predictor is at least one decision tree or an ensemble of decision trees.

[0032] The predictor generates a fixed displacement vector that depends only on the pixel position of the characteristic point related to the eye.

[0033] After training the predictor applied to each pixel in the eye region, the following requirements should be met:

[0034] a. When applied to a pixel, the predictor shall generate an adjustment displacement vector that is used to replace the color components of a given pixel with the color components of another pixel determined by the given displacement vector;

[0035] b. The predictor is trained on multiple image pairs, where one image in a pair contains the original image of the face before the eyes are adjusted, and the other image contains an image of the person with the eyes gazed in a different direction.

[0036] The predictor applied to each pixel in the eye region satisfies the following requirements:

[0037] a. being applied to a pixel, the predictor shall generate an adjustment displacement vector for replacing the color components of the given pixel with the color components of the replacement pixel determined by the given displacement vector;

[0038] b. The predictor is trained on multiple image pairs, where one image in a pair contains an initial image of the face before the position of the eyes is adjusted, and the other image contains an image of the person with the desired eye position.

[0039] The displacement vectors predicted by the predictor are scaled according to the ratio of the size of the eye in the resized image to the training image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1Shows real-time correction of the interlocutor's eye image. On the left is an input frame with the gaze directed ten degrees below the camera. In the middle is an input frame with the gaze directed ten degrees higher than in the left frame (in this case, toward the camera). The right frame is an example of the effect of the invention, in which the gaze direction of the interlocutor in the image is adjusted upwards by ten degrees relative to the input image (shown on the left).

[0041] Figure 2 Pixel processing is shown for the case where the predictor is a decision tree.

[0042] Figure 3 Block diagram showing a machine learning approach to the predictor for correcting eye images.

[0043] Figure 4 A block diagram showing a method for correcting an eye image. DETAILED DESCRIPTION

[0044] Concepts and definitions necessary for the detailed disclosure of the present invention will be described below.

[0045] A predictor is a program or hardware implementation for comparison with a given sample of some object (e.g., a class label, a scalar or vector value), which depends on a considerable number of parameters that are trained by means of a machine learning process on a training set. Decision trees, neural networks, ensembles of decision trees may be given as embodiments of predictor architectures. Depending on the predictor, the parameters may include, for example, weights of a neural network, thresholds in a decision tree, etc.

[0046] The internal state of a predictor is the set of all parameter values ​​of the predictor that affect its predictions.

[0047] A decision tree is a method of representing rules in a hierarchical sequential structure, where a unique node (leaves) gives the solution for each object.

[0048] Color components are three or four values ​​in a numeric tuple that defines a color model for describing color representation.

[0049] There are many various types of color models, but computer graphics use the following color spaces as a rule: CMY, CMYK, CMYK256, RGB, HSB, HLS, L*a*b, YIQ, Grayscale.

[0050] For example, in the most well-known color model RGB, a tuple contains three color components. For example, (255, 64, 23) is a color containing a strong red component, a smaller amount of green, and an even smaller amount of blue.

[0051] A given invention can be implemented on a computer in the form of a system or on a machine-readable medium containing instructions for executing the aforementioned method.

[0052] The machine learning method for correcting the predictor of gaze orientation in an image comprises the following steps:

[0053] A plurality of image pairs are obtained, each pair contains images of the same person, and the image pairs differ only in gaze directions;

[0054] Multiple image pairs 302 ( Figure 3 ) from video camera 301( Figure 3 In order to provide the possibility of machine learning, a predetermined set of image pairs (I j ,O j ): Each image pair represents the same person's face with the same head position and equal visualization conditions, and the only difference is the person's gaze direction ( Figure 1 ). The difference in gaze direction is equal in all learning pairs. For the correct operation of the method, the given difference in gaze direction should be between 5 and 20 degrees.

[0055] defining the position of the eyes for each image pair;

[0056] The controlled gradient descent algorithm (a description of which can be found in reference [1]) can be used, for example, to determine the position of the eye. Methods based on the sequential application of decision trees can also be used. Various alternatives for implementing the specified method are described in the articles [2] and [3].

[0057] In addition, make the images equal in size. For each image pair matched to the right eye, focus on the image I j The eye in this figure matches the points (f1, g1), (f2, g2)…(f N ,g N ). In [1], the number of points N is 6. Obviously, it is not a fixed parameter. In addition, the boundary points of the axis-aligned rectangle B' for each eye are determined, and the characteristic radius Δ is defined as Furthermore, consider a rectangle B with a center at the same position as B', and a width W and height H proportional to Δ (i.e., for some constants α and β, W = α*Δ, H = β*Δ). The constants α and β are arbitrarily chosen in the range of 1 to 20. In the experiments performed, the values ​​α and β were 2.0 and 1.6, respectively. Thus, rectangle B is covariant with respect to scale and eye position, and has a side length ratio α:β. Image I is cropped according to rectangle B. j and O j, and scale these images by R / Δ times so that the characteristic radius of the eye in the image becomes constant. After regular reflection relative to the vertical axis, the left eye image is processed similarly. As a result, a plurality of cropped and scaled image pairs 303 ( Figure 3 ).

[0058] learning a predictor to generate adjustment displacement vectors such that, for each image pair, replacing a color component in each pixel of a first image in the pair with a color component of another pixel of the first image in the pair shifted according to a prediction of the predictor results in an image that is as similar as possible to a second image in the pair;

[0059] In each pixel (x, y), the value O j (x, y) is converted to I by means of the operation O(x, y) = I(x + u(x, y), y + v(x, y)). j Thus, each pixel (x, y) within the bounding rectangle B is assigned a tuple S = [(x, y), I, [f i ,g i ],O(x,y)], the tuple includes the (x,y) pixel position, the input image I, the point on the eye contour [f i ,g i ] and the color of the pixel in the output image O(x,y). Then, the decision tree is learned based on the education tuples (learning samples).

[0060] Each learning sample does not include a displacement vector (u(x,y), v(x,y)), but only includes the desired color O(x,y), while the same color component can be obtained with the help of various offsets.

[0061] The eye image is then processed by applying a machine trained predictor 304 ( Figure 3 ) is adjusted, and the predictor P is determined by the following two properties:

[0062] (1) When the predictor P processes a pixel, a displacement vector is predicted, which is used to replace the intensity (color component) of the pixel with the intensity (color component) of the pixel after the displacement according to the correction process.

[0063] (2) During training, for the plurality of image pairs (im1, im2), applying P should result in the transformation of im1 into im2, where im1 contains the input image and im2 contains the desired image.

[0064] Save the predictor;

[0065] Depending on the type of predictor (decision tree, decision tree ensemble, neural network), the parameters of a given predictor are stored on the information medium 305 ( Figure 3 ), the information medium 305 is a hard disk or a solid state drive, or an optical drive, or a flash memory. In addition, the predictor can be stored in a random access memory (RAM).

[0066] For example, if a decision tree is used as a predictor, the parameters could be: the feature being tested, the threshold τ, the connections between nodes, the number of leaves, the depth of the tree, the values ​​of the nodes, the entire topology of the tree.

[0067] Methods for correcting eye images include:

[0068] Load the predictor;

[0069] Depending on the type of predictor 404 (decision tree, decision tree ensemble, neural network), the parameters of a given predictor are obtained from the information medium 403 ( Figure 4 ) is loaded, the information medium 403 may be a hard disk or a solid state drive, or an optical drive, or a flash memory. In addition, the predictor may be loaded from a random access memory (RAM).

[0070] For example, if a decision tree is used as a predictor, the parameters could be: the feature being tested, the threshold τ, the connections between nodes, the number of leaves, the depth of the tree, the values ​​of the nodes, the entire topology of the tree.

[0071] obtaining at least one frame of a person's face;

[0072] The image can be obtained in real time from a video data source 401 ( Figure 4 ), or in the form of streaming video from storage, from a local video server or a central server. To send streaming video, standard protocols RTSP (Real Time Streaming Protocol), RTMP (Real Time Messaging Protocol), HLS (HTTP Live Streaming) and DASH (Dynamic Adaptive Streaming over HTTP) can be used. Thus, the speed and quality of the sent video data can be automatically adapted to the device communication channel.

[0073] The video data may be sent in compressed form, for example with the aid of an encoder H.264, VP8, MJPEG, JPEG, JPEG2000.

[0074] The video data can be sent in the form of a separate file. Thus, standard containers (e.g., WebM, OGV, MKV, MP4, TS, JPG, and others) can be used.

[0075] Video data can be sent over a wireless network, such as GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), LTE (Long Term Evolution), Wi-Fi (Wireless Fidelity). In some implementations of the present invention, obtaining and / or sending data is performed using the above-mentioned several technologies or receiving / sending technologies.

[0076] Define the location of the human eye in the image and two rectangular areas that tightly surround the eye;

[0077] Each eye focuses on the points (f1, g1), (f2, g2)… (f N ,g N ). In addition, the axis-aligned rectangular boundary point B' of each eye is determined, and the characteristic radius Δ is defined as Furthermore, consider a rectangle B with a center at the same position as B', and a width W and height H proportional to Δ (i.e., for some constants α and β, W = α*Δ, H = β*Δ). Thus, rectangle B is covariant with respect to scale and eye position, and has a side length ratio α:β.

[0078] replacing a color component of each pixel in the eye region with a color component of the pixel shifted according to a prediction of the predictor;

[0079] In determining the bounding rectangle 405 around the eye ( Figure 4 ), the color components of the pixels are changed for redirecting the gaze. The given steps of the method are obtained by using the predictor 404 ( Figure 4 ) is achieved by machine learning. As a result of matching the pixels in the rectangle of the input image and the images in the learning set, a two-dimensional (2-D) displacement vector (u(x,y), v(x,y)) is obtained at the (x,y) pixel coordinates. The final value of the pixel O(x,y) in the output image is calculated by the formula O(x,y)=I(x+u(x,y),y+v(x,y)). Thus, for the bounding rectangle 405 ( Figure 4 ) obtains a set of two-dimensional offset vectors 406 ( Figure 4 ).

[0080] In more detail ( Figure 2), the predictor (in this case a decision tree) matches each pixel (x,y) with the learning data, thereby determining the offset. As a pixel is passed through a given tree, it sequentially undergoes two types of tests. In each non-leaf node of the randomized decision tree, the type of test applied to the pixel is determined. The first type of test (the appearance test) is determined by the offset (dx,dy) in one of the channels {R;G;B} and a threshold τ, and within the frame of the test, the difference between the two values ​​of the pixel in that color channel is compared to the threshold as follows.

[0081]

[0082] The second type of test (the test for the position of a pixel relative to a feature point) is determined by the number of feature points i∈{1,...N} and the threshold τ, and within the frame of the test, xf i or yg i is compared with a threshold τ as follows:

[0083] or

[0084] Each leaf contains an unnormalized offset error distribution, which is referenced as a compatibility map. In each pixel of the compatibility map, the difference between the true color components of the pixel in the output image and the color components of the pixel in the input image shifted by (u,v) is summarized. If the difference is small enough, the pixel in the output image can be approximated by the pixel in the input image shifted by (u,v).

[0085] The total number of tests applied depends on the depth of the decision tree, and the type of test currently applied depends on the type of node we are at at the moment.

[0086] To improve the correction accuracy, a set of several independently learned decision trees can be applied. For a given pixel, each tree predicts a compatibility map (which is recorded in the corresponding leaf of the tree). The compatibility maps from different trees are then added together, which allows a more accurate estimate of the typical compatibility map for a given type of pixel. For the aggregated compatibility map, the most compatible shift for a given type of pixel is selected (as the position of the minimum on the map). This value (u, v) is used to perform the operation when restoring O(x, y) = I(x + u(x, y), y + v(x, y)).

[0087] This detailed description is written by presenting various embodiments without limiting and exhaustive features. At the same time, it is obvious to those skilled in the art that various substitutions, modifications or combinations of any embodiment disclosed herein (also partially disclosed) can be reproduced within the scope of the present invention. Thus, it is intended and clear that the present description of the present invention includes other alternatives whose essence is not stated in a clearly expressed form here. With reference to the embodiments given here and without limiting features, such embodiments can be obtained in the result of, for example, any combination, modification or conversion of any action, component, device, property, aspect, performance, constraint, etc.

[0088] References

[0089] 1. X. Xiong and F. De la Torre. Supervised descent method and its applications to face alignment. Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference, pp. 532-539, 2013 IEEE.

[0090] 2. B.A. Smith, Q. Yin, S.K. Feiner, and S.K. Nayar. Gaze locking: passive eye contact detection for human-object interaction. Proceedings of the 26 th annual ACM symposium on User interface software and technology, pages 271-280, 2013 ACM.

[0091] 3. S. Ren, X. Cao, Y. Wei, and JS0001. Face alignment at 3000fps via regressing local binary feature. CVPR, pp. 1685-1692, 2014.

Claims

1. A method for image enhancement on a computing device, the method comprising: receiving a digital input image depicting a human face; generating a gaze adjusted image from the digital input image by changing the gaze direction of the eyes on the human face based on replacing a color component of each pixel in the eyes in the digital input image with a color component of another pixel in the digital input image via a gaze adjustment machine learning model, wherein the gaze adjustment machine learning model comprises a predictor based on a plurality of learning image pairs that differ in gaze direction; and The gaze-adjusted image is output.

2. The method of claim 1, wherein the predictor is a single-layer or multi-layer neural network.

3. The method of claim 1, wherein the predictor is at least one decision tree or an ensemble of decision trees.

4. The method of claim 1, wherein the gaze directions between the learning image pairs differ by an angle in the range of 5 to 20 degrees.

5. The method of claim 1, wherein the predictor generates a fixed displacement vector that depends on the pixel position of a plurality of characteristic points related to the eye.

6. The method of claim 1, wherein the predictor is based on a plurality of learning image pairs containing views of the same person.

7. The method of claim 1, wherein for all learning image pairs, the difference in gaze direction in each image in the learning pair is the same.

8. The method of claim 1, wherein the method further comprises loading the predictor from an information medium storing the predictor, the information medium comprising one of the following: a hard disk, a solid state drive, a flash memory, an optical disk, a hybrid drive, a random access memory (RAM), a remote computer system, and a remote data storage.

9. The method of claim 1, wherein the step of generating a gaze adjusted image comprises identifying a plurality of landmarks in the digital input image.

10. The method of claim 9, wherein the step of generating a gaze-adjusted image further comprises generating a two-dimensional displacement vector field based on the plurality of landmarks, the two-dimensional displacement vector field indicating a displacement vector of the pixel for each of one or more pixels in the digital input image, and generating the gaze-adjusted image by shifting the one or more pixels in the digital input image according to the two-dimensional displacement vector field.

11. The method of claim 1, wherein the predictor is obtained by: obtaining a plurality of learning image pairs, the plurality of learning image pairs containing views of the same person in each image pair, the learning image pairs differing in gaze direction, and wherein for the learning image pairs, the difference in gaze direction in each image in the learning pair is the same; determining a position of an eye in each of said learning image pairs; as well as A predictor for generating an adjusted displacement vector field for each learning image pair is determined by replacing the color component of each pixel of the first image in the pair with the color component of another pixel of the first image in the pair based on the prediction of the predictor, thereby obtaining an image similar to the second image in the pair.

12. The method of claim 1, wherein the digital input image is a frame in a video stream comprising a plurality of frames.

13. A computing device, comprising: Logical Machines; as well as a storage machine storing instructions executable by the logic machine to: receiving a digital input image depicting a human eye; generating a gaze adjusted image from the digital input image by changing an apparent gaze direction of the human eye based on replacing a color component of each pixel in the human eye in the digital input image with a color component of another pixel in the digital input image via a gaze adjustment machine learning model, wherein the gaze adjustment machine learning model comprises a predictor based on a plurality of learning image pairs that differ in gaze direction; and The gaze-adjusted image is output.

14. A computing device as described in claim 13, wherein the step of generating a gaze-adjusted image includes generating a two-dimensional displacement vector field based on a plurality of landmarks identified in the digital input image, the two-dimensional displacement vector field indicating a displacement vector of one or more pixels in the digital input image for each pixel, and generating the gaze-adjusted image by shifting the one or more pixels in the digital input image according to the two-dimensional displacement vector field.

15. A method for image enhancement on a computing device, the method comprising: loading a predictor based on a plurality of learned image pairs containing views of the same person; Receiving at least one frame having a face of a person; Determine the position of the eyes of the person in the frame, and form two rectangular eye regions that closely surround the eyes; as well as The color components of each pixel in the rectangular eye region in the frame are replaced with the color components of the pixel in the frame according to the machine learning based prediction of the predictor.

16. The method of claim 15, wherein the difference in gaze direction in each image in the learning pair is the same for all learning image pairs.

17. The method of claim 15, wherein for all learning image pairs, the difference in gaze direction in each image in the learning pair is in the range of 5 to 20 degrees.

18. The method of claim 15, wherein the predictor is a single-layer or multi-layer neural network.

19. The method of claim 15, wherein the predictor is at least one decision tree or an ensemble of decision trees.

20. The method of claim 15, wherein the at least one frame is from a video stream comprising a plurality of frames.

Images