A gaze correction method and system based on codec and feature decoupling

Abstract

This invention provides a method and system for eye expression correction based on codec and feature decoupling. The method includes: Step a: Acquiring an original face image I, and obtaining the user's eye image I from the original face image. c and head pose information H gt Step b: For I c and H gt Feature extraction is performed to obtain the vector attribute code z representing the user's static attribute features. i Current eye pose information G, representing the current eye pose. pre And the rotation attribute code z, which characterizes the rotation attribute of the eye. r Step c: According to G pre and the pre-defined target eye pose information G tar For z r Perform a three-dimensional transformation to obtain the result with G tar The corresponding rotation attribute code z pre Step d: According to z i and z pre Generate target eye image I pre ; and step e: to I pre Paste back the original face image I, and output the face image after eye correction. The method of this invention effectively solves the image distortion problem caused by the confusion of attribute and pose features in traditional methods, and greatly improves the accuracy and naturalness of eye correction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, and particularly relates to a gaze correction method and system based on codec and feature decoupling. BACKGROUND

[0002] Gaze correction, as an innovative technology applied to face feature processing, aims to align the user's gaze with the target line of sight by adjusting the direction or position of the eyeball. This technology has wide application value in video conferencing, virtual reality, augmented reality, etc.

[0003] At present, the technical solutions for realizing gaze correction mainly include hardware methods and software methods.

[0004] Traditional gaze correction relies on dedicated hardware devices, such as eye tracking instruments or infrared cameras. These devices capture the eye movement trajectory and position, track the user's gaze in real time, and change the gaze direction through optical correction or hardware adjustment. However, this method often has high equipment cost, and often requires additional user wearing or fixing devices, resulting in poor user experience, and often has limited use, making it difficult to use in video conferencing and other public scenarios.

[0005] Software methods mainly rely on traditional image processing algorithms to analyze facial feature points (such as eyes, nose, mouth, etc.) in images, and perform geometric transformation or pixel replacement on the eye region in the image to achieve gaze correction. This method has high flexibility and speed, and does not require special hardware support. However, this traditional algorithm is usually based on static rules, has limited processing capacity, and lacks adaptive optimization capability, making it difficult to handle diverse user needs.

[0006] With the development of deep learning models such as Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs), gaze correction technology based on neural networks has gradually become a research hotspot. However, existing gaze correction technology based on neural networks still has problems such as insufficient accuracy, poor naturalness, and low computational efficiency. SUMMARY

[0007] In view of the above shortcomings of the prior art, the present application provides a gaze correction method and system based on codec and feature decoupling.

[0008] One aspect of the present application provides a gaze correction method based on codec and feature decoupling, comprising the following steps:

[0009] Step a: Acquire an original face image I using an image acquisition device, and obtain the user's eye image I from the original face image. c and head pose information H gt ;

[0010] Step b: For the eye image I c and head pose information H gt Feature extraction is performed to obtain the vector attribute code z representing the user's static attribute features. i Current eye pose information G, representing the current eye pose. pre And the rotation attribute code z, which characterizes the rotation attribute of the eye. r ;

[0011] Step c: Based on the current eye pose information G pre and the pre-defined target eye pose information G tar The rotation attribute z is encoded r Perform a three-dimensional transformation to obtain the target eye pose information G. tar The corresponding rotation attribute code z pre ;

[0012] Step d: Encode z according to the vector attribute. i and the rotation attribute encoding z pre Generate target eye image I pre ;as well as

[0013] Step e: Transfer the target eye image I pre Paste back the original face image I and output the face image after eye correction.

[0014] Another aspect of the present invention provides an eye-correction system based on codec and feature decoupling, comprising:

[0015] The data input module acquires a raw face image I using an image acquisition device, and obtains the user's eye image I from the raw face image. c and head pose information H gt

[0016] The feature extraction module extracts features from the eye image I. c and head pose information H gt Feature extraction is performed to obtain the vector attribute code z representing the user's static attribute features. i Current eye pose information G, representing the current eye pose. pre And the rotation attribute code z, which characterizes the rotation attribute of the eye. r ;

[0017] The feature transformation module, based on the current eye pose information Gpre and the preset target eye posture information G tar encoding the rotation attribute z r performing three-dimensional transformation to obtain the target eye posture information G tar corresponding rotation attribute z pre ;

[0018] a decoding module, configured to decode the vector attribute coding z i and the rotation attribute coding z pre generate a target eye image I pre ;

[0019] an image generation module, configured to paste the target eye image I pre back to the original face image I, and output a face image after eye correction.

[0020] The present application has the following beneficial effects:

[0021] 1. High correction accuracy and natural visual effect: The present application uses feature coding decoupling technology to ensure that the generated image can accurately and naturally adjust the eye contact while maintaining individual characteristics, and combines generative adversarial network to optimize the texture details generated, ensuring that the corrected image is visually natural and realistic.

[0022] 2. Strong real-time performance and suitable for low-resource devices: By using reparameterization processing on the trained multi-layer neural convolutional network, the present application significantly reduces the computational overhead during eye correction testing, enabling real-time correction in mobile devices and making it suitable for video conferencing, virtual reality and other scenarios with high requirements for interactivity and real-time performance. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a flowchart of an eye correction method based on codec and feature decoupling according to the preferred embodiment of the present application.

[0024] Figure 2 is a block diagram of an eye correction system based on codec and feature decoupling according to the preferred embodiment of the present application.

[0025] Figure 3 is a schematic diagram of reparameterization processing of the multi-layer neural convolutional network of the encoding module and the decoding module according to the preferred embodiment of the present application. DETAILED DESCRIPTION

[0026] The present application will be further described by examples, which are only intended to better understand the research content of the present application and not to limit the protection scope of the present application.

[0027] As Figure 1As shown, the method of one preferred embodiment of the present application comprises steps a-e.

[0028] Firstly, step a: an original face image I is collected by an image collection device such as a camera, the original face image includes face information, and an eye image I of a user is obtained from the original face image. c and head pose information H gt .

[0029] In a preferred embodiment, obtaining the eye image of the user in step a further comprises the following steps:

[0030] Step a1: identifying corresponding 106 face key points from the original face image I;

[0031] Step a2: positioning an eye region on the 106 face key points; preferably, the eye region maintains a left-right edge distance of 16 pixels from the left and right corners of the eyes, and a lower edge distance of 16 pixels from the lower eye socket;

[0032] Step a3: cropping the original face image according to the eye region to generate the eye image I c . Preferably, the image height is 96 pixels and the height is 64 pixels.

[0033] In addition, in step a, the head pose information H gt includes a pitch angle and a yaw angle.

[0034] Then step b: performing feature extraction on the eye image I c and head pose information H gt to obtain a vector attribute code z i representing user static attribute features, current eye pose information G pre representing the current eye pose, and a rotation attribute code z r representing the eye rotation attribute.

[0035] Preferably, in step b, the feature extraction module uses a convolutional neural network after reparameterization processing for feature extraction operation, and the convolutional neural network is composed of multiple cascaded convolutional layers and activation functions. Preferably, the user static attribute features are user personalized features, including gender, age, skin color, etc. Preferably, the vector attribute code z i is a 256-dimensional vector. The current eye pose information G pre before correction = (P pre , Y pre ). P pre represents the current pitch angle, and Y pre represents the current yaw angle. Preferably, the rotation attribute code z r is 48-dimensional.

[0036] Step c: Based on the current eye pose information G pre and the pre-defined target eye pose information G tar The rotation attribute z is encoded r Perform a three-dimensional transformation to obtain the target eye pose information G. tar The corresponding rotation attribute code z pre .

[0037] Preferably, the target eye pose information G needs to be preset before this step. tar =(P tar ,Y tar ), where P tar Y represents the target pitch angle. tar The target yaw angle is expressed in radians. Preferably, in a video call scenario, the target attitude can be set to the user's facing direction, i.e., both the pitch and yaw angles are 0 degrees, resulting in a target attitude of (0,0). In other scenarios, the target attitude can be manually specified by the user, for example, a target looking upwards at 15° and to the left at 10°. The target is looking down 15° and to the right 10°.

[0038] Preferably, step c further includes the following steps c1-c3. Before steps c1-c3, z is first... r The size was reorganized into 3*16, which were used as 16 three-dimensional representations.

[0039] Step c1: Obtain the current eye pose information G pre The corresponding three-dimensional rotation matrix R pre :

[0040]

[0041] Step c2: Obtain the target eye pose information G tar The corresponding three-dimensional rotation matrix R tar :

[0042]

[0043] Among them, R pre The matrix includes left and right rotations and up and down rotations, R tar The matrix includes left and right rotations and up and down rotations.

[0044] Step c3: For R pre Rotation inversion operation, and through R tar The rotation matrix is ​​transformed to obtain the rotation attribute code corresponding to the target eye pose. Finally, the 3*16 size z pre is reorganized into 48 dimensions, so as to be consistent with z r . This process completes the natural adjustment of the eye pose from the current direction G pre to the target direction G tar in the latent space.

[0045] The solution is step d: generating a target eye image I i according to the vector attribute encoding z pre and the rotation attribute encoding z pre .

[0046] Step d further comprises: a decoding module composed of a multi-layer convolutional neural network performs feature extraction on the vector attribute encoding z i and the rotation attribute encoding z pre , to generate a target eye image I pre .

[0047] Preferably, in step d, the decoding module uses a convolutional neural network after reparameterization processing for feature extraction operation, which is composed of a bilinear upsampling layer, a plurality of cascaded convolutional layers and an activation function. Wherein, the decoding module adopts bilinear upsampling operation, which performs excellently in image detail preservation and edge smoothing, thereby effectively avoiding the artifact problem brought by deconvolution method.

[0048] Finally, step e: paste the target eye image I pre back to the original face image I, and output the face image after eye correction. The system pastes the generated target eye image I pre back to the original face image I according to the key points, so as to realize seamless integration with the original face and avoid the feeling of fragmentation in the overall visual effect. The corrected complete face image will be displayed on the screen in real time, and the user can directly view the adjusted eye effect. In video communication and other scenarios, the adjusted image can be directly used for real-time transmission.

[0049] As described above, the present application independently encodes the personalized attributes of the user and the eye pose features through the feature extraction module, respectively generates attribute encoding z i and rotation attribute encoding z r , and realizes feature decoupling. Specifically, the attribute encoding z i represents the personalized features of the user, such as static information of the face, such as gender, age, skin color, etc.; the rotation attribute encoding z r is used to describe the dynamic rotation features of the user's eyes. In the eye correction process, only the rotation attribute encoding z r is adjusted, and the natural rotation process of the eyeball is simulated through three-dimensional pose transformation. At the same time, the attribute encoding z iThis process remains unchanged, ensuring that the generated image retains personalized features while accurately and naturally adjusting the gaze.

[0050] An embodiment of the present invention also provides an eye-tracking correction system 20 based on codec and feature decoupling, comprising:

[0051] Data input module 21 acquires an original face image I through an image acquisition device, and obtains the user's eye image I from the original face image. c and head pose information H gt ;

[0052] Feature extraction module 22, for the eye image I c and head pose information H gt Feature extraction is performed to obtain the vector attribute code z representing the user's static attribute features. i Current eye pose information G, representing the current eye pose. pre And the rotation attribute code z, which characterizes the rotation attribute of the eye. r ;

[0053] Feature transformation module 23, based on the current eye pose information G pre and the pre-defined target eye pose information G tar The rotation attribute z is encoded r Perform a three-dimensional transformation to obtain the target eye pose information G. tar The corresponding rotation attribute code z pre ;

[0054] Decoding module 24 encodes z according to the vector attribute. i and the rotation attribute encoding z pre Generate target eye image I pre ;

[0055] Image generation module 25 generates the target eye image I pre Paste back the original face image I and output the face image after eye correction.

[0056] The data input module 21 further includes: a face recognition module 211 and a head posture prediction module 212.

[0057] The facial recognition module 211 performs the following steps:

[0058] Step a1: Identify the corresponding 106 facial key points from the original face image I;

[0059] Step a2: locating an eye region based on the 106 facial landmarks; preferably, the eye region keeps a left and right edge distance of 16 pixels from the left and right corners of the eye, and a lower edge distance of 16 pixels from the lower eye socket;

[0060] Step a3: cropping the original face image according to the eye region to generate the eye image I c .

[0061] The head pose prediction module 212 is configured to predict the head pose H of the face in the image I gt , including the up-down deflection angle (pitch angle) and the left-right deflection angle (yaw angle).

[0062] Preferably, the multi-layer neural convolutional network of the feature extraction module (encoding module) 22 and the decoding module 24 of the present application adopts a simplified and optimized network design, thereby solving the problem that the prior art cannot realize real-time under resource-limited conditions.

[0063] In the preferred embodiment, the feature extraction module (encoding module) 22 and the decoding module 24 of the trained multi-layer neural convolutional network adopt a reparameterization process, thereby reducing the amount of calculation and memory read-write during testing. Specifically, for a convolution module, the module structure during training is as shown in Figure 3 , there are three branches, the first branch is a 3*3 convolution layer connected with a batch normalization layer (BN), the second branch is a 1*1 convolution layer connected with a batch normalization layer (BN), and the third branch is a residual connection (assuming that the input channel is the same as the output channel, otherwise there is no third branch). That is, during the training stage, the three branches are trained together, thereby effectively simulating the residual structure and the feature transformation structure independent of space. During the testing stage (eye correction processing stage), the 1*1 convolution can be regarded as a 3*3 convolution kernel with a 0 periphery, the residual structure can be regarded as a 3*3 convolution kernel with a 1 center and a 0 periphery, then the parameters of the batch normalization layer can be combined into the convolution kernel, and finally the parameters of the three kernels are added, that is, two convolutions, two batch normalization layers and a residual connection are combined into a convolution layer (3*3 convolution), thereby greatly reducing the number of layers, the amount of calculation and memory read-write.

[0064] In the preferred embodiment, all convolution layers use 3*3 convolution kernels. Because the operation efficiency of the 3*3 convolution kernel is much higher than that of other convolution kernels, while considering the neighborhood information.

[0065] In the preferred embodiment, all convolution modules are single-path architectures during testing, without any side branches. Because residual, cross-layer connection and other operations have small amount of calculation, but have a large amount of additional memory occupation, thereby reducing the operation efficiency. Therefore, the present application does not have any residual and cross-layer connection structure in actual use.

[0066] In preferred embodiments, the gaze correction system 20 further comprises a supervised optimization module 26 that optimizes the parameters of each module in the system through data-driven training. The module 26 uses five loss functions to improve the quality of the generated images and the accuracy of the pose adjustment.

[0067] (1). Reconstruction loss: measures the pixel-level difference between the generated image and the ground truth image by calculating the pixel error between the predicted image I pre and the target image I gt during training.

[0068] (2). Perception loss: measures the perceptual feature difference between the generated image and the ground truth image by calculating the perceptual feature difference between the input image I pre and the target image I gt during training using a pre-trained vgg16 network.

[0069] (3). Adversarial loss: based on the discriminator feedback of the generative adversarial network (GAN), the discriminator tries to distinguish the predicted image I pre as fake and the target image I gt as real, while the generator tries to confuse the discriminator, thus improving the realism of the generated image I pre .

[0070] (4). Pose loss: measures the angular error between the current predicted eye pose and the target pose by calculating the squared angular distance between the pose information G pre and the target pose G tar during training.

[0071] (5). Consistency loss: calculates the cosine similarity between the attribute encoding z i of the input image and the attribute encoding z t of the target image of the same person to ensure that different images of the same person have similar attribute encodings.

[0072] By jointly optimizing the five loss functions, the supervised optimization module 26 ensures that the generated correction images not only look realistic visually, but also have accurate pose adjustments.

[0073] Obviously, those skilled in the art should realize that the above embodiments are only used to illustrate the present application, and are not used as a limitation on the present application, and any changes and modifications to the above described embodiments within the spirit and principles of the present application will fall within the scope of the claims of the present application.

Claims

1. A gaze correction method based on codec and feature decoupling, characterized in that, Comprising the following steps: Step a: acquiring an original face image I by an image acquisition device, and obtaining an eye image I of a user from the original face image c and head pose information H gt ; Step b: performing feature extraction on the eye image I c and head pose information H gt to obtain a vector attribute code z representing static attribute features of the user i , current eye pose information G representing the current eye pose pre and a rotation attribute code z representing the eye rotation attribute r ; Step c: encoding z according to the current eye pose information G pre and the pre-set target eye pose information G tar the rotation attribute encoding z r performing three-dimensional transformation to obtain the corresponding rotation attribute encoding z tar corresponding to the target eye pose information G pre ; Step d: encoding z for the vector attribute i and z for the rotation attribute pre performing feature extraction to generate a target eye image I pre ; and Step e: pasting back the original face image I, outputting the face image after the eye spirit correction, pre the original face image I, outputting the face image after the eye spirit correction, Wherein, step c further comprises the following steps: Step cl: obtaining said current eye pose information G pre corresponding three-dimensional rotation matrix R pre ; Step c2: obtaining the target eye pose information G tar The corresponding three-dimensional rotation matrix R tar ; Step c3: R pre The inverse operation of rotation is taken, and the rotation attribute code corresponding to the target eye pose is obtained by R tar The rotation matrix is transformed to obtain the rotation attribute code corresponding to the target eye pose 2. The codec and feature decoupled gaze correction method of claim 1, wherein, The step a of acquiring the eye image of the user further comprises the following steps: Step a1: identifying a plurality of facial key points corresponding to the original facial image I from the original facial image I; Step a2: positioning the eye region on the plurality of facial key points; Step a3: cropping the original face image according to the eye region to generate the eye image I c .

3. The gaze correction method based on codec and feature decoupling according to claim 1, characterized in that, In step a, the head pose information H gt comprises a vertical deflection angle, i.e. a pitch angle, and a horizontal deflection angle, i.e. a yaw angle.

4. The codec and feature decoupled gaze correction method of claim 1, wherein, In step b, the feature extraction operation is performed using the convolutional neural network after the reparameterization processing, and the convolutional neural network is composed of a plurality of cascaded convolutional layers and activation functions.

5. The codec and feature decoupled gaze correction method of claim 1, wherein, In steps cl and c2, the current eye pose information G pre = (P pre , Y pre ), where P pre denotes the current pitch angle, Y pre denotes the current yaw angle, the target eye pose information G tar = (P tar , Y tar ), where P tar denotes the target pitch angle, Y tar denotes the target yaw angle, In the above equation, R pre The matrix includes a left and right rotation and an up and down rotation, R tar The matrix includes a left and right rotation and an up and down rotation.

6. The gaze correction method based on codec and feature decoupling according to claim 1, characterized in that, In step d, the feature extraction operation is performed using the convolutional neural network after the reparameterization processing, and the convolutional neural network is composed of a bilinear upsampling layer, a plurality of cascaded convolutional layers and activation functions.

7. A gaze correction system based on codec and feature decoupling, using the method of any one of claims 1-6, characterized in that, Comprising: The data input module acquires an original face image I through an image acquisition device, and acquires an eye image I of a user from the original face image c and head pose information H gt a feature extraction module to perform feature extraction on the eye image I c and head pose information H gt to obtain a vector attribute code z representing static attribute features of the user i , current eye pose information G representing a current eye pose pre and a rotation attribute code z representing an eye rotation attribute r ; a feature transformation module, according to the current eye posture information G pre and the pre-set target eye posture information G tar encoding the rotation attribute z r performing three-dimensional transformation to obtain the corresponding rotation attribute encoding z tar corresponding to the target eye posture information G pre ; a decoding module to decode the vector attribute encoding z i and the rotation attribute encoding z pre to generate a target eye image I pre ; an image generation module, which generates a target eye image I pre pasting the original face image I, and outputting the face image after the eye spirit correction.

8. The eye correction system based on codec and feature decoupling according to claim 7, characterized in that, The feature extraction module and the decoding module use the reparameterization processing on the trained multi-layer neural convolutional network for eye correction.

9. The eye correction system based on codec and feature decoupling according to claim 7, characterized in that, Further comprising a supervised optimization module, which optimizes the parameters in the neural convolutional network of the feature extraction module and the decoding module through a data-driven training method, and the supervised optimization module uses a plurality of loss functions.

Images