Sight line redirection method, apparatus, device, and storage medium

Abstract

The application discloses a line-of-sight redirection method and device, equipment and a storage medium, and relates to the technical field of image processing, aiming to at least solve the problem of low authenticity of image information of a generated image. The method comprises the following steps: extracting a first target human eye image, target head posture information and target human eye gaze direction information from a to-be-processed image; based on the target head posture information and the target human eye gaze direction information, performing redirection adjustment on the line of sight of the human eye in the first target human eye image to obtain a second target human eye image; and the line of sight of the human eye in the second target human eye image matches the target head posture information and the target human eye gaze direction information.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a gaze redirection method, apparatus, device, and storage medium. Background Technology

[0002] Currently, gaze redirection is used to track the gaze point based on the position and direction of the human eye. This allows for the re-rendering of the gazed object based on the gaze point's location and camera parameters, resulting in more realistic image information in images or videos and providing users with a sense of realism and depth. However, this gaze redirection process adjusts the gaze point solely based on the eye's viewing angle. This means the generated image only reflects the influence of eye range information on the gaze point and cannot fully reflect the characteristics of human eye behavior, thus reducing the realism of the generated image. Summary of the Invention

[0003] This invention provides a gaze redirection method, apparatus, device, and storage medium to at least solve the problem of low image fidelity in generated images. The technical solution of this invention is as follows:

[0004] According to a first aspect of the present invention, a gaze redirection method is provided, the method comprising: extracting a first target human eye image, target head posture information, and target human eye gaze direction information from an image to be processed; redirecting and adjusting the gaze of the human eye in the first target human eye image based on the target head posture information and the target human eye gaze direction information to obtain a second target human eye image; wherein the gaze of the human eye in the second target human eye image matches the target head posture information and the target human eye gaze direction information.

[0005] In one possible implementation, the line of sight in the first target eye image is redirected and adjusted based on the target head pose information and the target eye gaze direction information to obtain the second target eye image; this includes: inputting the head pose information, the eye gaze direction information, and the first target eye image into a preset model to obtain the second target eye image; the preset model is used to redirect the line of sight in the corresponding eye image based on the head pose information and the eye gaze direction information to obtain the redirected eye image; the preset model is trained through multiple sets of different training samples, wherein each set of training samples includes a first sample eye image, a second sample eye image, the first eye gaze direction and the first head pose information corresponding to the first sample eye image, and the second eye gaze direction and the second head pose information corresponding to the second sample eye image.

[0006] In another possible implementation, before inputting head pose information, eye gaze direction information, and a first target eye image into a preset model to obtain a second target eye image, the method further includes: constructing an initial model; wherein the initial model includes an initial generator network; the initial generator network is used to redirect the eye gaze in the image and generate a corresponding eye image; using a first sample eye image, a second eye gaze direction, and second head pose information from multiple sets of different training samples as input parameters to the initial generator network, and using the second sample eye image corresponding to the second eye gaze direction and second head pose information as the target image, the network parameters of the initial generator network are adjusted to obtain a target generator network; the matching degree between the eye image output by the target generator network and the corresponding target image meets a preset condition; and the target generator network is determined as the preset model.

[0007] In another possible implementation, the network parameters of the initial generation network are adjusted using the first sample human eye image, the second human eye gaze direction, and the second head pose information from multiple sets of different training samples as input parameters of the initial generation network, and the target image is the second sample human eye image corresponding to the second human eye gaze direction and the second head pose information as the output human eye image of the initial generation network. This adjustment, to obtain the target generation network, includes: for any parameter adjustment during the adjustment process of the initial generation network parameters, inputting the first sample human eye image, the second human eye gaze direction, and the second head pose information from each set of training samples into the initial generation network to obtain the first output sample human eye image corresponding to each set of training samples after retargeting; and inputting the first sample human eye image corresponding to each set of training samples into the target image. An output sample human eye image and its corresponding first eye gaze direction and first head pose information are input into an initial generator network to obtain a redirected second output sample human eye image. Based on the first loss function value corresponding to the first and second differences, the network parameters of the initial generator network are adjusted. The first difference characterizes the difference between each second output sample human eye image and the first sample human eye image corresponding to each group of training samples. The second difference characterizes the difference between each first output sample human eye image and the second sample human eye image corresponding to each group of training samples. When the first loss function value of the first difference and the second difference satisfies the first loss condition, the initial generator network adjusted under the network parameters corresponding to the first difference and the second difference is determined as the target generator network.

[0008] In another possible implementation, the initial model further includes an initial discriminant network used to predict the realism of an image and the corresponding image information. The method further includes: for any parameter adjustment during the adjustment of the network parameters of the initial generator network, inputting the first output sample human eye image and the corresponding second sample human eye image corresponding to each group of training samples into the initial discriminant network to predict the realism of the first output sample human eye image relative to the second sample human eye image, as well as the corresponding predicted head pose information and predicted human eye gaze direction; the realism characterizes the probability of minimizing the second sample human eye image being classified as a generated image; the second human eye gaze direction and the second head pose information are respectively the difference between the corresponding predicted head pose information and the predicted human eye gaze direction, and the weighted sum between the realism, and are determined as the second loss function value; based on the second loss function value, the network parameters of the initial discriminant network are adjusted; when the second loss function value satisfies the second loss condition, the initial discriminant network adjusted under the network parameters corresponding to the second loss function value is determined as the target discriminant network.

[0009] In another possible implementation, the method further includes: predicting the target realism of the first output sample human eye image relative to the second sample human eye image, as well as the corresponding target predicted head pose information and target predicted human eye gaze direction, based on the target discrimination network; and determining the second difference degree based on the weighted sum of the differences between the second human eye gaze direction and the second head pose information and the corresponding target predicted head pose information and target predicted human eye gaze, and the negative number of the target realism.

[0010] In another possible implementation, the initial model also includes a VGG network. The method further includes: inputting each second output sample human eye image and the first sample human eye image corresponding to each group of training samples into the VGG network to obtain the perceptual loss between each first output sample human eye image and the second sample human eye image corresponding to each group of training samples; determining the second difference based on the weighted sum of the difference between the second human eye gaze direction and the second head pose information and the corresponding target predicted head pose information and the target predicted human eye gaze, and the negative number of the target realism, including: determining the second difference as the weighted sum of the difference between the second human eye gaze direction and the second head pose information and the corresponding target predicted head pose information and the target predicted human eye gaze, the negative number of the target realism, and the perceptual loss.

[0011] In another possible implementation, extracting a first target human eye image from the image to be processed includes: detecting target key parts in the image to be processed based on facial key points; configuring the detected target key parts into a preset part area of ​​a preset contour template to obtain a target face image; cropping a first eye image from the target face image; the target eye image includes the eye area covered by any one eye in the target face image; mirroring the first eye image in the horizontal direction to obtain a second eye image; and determining the first eye image or the second eye image as the first target human eye image.

[0012] According to a second aspect of the present invention, a gaze redirection device is provided, the device comprising: an extraction unit configured to extract a first target human eye image, target head posture information, and target human eye gaze direction information from an image to be processed; and a gaze adjustment unit configured to redirect and adjust the gaze of a human eye in the first target human eye image based on the target head posture information and the target human eye gaze direction information to obtain a second target human eye image; wherein the gaze of a human eye in the second target human eye image matches the target head posture information and the target human eye gaze direction information.

[0013] In one possible implementation, the gaze adjustment unit is specifically configured to: input head pose information, human eye gaze direction information, and a first target human eye image into a preset model to obtain a second target human eye image; the preset model is used to redirect the human eye gaze in the corresponding human eye image based on the head pose information and human eye gaze direction information to obtain a redirected human eye image; the preset model is trained through multiple sets of different training samples, wherein each set of training samples includes a first sample human eye image, a second sample human eye image, a first human eye gaze direction and a first head pose information corresponding to the first sample human eye image, and a second human eye gaze direction and a second head pose information corresponding to the second sample human eye image.

[0014] In another possible implementation, the gaze adjustment unit includes a preset model. Before inputting head pose information, eye gaze direction information, and a first target eye image into the preset model to obtain a second target eye image, the gaze adjustment unit is further configured to: construct an initial model; wherein the initial model includes an initial generation network; the initial generation network is used to redirect the gaze of the human eye in the image and generate the corresponding eye image; using the first sample eye image, the second eye gaze direction, and the second head pose information from multiple sets of different training samples as input parameters of the initial generation network, and using the second sample eye image corresponding to the second eye gaze direction and the second head pose information as the target image, the network parameters of the initial generation network are adjusted to obtain a target generation network; the matching degree between the eye image output by the target generation network and the corresponding target image meets a preset condition; and the target generation network is determined as the preset model.

[0015] In another possible implementation, the gaze adjustment unit is further configured to: for any parameter adjustment during the adjustment of the network parameters of the initial generator network, input the first sample human eye image, the second human eye gaze direction, and the second head pose information from each group of training samples into the initial generator network to obtain the first output sample human eye image corresponding to each group of training samples after redirection; input the first output sample human eye image and the corresponding first human eye gaze direction and first head pose information corresponding to each group of training samples into the initial generator network to obtain the second output sample human eye image after redirection; adjust the network parameters of the initial generator network based on the first loss function value corresponding to the first difference degree and the second difference degree; the first difference degree characterizes the difference between each second output sample human eye image and the first sample human eye image corresponding to each group of training samples; the second difference degree characterizes the difference between each first output sample human eye image and the second sample human eye image corresponding to each group of training samples; when the first loss function value of the first difference degree and the second difference degree satisfies the first loss condition, determine the initial generator network adjusted under the network parameters corresponding to the first difference degree and the second difference degree as the target generator network.

[0016] In another possible implementation, the initial model further includes an initial discriminant network, which is used to predict the realism of the image and the corresponding image information. The gaze adjustment unit is specifically configured to: for any parameter adjustment during the adjustment of the network parameters of the initial generator network, input the first output sample human eye image and the corresponding second sample human eye image corresponding to each group of training samples into the initial discriminant network, predict the realism of the first output sample human eye image relative to the second sample human eye image, as well as the corresponding predicted head pose information and predicted human eye gaze direction; the realism characterizes the probability of minimizing the second sample human eye image being judged as a generated image; the second human eye gaze direction and the second head pose information are respectively the difference between the corresponding predicted head pose information and the predicted human eye gaze direction, and the weighted sum between the realism, and are determined as the second loss function value; based on the second loss function value, the network parameters of the initial discriminant network are adjusted; when the second loss function value satisfies the second loss condition, the initial discriminant network adjusted under the network parameters corresponding to the second loss function value is determined as the target discriminant network.

[0017] In another possible implementation, the gaze adjustment unit is further configured to: predict the target realism of the first output sample human eye image relative to the second sample human eye image, as well as the corresponding target predicted head pose information and target predicted human eye gaze direction, based on the target discrimination network; and determine the second difference degree by weighted sum of the differences between the second human eye gaze direction and the second head pose information and the corresponding target predicted head pose information and target predicted human eye gaze, respectively, and the negative number of the target realism.

[0018] In another possible implementation, the initial model also includes a VGG network, and the gaze adjustment unit is specifically configured to: input each second output sample human eye image and the first sample human eye image corresponding to each group of training samples into the VGG network to obtain the perceptual loss between each first output sample human eye image and the second sample human eye image corresponding to each group of training samples; determine the second difference based on the weighted sum of the difference between the second human eye gaze direction and the second head posture information and the corresponding target predicted head posture information and the target predicted human eye gaze, and the negative number of the target realism, including: determining the second difference as the weighted sum of the difference between the second human eye gaze direction and the second head posture information and the corresponding target predicted head posture information and the target predicted human eye gaze, the negative number of the target realism, and the perceptual loss.

[0019] In another possible implementation, the extraction unit is specifically configured to: detect target key parts in the image to be processed based on facial key points; configure the detected target key parts into a preset part area of ​​a preset contour template to obtain a target facial image; crop out a first eye image from the target facial image; the target eye image includes the eye area covered by any one eye in the target facial image; mirror the first eye image in the horizontal direction to obtain a second eye image; and determine the first eye image or the second eye image as the first target human eye image.

[0020] According to a third aspect of the present invention, an image processing apparatus is provided, the image processing apparatus being configured to perform a gaze redirection method as described in the first aspect and any possible implementation thereof.

[0021] According to a fourth aspect of the present invention, a computer device is provided, comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement a gaze redirection method as described in the first aspect and any possible implementation thereof.

[0022] According to a fifth aspect of the present invention, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a processor of a computer device, the computer device is enabled to perform a gaze redirection method as described in the first aspect and any possible implementation thereof.

[0023] According to a sixth aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions, which, when executed on a computer device, cause the computer device to perform the gaze redirection method described in the first aspect and any possible implementation thereof.

[0024] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects: Based on the target head posture information and target eye gaze direction information extracted from the image to be processed, the eye gaze of the first target eye image extracted from the image to be processed is redirected and adjusted so that the eye gaze in the adjusted second target eye image matches the target head posture information and target eye gaze direction information in the image to be processed. This allows the second target eye image to simultaneously reflect the influence of the intrinsic relationship between the eye gaze and head posture on the gaze point, so as to fully reflect the human eye behavior feature information and thus improve the realism of the image information of the generated image.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0027] Figure 1 This is a flow chart of a gaze redirection method according to an exemplary embodiment. Figure 1 ;

[0028] Figure 2 This is a flowchart illustrating a gaze redirection method according to an exemplary embodiment. Figure 2 ;

[0029] Figure 3 This is a schematic diagram illustrating a line-of-sight redirection process according to an exemplary embodiment;

[0030] Figure 4 This is a block diagram illustrating a line-of-sight redirection device according to an exemplary embodiment;

[0031] Figure 5 This is a schematic diagram of an image processing device according to an exemplary embodiment. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to image data used for analysis, stored image data, displayed image data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0034] Before providing a detailed description of the line-of-sight redirection method provided in the embodiments of this application, let's first briefly introduce the application scenarios involved in the embodiments of this application.

[0035] Currently, gaze redirection is used to track the gaze point based on the position and direction of the human eye. This allows for the re-rendering of the gazed object based on the gaze point's location and camera parameters, resulting in more realistic image information in images or videos and providing users with a sense of realism and depth. However, this gaze redirection process adjusts the gaze point solely based on the eye's viewing angle. This means the generated image only reflects the influence of eye range information on the gaze point and cannot fully reflect the characteristics of human eye behavior, thus reducing the realism of the generated image.

[0036] Specifically, in traditional virtual scenes, objects and actions are rendered according to pre-designed paths and architectures. However, with the continuous development of virtual reality technology, users require greater interactivity and flexibility, namely, the ability to freely control the viewpoint and gaze point within the virtual scene. Therefore, to meet this need, the Monocular Gaze Redirection Model was proposed.

[0037] A monocular gaze redirection model is a model for human eye gaze points in computer graphics and virtual reality. It improves the realism and interactivity of images and videos by addressing the gaze point problem in virtual scenes. A monocular gaze redirection model can be divided into two main parts: the first is gaze point estimation based on human eye movement, which uses the trajectory of human eye movement to estimate the position and orientation of the eyes, thereby calculating the location of the gaze point. The second is gaze point feedback based on object rendering, which re-renders the gazed object according to the location of the gaze point and camera parameters, allowing the user to experience realistic vision and depth.

[0038] The aforementioned gaze redirection techniques often focus on adjusting the gaze angle and generating corresponding images, neglecting the intrinsic connection between human gaze and head posture. In real life, the same gaze direction will present completely different visual effects under different head postures. Therefore, traditional methods that only consider the gaze angle without considering head posture cannot truly reflect human behavior.

[0039] To address the aforementioned issues, this application provides a gaze redirection method. Based on the target head posture information and target eye gaze direction information extracted from the image to be processed, the gaze of the target human eye image extracted from the image to be processed is redirected and adjusted so that the gaze of the target human eye image in the adjusted second target human eye image matches the target head posture information and target eye gaze direction information in the image to be processed simultaneously. This allows the second target human eye image to simultaneously reflect the influence of the intrinsic relationship between the gaze of the human eye and the head posture on the gaze point, so as to fully reflect the human eye behavior feature information and improve the realism of the image information of the generated image.

[0040] The gaze redirection method provided in this application can be applied to image processing devices with human eye image extraction capabilities.

[0041] For ease of understanding, the line-of-sight redirection method provided in this application will be described in detail below with reference to the accompanying drawings.

[0042] Figure 1 This is a flowchart illustrating a gaze redirection method according to an exemplary embodiment, such as... Figure 1 As shown, the gaze redirection method includes the following steps.

[0043] S11, the image processing device extracts the first target human eye image, the target head posture information, and the target human eye gaze direction information from the image to be processed.

[0044] The target head pose information mentioned above can be understood as the angle of a person's head relative to the head coordinate system. Target head pose information typically includes the following aspects: head rotation angle, tilt angle, and lateral angle.

[0045] Specifically, the following parameters can be used to describe head posture information: 1. Roll: The angle of head rotation around the vertical axis, used to describe the left and right rotation of the head; 2. Pitch: The angle of head rotation up and down, used to describe the up and down tilt of the head; 3. Yaw: The angle of head rotation around the longitudinal axis, used to describe the left and right tilt of the head.

[0046] The target person's eye gaze direction information includes the horizontal gaze direction and the vertical gaze direction of the eyeball.

[0047] It should be noted that the gaze direction mentioned below (e.g., the first gaze direction and the second gaze direction) includes both the horizontal and vertical gaze directions of the eyeball. Similarly, the head posture direction mentioned below (e.g., the first and second head posture information) includes roll, pitch, and yaw.

[0048] S12, the image processing device redirects and adjusts the line of sight of the human eye in the first target human eye image based on the target head posture information and the target human eye gaze direction information to obtain the second target human eye image.

[0049] The line of sight of the human eye in the second target human eye image mentioned above matches the target head posture information and the target human eye gaze direction information.

[0050] Through the above implementation method, based on the target head posture information and target eye gaze direction information extracted from the image to be processed, the eye gaze of the first target eye image extracted from the image to be processed is redirected and adjusted so that the eye gaze in the adjusted second target eye image matches the target head posture information and target eye gaze direction information in the image to be processed. This allows the second target eye image to simultaneously reflect the influence of the intrinsic relationship between the eye gaze and head posture on the gaze point, so as to fully reflect the human eye behavior feature information and improve the realism of the generated image information.

[0051] Furthermore, based on the above implementation method, the gaze redirection operation in the image processing device is no longer limited to a fixed head posture. Thus, it can not only provide a richer and more natural user experience in applications, but also generate the correct gaze direction for different head postures, thereby expanding the applicability of gaze redirection technology.

[0052] As a refinement and extension of the specific implementation of the above embodiments, in order to fully explain the specific implementation process of this embodiment, this application provides some other implementation methods of line-of-sight redirection.

[0053] Combination Figure 1 like Figure 2 As shown, the method for extracting the first target human eye image in step S11 above can be implemented through the following steps.

[0054] S111, the image processing device detects and extracts key target parts in the image to be processed based on facial key points.

[0055] Based on facial key points, target key parts representing facial information are extracted from the image to be processed.

[0056] S112, the image processing device configures the detected key parts of the target into the preset part area of ​​the preset contour template to obtain the target facial image.

[0057] The preset contour template is used to indicate the preset regions in which each key target feature is located in the target facial image. In the preset contour template, different key target features correspond to different preset regions, and there is a one-to-one correspondence between key target features and preset regions.

[0058] The aforementioned key target areas include the eye area, and may also include at least one or more of the following: the facial contour area, the eyebrow area, the nose area, the mouth area, the cheek area, the jaw area, and the ear area.

[0059] The aforementioned key target parts are configured into the preset part areas of a unified preset contour template, so that the acquired target facial images are all images of the same size as the preset contour template, and each key target part of the target facial image is located in the corresponding preset part area.

[0060] In some embodiments, the above-mentioned preset contour template is a face image template with uniform size and uniform key part layout.

[0061] S113, the image processing device crops out the first eye image from the target facial image according to the preset cropping rules.

[0062] This preset contour template represents the correspondence between key facial features and their locations within the facial contour area.

[0063] Its preset cropping rules are used to instruct the cropping of preset areas, including the eyes, in the target facial image.

[0064] The target eye image includes the eye region covered by either eye in the target face image.

[0065] S114, the image processing device mirrors the first eye image in the horizontal direction to obtain the second eye image.

[0066] To eliminate the influence of differences between the left and right eyes in human eye data for practical applications, it is typically necessary to acquire any target eye image corresponding to any eye region, i.e., select either the right or left eye. This target eye image is then mirrored horizontally to obtain a mirrored eye image. This ensures that the processing flow for the left and right eyes is consistent during initial model training. Consequently, the generated human eye dataset allows the initial model to focus more on learning specific targets without needing to consider the differences between the left and right eyes.

[0067] S115, the image processing device determines the first eye image and the second eye image as the first target human eye image.

[0068] In one embodiment, the specific process of cropping the first eye image from the target facial image is as follows: For any eye region corresponding to each of the two eyes in the target facial image, a predetermined number of target eye key points are determined in the target facial image. These predetermined number of target eye key points are used to characterize the eye feature information included in the target facial image. Then, using the minimum bounding circle algorithm, a minimum circle region corresponding to the minimum bounding circle enclosed by the predetermined number of target eye key points on the target facial image is constructed. Based on this minimum circle region, the target eye image is cropped from the target facial image. This target eye image includes the minimum circle region.

[0069] Specifically, based on the minimum circular region, the target eye image is cropped from the target face image, including: determining a square region covered by the target eye image with the center point of the minimum circular region as the center and the target length as the side length; and cropping the target eye image from the target face image using the square region as the target region.

[0070] The target length is the radius of the smallest circular region that is a multiple of a preset number, where the preset multiple is an integer greater than 1 and less than 6.

[0071] Based on the above-mentioned method of acquiring human eye images, when cropping the target eye image from the target face image, it is only necessary to crop according to the cropping requirements of the target eye image, using the preset area corresponding to the eye in the key parts of the target, so that the layout and image size of the key parts in different target eye images obtained according to the same cropping requirements are consistent.

[0072] Therefore, by using the above-mentioned standardized processing method for human eye images, it is possible to facilitate the creation / preprocessing of large human eye image datasets, reduce the differences in human eye image data, and thus improve the accuracy of the processing or application results corresponding to human eye image data. At the same time, it can also ensure that the human eye data of different individuals achieves consistent processing results and can be seamlessly embedded into the entire human eye data application process without producing any flaws.

[0073] In one implementation, to ensure the speed of acquiring the second target eye image, step S12 is specifically implemented based on a preset model. Specifically, this includes: the image processing device inputs head posture information, eye gaze direction information, and the first target eye image into the preset model to obtain the second target eye image.

[0074] This pre-defined model is used to redirect the gaze direction in a corresponding human eye image based on head pose information and eye gaze direction information, thus obtaining a redirected human eye image. This pre-defined model is trained using multiple sets of different training samples. Each set of training samples is labeled according to the input sample and output sample categories, and each set of training samples includes a first sample human eye image, a second sample human eye image, the first eye gaze direction and first head pose information corresponding to the first sample human eye image, and the second eye gaze direction and second head pose information corresponding to the second sample human eye image.

[0075] The aforementioned preset model has functions such as gaze update, denormalization processing, and inverse alignment transformation to complete the gaze redirection function of the original face image. Among them, denormalization processing restores the original eye position, so that the face can be seamlessly pasted back; inverse alignment transformation pastes the aligned face back into the original image, thus completing the entire gaze redirection process.

[0076] The aforementioned second sample human eye image can be the human eye image corresponding to the actual line of sight of the human eye in the first sample human eye image under the second human eye's gaze direction and second head posture information. The aforementioned first sample human eye image and second sample human eye image are the same human eye image.

[0077] The specific training process for the above-mentioned preset model is as follows.

[0078] First, an initial model is constructed, including an initial generator network. The initial generator network redirects the human eye's gaze in the image and generates the corresponding eye image. Understandably, the initial discriminator network is used to identify the realism of the image, the direction of the human eye's gaze, and head pose information.

[0079] Then, multiple sets of training samples are selected, and the network parameters of the initial generated network are trained using these multiple sets of training samples.

[0080] Specifically, to ensure that the initial model and the preset model can learn gaze redirection information under different head poses, multiple sets of training samples will include human facial images from multiple angles, with various expressions, and under different lighting conditions (i.e., different first sample human eye images and second sample human eye images). These images are labeled with image numbers, corresponding head poses, and gaze angles to provide accurate head pose information and human eye gaze direction data, thereby ensuring the learning and generalization of the initial model.

[0081] Furthermore, using the first sample human eye image, the second human eye gaze direction, and the second head pose information from multiple sets of different training samples as the input parameters of the initial generation network, and the second sample human eye image corresponding to the second human eye gaze direction and the second head pose information as the target image of the human eye image output by the initial generation network, the network parameters of the initial generation network are adjusted to obtain the target generation network. The matching degree between the human eye image output by the target generation network and the corresponding target image meets the preset conditions.

[0082] Finally, the aforementioned target generation network is selected as the preset model. In practical applications, the head pose information, eye gaze direction information, and the first target eye image are directly input into the target generation network layer of the initial model, after the network parameters of the initial generation network have been adjusted, to obtain the generated second target eye image.

[0083] In some embodiments, the initial generation network described above may be a G network, a generator, or a generation module.

[0084] like Figure 3 As shown, the specific parameter adjustment process for any one of the network parameters in the initial generation network is as follows, in order to generate the target generation network.

[0085] First, the first sample human eye image, the second human eye gaze direction, and the second head pose information from each group of training samples are input into the initial generation network. Then, the initial generation network after network parameter adjustment is called to redirect the human eye gaze in the first sample human eye image of each group of training samples, so as to obtain the first output sample human eye image corresponding to each group of training samples after redirection.

[0086] Furthermore, the first output sample human eye image corresponding to each group of training samples, along with the corresponding first human eye gaze direction and first head pose information, are input into the initial generation network to call the initial generation network to adjust the human eye gaze in the first output sample human eye image, thereby obtaining the redirected second output sample human eye image.

[0087] Secondly, the first output sample human eye image and the corresponding second sample human eye image are input into the initial discrimination network to train the initial discrimination network, so as to ensure that the target discrimination network can judge the authenticity of the image, evaluate the gaze angle and the head posture angle.

[0088] Simultaneously, the human eye images of each second output sample and the first sample human eye images corresponding to each group of training samples are input into the VGG network to reconstruct human eye images that are completely similar to the human eye texture and structure of the input images.

[0089] Specifically, the difference between each second output sample human eye image and the first sample human eye image corresponding to each group of training samples is defined as the first difference degree, and the difference between each first output sample human eye image and the second sample human eye image corresponding to each group of training samples is defined as the second difference degree.

[0090] Based on the first loss function values ​​corresponding to the first and second differences, the network parameters of the initial generator network are adjusted so that when the first loss function values ​​of the first and second differences satisfy the first loss condition, the initial generator network adjusted under the network parameters corresponding to the first and second differences is determined as the target generator network for the preset model.

[0091] The first loss condition mentioned above is that the first loss function value is less than a preset threshold or the first loss function value is the smallest among the preset number of first loss function values ​​corresponding to the preset number of adjustments.

[0092] Furthermore, in order to ensure more accurate recognition of generated image information during the initial network training process of the initial model, thereby enabling the target generation network to generate more realistic images, the initial discriminant network set in the initial model is further trained. This initial discriminant network is used to predict the realism of the image and the corresponding image information.

[0093] Based on this, for any parameter adjustment during the process of adjusting the network parameters of the initial generation network, the first output sample human eye image and the corresponding second sample human eye image corresponding to each group of training samples are input into the initial discriminator network to predict the realism of the first output sample human eye image relative to the second sample human eye image, as well as the predicted head pose information and predicted human eye gaze direction corresponding to the first output sample human eye image and the corresponding second sample human eye image.

[0094] Specifically, the realism characterizes the probability of the second sample human eye image being classified as a generated image. The second loss function value is determined by the weighted sum of the difference between the second human eye gaze direction and the second head pose information (corresponding to the predicted head pose information and predicted human eye gaze direction, respectively) and the realism value. Based on the second loss function value, the network parameters of the initial discrimination network are adjusted so that when the second loss function value satisfies the second loss condition, the adjusted initial discrimination network with the network parameters corresponding to the second loss function value is determined as the target discrimination network.

[0095] Furthermore, based on the aforementioned target discrimination network, the target realism of the first output sample human eye image relative to the second sample human eye image, as well as the target predicted head pose information and target predicted human eye gaze direction corresponding to the first output sample human eye image, are predicted. A second difference degree is determined by a weighted sum of the differences between the second human eye gaze direction and the second head pose information, respectively, and the corresponding target predicted head pose information and target predicted human eye gaze, along with the negative value of the target realism.

[0096] In some implementations, the initial model described above is a network structure of a Generative Adversarial Network (GAN). This initial discriminator network can also be referred to as a D network, discriminator, distinguisher, or discriminator module.

[0097] In the training for the second target, human eye image extraction, the D network is typically used to distinguish between generated eye images and real eye images, i.e., to determine whether the generated eye images are authentic. In Generative Adversarial Networks (GANs), the D network is part of a GAN network and is responsible for learning to distinguish between generator-generated eye images and real eye images. The goal of the D network is to maximize two aspects of the loss function: firstly, minimizing the probability that the generator-generated eye image is classified as a real image to improve the generator's generation ability; secondly, minimizing the probability that a real eye image is classified as a generated image to improve the D network's ability to distinguish real images.

[0098] Through training, the D network gradually learns to distinguish between real and generated eye images, providing feedback to the generator and prompting it to produce more realistic eye images. This adversarial relationship between the generator and the discriminator (D network) allows the generator to continuously improve its generation capabilities during training, resulting in more realistic eye images. Therefore, in the above-mentioned human eye image extraction training, the D network is used to distinguish between generated and real eye images, thereby driving the generator to produce more realistic and higher-quality eye images.

[0099] Furthermore, to ensure the accuracy of the pre-defined model of the target generation network obtained after training, a VGG network is also set up in the initial model during training. This VGG network is used to calculate the perceptual loss during training on human eye image extraction. This perceptual loss can be understood as a loss function that measures the similarity between generated images and real images. Specifically, the human eye images of each second output sample and the first sample human eye images corresponding to each group of training samples are input into the VGG network to obtain the perceptual loss between each first output sample human eye image and the second sample human eye images corresponding to each group of training samples. The perceptual loss includes content loss and / or style loss.

[0100] Based on the perceptual loss obtained in the above manner, the second difference degree is determined, specifically including: the weighted sum of the difference between the second human eye gaze direction and the second head posture information and the corresponding target predicted head posture information and target predicted human eye gaze, the negative number of target realism, and the perceptual loss degree, which is determined as the second difference degree.

[0101] For example, the first loss function formed by the first loss condition is as follows (1), and the second loss function formed by the second loss condition is as follows (2).

[0102]

[0103] Wherein, the first loss function value is L G The second loss function value is L. D L adv For target realism; L rec The first degree of difference; This represents the initial loss value of the generator network in the human eye gaze direction, which is the difference between the second human eye gaze direction and the corresponding target predicted human eye gaze. L represents the initial loss value of the generator network on head pose information, i.e., the difference between the second head pose information and the corresponding predicted head pose information of the target; p The loss is defined as perceptual loss; r1, r2, r3, and r4 are the corresponding weight values. The initial discrimination network loss value in the human eye gaze direction is the difference between the second human eye gaze direction and the corresponding actual human eye gaze. The initial loss value of the discrimination network in terms of head pose information is the difference between the second head pose information and the corresponding actual head pose information.

[0104] Typically, the generator or initial generator network described above can employ a CNN structure with decoding and encoding, while the discriminator or initial discriminator network uses a WGAN-GP structure. Furthermore, the modified initial discriminator network has three output branches: one outputting a discriminative structure that distinguishes between real and fake images, and the other two outputting gaze estimation and head pose estimation results, respectively. Additionally, the VGG model described above is a pre-trained model on ImageNet; the training process only outputs the perceptual loss and does not update the parameters.

[0105] To achieve the above functions, the gaze redirection device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art will readily recognize that, based on the algorithmic steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0106] This application embodiment also provides a method such as Figure 4 The shown gaze redirection device 40 includes an extraction unit 401 and a gaze adjustment unit 402.

[0107] Extraction unit 401 is configured to extract a first target human eye image, target head posture information, and target human eye gaze direction information from the image to be processed. Gaze adjustment unit 402 is configured to redirect the gaze direction in the first target human eye image based on the target head posture information and the target human eye gaze direction information to obtain a second target human eye image; the gaze direction in the second target human eye image matches the target head posture information and the target human eye gaze direction information.

[0108] In one possible implementation, the gaze adjustment unit 402 is specifically configured to: input head posture information, human eye gaze direction information, and a first target human eye image into a preset model to obtain a second target human eye image; the preset model is used to redirect the human eye gaze in the corresponding human eye image based on the head posture information and human eye gaze direction information to obtain a redirected human eye image; the preset model is trained through multiple sets of different training samples, wherein each set of training samples includes a first sample human eye image, a second sample human eye image, a first human eye gaze direction and a first head posture information corresponding to the first sample human eye image, and a second human eye gaze direction and a second head posture information corresponding to the second sample human eye image.

[0109] In another possible implementation, the gaze adjustment unit 402 includes a preset model. Before inputting head pose information, eye gaze direction information, and a first target eye image into the preset model to obtain a second target eye image, the gaze adjustment unit is further configured to: construct an initial model; wherein the initial model includes an initial generation network; the initial generation network is used to redirect the eye gaze in the image and generate the corresponding eye image; using the first sample eye image, the second eye gaze direction, and the second head pose information from multiple sets of different training samples as input parameters of the initial generation network, and using the second sample eye image corresponding to the second eye gaze direction and the second head pose information as the target image, the network parameters of the initial generation network are adjusted to obtain a target generation network; the matching degree between the eye image output by the target generation network and the corresponding target image meets a preset condition; and the target generation network is determined as the preset model.

[0110] In another possible implementation, the gaze adjustment unit 402 is further configured to: for any parameter adjustment during the process of adjusting the network parameters of the initial generation network, input the first sample human eye image, the second human eye gaze direction, and the second head pose information from each group of training samples into the initial generation network to obtain the first output sample human eye image corresponding to each group of training samples after redirection; input the first output sample human eye image and the corresponding first human eye gaze direction and first head pose information corresponding to each group of training samples into the initial generation network to obtain the second output sample human eye image after redirection; adjust the network parameters of the initial generation network based on the first loss function value corresponding to the first difference degree and the second difference degree; the first difference degree characterizes the difference between each second output sample human eye image and the first sample human eye image corresponding to each group of training samples; the second difference degree characterizes the difference between each first output sample human eye image and the second sample human eye image corresponding to each group of training samples; when the first loss function value of the first difference degree and the second difference degree satisfies the first loss condition, determine the initial generation network adjusted under the network parameters corresponding to the first difference degree and the second difference degree as the target generation network.

[0111] In another possible implementation, the initial model further includes an initial discriminant network, which is used to predict the realism of the image and the corresponding image information. The gaze adjustment unit 402 is specifically configured to: for any parameter adjustment during the adjustment of the network parameters of the initial generator network, input the first output sample human eye image and the corresponding second sample human eye image corresponding to each group of training samples into the initial discriminant network, predict the realism of the first output sample human eye image relative to the second sample human eye image, as well as the corresponding predicted head pose information and predicted human eye gaze direction; the realism characterizes the probability of minimizing the second sample human eye image being judged as a generated image; the second human eye gaze direction and the second head pose information are respectively the difference between the corresponding predicted head pose information and the predicted human eye gaze direction, and the weighted sum between the realism, and are determined as the second loss function value; based on the second loss function value, the network parameters of the initial discriminant network are adjusted; when the second loss function value satisfies the second loss condition, the initial discriminant network adjusted under the network parameters corresponding to the second loss function value is determined as the target discriminant network.

[0112] In another possible implementation, the gaze adjustment unit 402 is further configured to: predict the target realism of the first output sample human eye image relative to the second sample human eye image, as well as the corresponding target predicted head pose information and target predicted human eye gaze direction, based on the target discrimination network; and determine the second difference degree by weighted sum of the differences between the second human eye gaze direction and the second head pose information and the corresponding target predicted head pose information and target predicted human eye gaze, and the negative number of the target realism.

[0113] In another possible implementation, the initial model also includes a VGG network, and the gaze adjustment unit 402 is specifically configured to: input each second output sample human eye image and the first sample human eye image corresponding to each group of training samples into the VGG network to obtain the perceptual loss between each first output sample human eye image and the second sample human eye image corresponding to each group of training samples; determine the second difference based on the weighted sum of the difference between the second human eye gaze direction and the second head posture information and the corresponding target predicted head posture information and the target predicted human eye gaze, and the negative number of the target realism, including: determining the second difference as the weighted sum of the difference between the second human eye gaze direction and the second head posture information and the corresponding target predicted head posture information and the target predicted human eye gaze, the negative number of the target realism, and the perceptual loss.

[0114] In another possible implementation, the extraction unit 401 is specifically configured to: detect target key parts in the image to be processed based on facial key points; configure the detected target key parts into a preset part area of ​​a preset contour template to obtain a target facial image; crop out a first eye image from the target facial image; the target eye image includes the eye area covered by any one eye in the target facial image; mirror the first eye image in the horizontal direction to obtain a second eye image; and determine the first eye image or the second eye image as the first target human eye image.

[0115] Regarding the apparatus in the above embodiments, the specific manner in which each unit module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0116] Figure 5 This is a schematic diagram of an image processing device provided in this application. Figure 5 The image processing device 60 may include at least one processor 601 and a memory 603 for storing processor-executable instructions. The processor 601 is configured to execute the instructions in the memory 603 to implement the gaze redirection method in the following embodiments.

[0117] In addition, the image processing device 60 may also include a communication bus 602, at least one communication interface 604, an input device 606, and an output device 605.

[0118] The processor 601 may be a processor (central processing unit, CPU), a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0119] The communication bus 602 may include a path for transmitting information between the aforementioned components.

[0120] Communication interface 604 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0121] Input device 606 is used to receive input signals and output device 605 is used to output signals.

[0122] The memory 603 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processing unit via a bus. The memory may also be integrated with the processing unit.

[0123] The memory 603 stores instructions for executing the scheme of this application, and the processor 601 controls the execution. The processor 601 executes the instructions stored in the memory 603 to realize the functions of the method of this application.

[0124] In a specific implementation, as one embodiment, the processor 601 may include one or more CPUs, for example... Figure 5 CPU0 and CPU1 in the CPU.

[0125] In a specific implementation, as one example, the image processing device 60 may include multiple processors, such as... Figure 5 Processors 601 and 607 are described herein. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0126] The image processing device, such as Figure 5 The diagram includes a processor 601 and a memory 603 for storing executable instructions of the processor 601; wherein the processor 601 is configured to execute executable instructions to implement the gaze redirection method as described in any of the possible embodiments above. And it can achieve the same technical effect, so to avoid repetition, it will not be described again here.

[0127] This application also provides a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of a control device or an image processing device, enables the control device or image processing device to perform the gaze redirection method as described in any of the possible embodiments above. And it can achieve the same technical effect; to avoid repetition, it will not be described again here.

[0128] This application also provides a computer program product, including a computer program or instructions, which are executed by a processor as a gaze redirection method according to any of the possible implementations described above. The same technical effects can be achieved, and to avoid repetition, further details are omitted here.

[0129] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0130] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A gaze redirection method, characterized in that, The method includes: Extract the first target human eye image, the target head pose information, and the target human eye gaze direction information from the image to be processed; Based on the target head posture information and the target eye gaze direction information, the eye gaze in the first target eye image is redirected and adjusted to obtain a second target eye image; the eye gaze in the second target eye image matches the target head posture information and the target eye gaze direction information.

2. The method according to claim 1, characterized in that, The step of redirecting and adjusting the line of sight in the first target eye image based on the target head posture information and the target eye gaze direction information to obtain a second target eye image includes: The head posture information, the eye gaze direction information, and the first target eye image are input into a preset model to obtain the second target eye image. The preset model is used to redirect the eye gaze in the corresponding eye image based on the head posture information and the eye gaze direction information to obtain the redirected eye image. The preset model is trained through multiple sets of different training samples, wherein each set of training samples includes a first sample eye image, a second sample eye image, the first eye gaze direction and the first head posture information corresponding to the first sample eye image, and the second eye gaze direction and the second head posture information corresponding to the second sample eye image.

3. The method according to claim 2, characterized in that, Before inputting the head pose information, the eye gaze direction information, and the first target eye image into a preset model to obtain the second target eye image, the method further includes: Construct an initial model; wherein the initial model includes an initial generator network; the initial generator network is used to redirect the human eye's gaze in the image and generate a corresponding human eye image; Using the first sample human eye image, the second human eye gaze direction, and the second head pose information from the multiple sets of different training samples as input parameters to the initial generation network, and using the second sample human eye image corresponding to the second human eye gaze direction and the second head pose information as the target image output by the initial generation network, the network parameters of the initial generation network are adjusted to obtain the target generation network; the matching degree between the human eye image output by the target generation network and the corresponding target image meets a preset condition. The target generation network is determined as the preset model.

4. The method according to claim 3, characterized in that, The process of using the first sample human eye image, the second human eye gaze direction, and the second head pose information from the multiple sets of different training samples as input parameters for the initial generation network, and using the second sample human eye image corresponding to the second human eye gaze direction and the second head pose information as the target image output by the initial generation network, and adjusting the network parameters of the initial generation network to obtain the target generation network, includes: For any parameter adjustment during the process of adjusting the network parameters of the initial generation network, the first sample human eye image, the second human eye gaze direction and the second head pose information in each group of training samples are input into the initial generation network to obtain the first output sample human eye image corresponding to each group of training samples after retargeting. The first output sample human eye image corresponding to each group of training samples, along with the corresponding first human eye gaze direction and first head posture information, are input into the initial generation network to obtain the redirected second output sample human eye image. Based on the first loss function values ​​corresponding to the first and second differences, the network parameters of the initial generation network are adjusted; the first difference characterizes the difference between each second output sample human eye image and the first sample human eye image corresponding to each group of training samples; the second difference characterizes the difference between each first output sample human eye image and the second sample human eye image corresponding to each group of training samples. When the first loss function values ​​of the first difference degree and the second difference degree satisfy the first loss condition, the initial generation network adjusted under the network parameters corresponding to the first difference degree and the second difference degree is determined as the target generation network.

5. The method according to claim 4, characterized in that, The initial model further includes an initial discriminant network, which is used to predict the realism of the image and the corresponding image information. The method further includes: For any parameter adjustment during the network parameter adjustment process of the initial generation network, the first output sample human eye image and the corresponding second sample human eye image corresponding to each group of training samples are input into the initial discrimination network to predict the realism of the first output sample human eye image relative to the second sample human eye image, as well as the corresponding predicted head pose information and predicted human eye gaze direction; the realism characterizes the probability of minimizing the second sample human eye image being judged as a generated image; The second loss function value is determined by the weighted sum of the difference between the second human eye gaze direction and the second head posture information, which are respectively the corresponding predicted head posture information and predicted human eye gaze direction, and the realism. Based on the second loss function value, the network parameters of the initial discrimination network are adjusted; When the second loss function value satisfies the second loss condition, the initial discriminant network adjusted under the network parameters corresponding to the second loss function value is determined as the target discriminant network.

6. The method according to claim 5, characterized in that, The method further includes: Based on the target discrimination network, the target realism of the first output sample human eye image relative to the second sample human eye image, as well as the corresponding target predicted head pose information and target predicted human eye gaze direction are predicted. The second difference degree is determined by the weighted sum of the differences between the second human eye gaze direction and the second head posture information and the corresponding target predicted head posture information and target predicted human eye gaze, respectively, and the negative number of the target realism. The initial model further includes a VGG network, and the method further includes: inputting each second output sample human eye image and the first sample human eye image corresponding to each group of training samples into the VGG network to obtain the perceptual loss between each first output sample human eye image and the second sample human eye image corresponding to each group of training samples. The step of determining the second difference degree by weighting the differences between the second human eye gaze direction and the second head posture information and the corresponding target predicted head posture information and target predicted human eye gaze, respectively, and the negative number of the target realism, includes: The second difference degree is determined by weighting the difference between the second human eye gaze direction and the second head posture information and the corresponding target predicted head posture information and target predicted human eye gaze, the negative number of the target realism, and the perceptual loss degree.

7. The method according to any one of claims 1 to 6, characterized in that, The step of extracting the first target human eye image from the image to be processed includes: Based on facial key points, target key parts in the image to be processed are detected; The detected key parts of the target are configured into the preset part area of ​​the preset contour template to obtain the target facial image; A first eye image is cropped from the target facial image; the target eye image includes the eye region covered by any one eye in the target facial image; The first eye image was mirrored in the horizontal direction to obtain the second eye image; The first eye image or the second eye image is determined as the first target human eye image.

8. A gaze redirection device, characterized in that, The device includes: The extraction unit is configured to extract a first target human eye image, target head pose information, and target human eye gaze direction information from the image to be processed. The gaze adjustment unit is configured to redirect and adjust the gaze of the human eye in the first target human eye image based on the target head posture information and the target human eye gaze direction information to obtain a second target human eye image; the gaze of the human eye in the second target human eye image matches the target head posture information and the target human eye gaze direction information.

9. An image processing device, characterized in that, The device is configured to perform the line-of-sight redirection method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the image processing device, the image processing device performs the gaze redirection method as described in any one of claims 1-7.

Images