Method and device for training an image processing model, image processing method and device, electronic device, and computer program

By training an image processing model using environment-matched samples and style transfer processing, the method addresses inefficiencies in gaze correction, enhancing model robustness and reducing computational costs.

JP7592160B2Active Publication Date: 2024-11-29TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023521165
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-01-06
Filing Date
2021-12-23
Publication Date
2024-11-29
Estimated Expiration
2041-12-23

AI Technical Summary

Technical Problem

Existing image processing methods for gaze correction face challenges in handling variations in image size, resolution, viewing angle, lighting, and texture, leading to high computational costs and low efficiency, especially when using deep neural networks, due to the need for large-scale training samples and complex 3D modeling.

Method used

A method and apparatus for training an image processing model that involves obtaining correction target eye images, performing style transfer processing, and training a correction network using samples that match the usage environment, thereby improving training efficiency and reducing computational resources while maintaining accuracy.

Benefits of technology

The approach enhances the training efficiency and generalization ability of image processing models, reducing complexity and cost, and improves the robustness of gaze correction in different environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007592160000010
    Figure 0007592160000010
  • Figure 0007592160000011
    Figure 0007592160000011
  • Figure 0007592160000012
    Figure 0007592160000012
Patent Text Reader

Abstract

The present invention provides a method and device for training an image processing model, an image processing method and device, an electronic device, a computer-readable storage medium, and a computer program product. The method for training an image processing model includes the steps of: obtaining a set of correction target eye images that match an environment in which the image processing model is used; performing style transfer processing on the correction target eye images in the set of correction target eye images using a style transfer network included in the image processing model to obtain target eye images; obtaining training samples that match the environment in which the image processing model is used based on the correction target eye images and the target eye images, where the training samples include target eye images that match different gaze positions; training the correction network included in the image processing model using the training samples that match the environment in which the image processing model is used, obtaining model update parameters that are compatible with the correction network, and generating a trained image processing model based on the model update parameters. The eye images in the environment in which the image processing model is used are corrected by the trained image processing model.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is filed based on the Chinese patent application bearing application number 202110014666.2, filed with the China Patent Office on January 6, 2021, and claims priority to the Chinese patent application, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to image processing technology, and in particular to a method and apparatus for training an image processing model, an image processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]

[0003] In the video processing process, gaze correction is processing a photo containing the eyes of an object (e.g., an image of a real person, an animal, an animated character, etc.), and can change the gaze direction of the eyes in the video. Taking the object as an example of a real person, gaze correction has real value and a wide foreground in communication scenarios such as video calls. However, when collecting images or videos containing eyes, the images or videos vary greatly in aspects such as size, resolution, viewing angle, lighting, texture, occlusion, etc., so that images rendered based on 3D modeling usually have a large difference from real eye images in terms of texture and material, and are not natural enough, and the hardware cost of realizing gaze correction using 3D equipment is high, and a large number of training samples are required to realize gaze correction by a deep neural network, and sample collection and annotation are required, which results in low model training efficiency and high consumption of related computing resources. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a method and device for training an image processing model, an image processing method and device, an electronic device, a computer-readable storage medium, and a computer program product that can improve the training efficiency of an image processing model and save associated computing resources while ensuring the classification accuracy of the image processing model.

[0005] The technical solutions of the embodiments of the present application are realized as follows.

[0006] An embodiment of the present application provides a method for training an image processing model, the method comprising: Obtaining a set of correction target eye images that match the environment in which the image processing model is used; performing a style transfer process on the correction target eye image in the correction target eye image set by a style transfer network included in the image processing model to obtain a target eye image; obtaining a training sample matching a usage environment of the image processing model based on the correction target eye image and the target eye image, where the training sample includes target eye images matching different gaze positions; training a correction network included in the image processing model using a training sample that matches a usage environment of the image processing model, obtaining model update parameters adapted to the correction network, and generating a trained image processing model based on the model update parameters; Here, the eye image in the use environment is corrected by the trained image processing model.

[0007] An embodiment of the present application provides an image processing method, the method comprising: acquiring a facial image of a target subject; determining corresponding correction target eye images based on the facial images of the target subject, where the correction target eye images include a left eye image and a right eye image of the target subject; performing a correction process on the correction target eye image using an image processing model to obtain a corrected eye image; replacing a correction target eye image in the face image with the corrected eye image; Here, the image processing model is obtained by training based on the above-mentioned image processing model training method.

[0008] An embodiment of the present application provides an apparatus for training an image processing model, the apparatus comprising: A first information transmission module configured to obtain a correction target eye image set that matches the use environment of the image processing model; a training module configured to perform a style transfer process on the correction target eye image in the correction target eye image set by a style transfer network included in the image processing model to obtain a target eye image; The training module is configured to obtain a training sample matching a usage environment of the image processing model based on the correction target eye image and the target eye image, where the training sample includes target eye images matching different gaze positions; The training module is configured to train a correction network included in the image processing model using training samples matching a usage environment of the image processing model, obtain model update parameters adapted to the correction network, and generate a trained image processing model based on the model update parameters, wherein eye images in the usage environment are corrected by the trained image processing model.

[0009] An embodiment of the present application further provides an image processing apparatus, the apparatus comprising: A second information transmission module configured to acquire a facial image of a target subject; and an information processing module configured to determine a corresponding correction target eye image based on the face image of the target subject, where the correction target eye image includes a left eye image and a right eye image of the target subject; The information processing module is configured to perform a correction process on the correction target eye image by an image processing model to obtain a corrected eye image; the information processing module is configured to replace a correction target eye image in the face image with the corrected eye image; Here, the image processing model is obtained by training based on a training method for an image processing model.

[0010] An embodiment of the present application provides an electronic device, the electronic device comprising: A memory storing executable instructions; A processor for performing the above-mentioned method for training an image processing model or the above-mentioned image processing method when executing executable instructions stored in the memory.

[0011] An embodiment of the present application provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform a method for training an image processing model or an image processing method according to an embodiment of the present application. An embodiment of the present application provides a computer program product including a computer program or instructions for causing a computer to execute the above-mentioned method for training an image processing model or the above-mentioned image processing method.

[0012] The embodiments of the present application have the following beneficial effects: Based on the eye image to be corrected and the target eye image, a training sample matching the usage environment of the image processing model is obtained, and the correction network included in the image processing model is trained using the training sample. The eye image in the usage environment is corrected using the trained image processing model, so as to ensure the discrimination accuracy of the image processing model while improving the training efficiency of the image processing model and reducing the complexity of image processing, thereby reducing the associated computing resources, and while taking into account the training accuracy, enhancing the generalization ability and data processing ability of the image processing model, adapting to different data processing environments, and enhancing the robustness of the image processing model. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 is an exemplary structural diagram of an image processing model according to an embodiment of the present application; [Diagram 2] FIG. 2 is an exemplary structural diagram of an electronic device according to an embodiment of the present application; [Diagram 3] 1 is an exemplary flowchart of a method for training an image processing model according to the present application. [Figure 4] FIG. 2 is a schematic diagram of facial image acquisition according to an embodiment of the present application. [Diagram 5] FIG. 2 is a schematic diagram of the style transfer effect of the image processing model according to the present application. [Figure 6] 1 is an exemplary flowchart of a method for training an image processing model according to the present application. [Figure 7] 1 is an exemplary flowchart of an image processing method according to the present application. [Figure 8] FIG. 2 is a schematic diagram of a usage scenario of the image processing method according to an embodiment of the present application; [Figure 9] FIG. 2 is a schematic diagram of a usage scenario of the image processing method according to an embodiment of the present application; [Figure 10] FIG. 2 is a schematic comparison diagram of the image processing effect of the embodiments of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is hereinafter described in more detail with reference to the drawings, and the described embodiments should not be regarded as limitations on the present application, and all other embodiments obtained by those skilled in the art without creative efforts shall be included in the protection scope of the present application.

[0015] In the following description, the term "some embodiments" refers to a subset of all possible embodiments, and it will be understood that the terms "some embodiments" may refer to the same or different subsets of all possible embodiments, which may be combined with each other without conflict.

[0016] Before describing the present embodiment in more detail, the nouns and terms related to the present embodiment are explained, and the nouns and terms related to the present embodiment are applicable to the following description.

[0017] 1) The term “in response to” is used to express a condition or state upon which an operation being performed depends, and when the dependent condition or state is met, the operation or operations performed may be in real time or may have a configured delay, and unless otherwise specified, the operations performed have no time order of execution restrictions.

[0018] 2) A client is a carrier that performs a specific function on a terminal. For example, a mobile client (APP) is a carrier in a mobile terminal that performs a specific function, such as the function of online live (video streaming) or the function of broadcasting online videos.

[0019] 3) Convolutional Neural Networks (CNNs) are feed-forward neural networks that include convolutional calculations and have a deep structure, and are one of the representative algorithms of deep learning. Convolutional neural networks have the ability of representation learning and can perform shift-invariant classification of input information according to their hierarchical structure.

[0020] 4) Model training is a method of performing multi-class learning on an image data set. The model is constructed using deep learning frameworks such as Tensor Flow and Torch, and uses multiple layers of neural networks such as CNN to combine and configure image processing models.

[0021] 5) Neural network (NN) or artificial neural network (ANN), also known as a neural network or nerve net, is a mathematical or computational model in the fields of machine learning and cognitive science that mimics the structure and function of biological neural networks (animal central nervous systems, particularly the brains) and is used to estimate and approximate functions.

[0022] 6) A component is a functional module of a mini-program view, also known as a front-end component. Components such as buttons, titles, tables, sidebars, content and footers within a screen contain modularized code so that they can be used repeatedly on different screens of a mini-program.

[0023] 7) A mini program is a program developed based on a front-end oriented language (e.g., JavaScript) that realizes a service on a HyperText Markup Language (HTML) screen, and is software that is downloaded by a client (e.g., a browser or any client incorporating a browser core) via a network (e.g., the Internet) and analyzed and executed in the client's browser environment, and does not need to be installed on the client. For example, a voice command may be used to wake up a mini program on a terminal, allowing a social network client to download and execute mini programs for various services such as image editing and eye image correction.

[0024] Before introducing the image processing method according to the embodiment of the present application, we first describe an image processing model in an image processing scenario in the related art, through which the eye image of the target user can be corrected to achieve the effect of gaze correction.

[0025] In this process, the eye image is processed by a graphic-based gaze correction method and a pixel distortion-based gaze correction method. For the first type, the graphic-based gaze correction method mainly uses a 3D eye model with artificial texture to simulate the continuous movement of the eye and head, and uses a dynamic and controllable eye model to draw the eye image by geometrically large-scale rendering. However, the eye image synthesized by this method has a large difference from the real eye image. At the same time, a 3D model of the eye is required when applied, but due to the high cost of constructing the 3D model, such a method has limitations in practical application. For the second type, the distortion-based gaze correction method predicts the distortion flow field by learning the distortion function, and thereby directly generates the gaze-corrected image from the original eye image. For example, there is a depth forward feedback system that combines the operation principles of two-stage processing of coarseness and fineness, image distortion, and intensity correction. The eye distortion field method, which is implemented by a random forest predictor and runs in real time on a central processing unit (CPU), can use eye images with different gaze directions and head poses to synthesize more realistic images because the distortion function is pose-specific, and has dealt with changes in head pose and gaze angle in practical applications. However, eye images usually have complex texture, lighting, occlusion, and other conditions, and the effects of these specific factors are difficult to realize in a global correction operation. In addition, images rendered based on 3D modeling usually have a large difference from real eye images in terms of texture and material, resulting in a lack of naturalness. The hardware cost of realizing gaze correction using 3D equipment is high, and a large number of training samples are required to realize gaze correction using a deep neural network, which requires sample collection and annotation, resulting in increased usage costs.

[0026] In order to solve the above deficiencies, the embodiments of the present application provide a method and device for training an image processing model, an image processing method and device, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the training efficiency of the image processing model while ensuring the identification accuracy of the image processing model, save related computing resources, and reduce the complexity of image processing, and can enhance the generalization ability and data processing ability of the image processing model while taking into account the training accuracy, adapt to different data processing environments, and enhance the robustness of the image processing model.

[0027] Fig. 1 is a schematic diagram of a usage scenario of training an image processing model according to an embodiment of the present application, referring to Fig. 1, a client or mini-program having an image processing function is arranged in a terminal (including terminal 10-1 and terminal 10-2), the image processing device in the embodiment of the present application can be a server 200, and the one displaying the image processing result of the image processing device can be a terminal running various clients, and the two are connected by a network 300, where the network 300 can be a wide area network or a local area network, or a combination of both, and data transmission is performed by a wireless link. The terminal 10-1 submits an eye image to be processed, and the image processing device realizes image processing in response to a trigger operation on an image conversion function option, and the terminal 10-1 obtains and presents a corrected eye image.

[0028] In an embodiment of the present application, a video client can be executed on the terminal 10-2, and the video client can submit a corresponding image processing request to the server according to the correction target eye 120 and the target face 110 indicated by the user through various human-computer interaction (HCI) ways (such as gestures, voice, etc.) on the playback interface, and when the executable instructions in the storage medium of the server 200 are executed by the processor, the image processing method according to the present application is realized to achieve the corresponding eye image correction effect. For example, the above image processing process can be transferred to the server, and the hardware resources of the server depending on the video conference APP can re-encode the replaced images of different frames to form a video with eye image correction effect, which can be obtained by the user through an instant messaging client applet or shared with different application processes of the user terminal 10-1.

[0029] As an example, the server 200 is used to obtain a facial image of a target user collected by a terminal in response to an eye correction request, and configure an image processing model for determining a corresponding correction target eye image based on the facial image of the target user, where the correction target eye image includes the left eye image and the right eye image of the target user, and process the correction target eye image through the image processing model to obtain a corrected eye image, and replace the correction target eye image in the facial image with the corrected eye image. Of course, before processing different facial images through the image processing model to generate corresponding classification results, the image processing model needs to be further trained, specifically, The method includes the steps of obtaining a set of correction target eye images that match the usage environment of the image processing model, processing different correction target eye images in the correction target eye image set by a style transfer network of the image processing model respectively to form a target eye image, obtaining training samples that match the usage environment of the image processing model based on the correction target eye images and the target eye images, where the training samples include user eye images that match different gaze positions, and training a correction network of the image processing model by a training sample set that matches the usage environment of the image processing model, obtaining model update parameters adapted to the correction network, generating a trained image processing model based on the model update parameters, and correcting the eye images in the usage environment by the image processing model.

[0030] In an embodiment of the present application, a video client can be executed on the terminal 10-2, and the video client can obtain the correction target eye 120 and the target face 110 indicated by the user through various human-computer interaction methods (such as gestures, voice, etc.) on the playback interface, and generate a corresponding image processing request. In this way, when the executable instructions in the storage medium of the terminal 10-2 are executed by the processor, the image processing method according to the present application is realized, the corresponding eye image correction effect is achieved, and a video with eye image correction effect is formed. The video can be retrieved by the user in the mini-program of the instant messaging client, or shared to different application processes of the user terminal 10-1.

[0031] As an example, the terminal 10-2 is used to obtain a facial image of a target user collected by the terminal in response to an eye correction request, and configure an image processing model to determine a corresponding eye image to be corrected based on the facial image of the target user, where the eye image to be corrected includes a left eye image and a right eye image of the target user, and process the eye image to be corrected by the image processing model to obtain a corrected eye image, and replace the eye image to be corrected in the facial image with the corrected eye image. Of course, before the image processing model processes different facial images to generate corresponding classification results, the image processing model needs to be further trained, specifically, The method includes the steps of obtaining a set of correction target eye images that match the usage environment of the image processing model, processing different correction target eye images in the correction target eye image set by a style transfer network of the image processing model to form a target eye image, obtaining training samples that match the usage environment of the image processing model based on the correction target eye images and the target eye images, where the training samples include user eye images that match different gaze positions, and training a correction network of the image processing model with the training sample set that matches the usage environment of the image processing model, and correcting the eye images in the usage environment by the image processing model.

[0032] Of course, the image processing device according to the embodiment of the present application can be applied to video conference activities through virtual resources or physical resources, or payment environments by physical video conference resources (including but not limited to image processing environments for payment of various physical video conference resources), or use environments of information exchange by social software, and usually processes video conference information from different data sources when performing video conference activities with various kinds of physical video conference resources or payment of virtual resources, and finally presents the detection result matching the target user on the user interface (UI), and judges whether the detected image is the user's biometric face image or attack information. Correct the eye image in the current display interface to obtain a corrected eye image, where the corrected eye image, the face classification result obtained from the corresponding face image, or the eye image detection result can be called by the application program.

[0033] Here, the image processing method according to the embodiment of the present application is realized based on artificial intelligence, which is a theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate and extend human intelligence, sense the environment, obtain knowledge, and use the knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science that aims to understand the essence of intelligence and produce new intelligent machines that can react in a manner similar to human intelligence. Artificial intelligence studies the design principles and realization methods of various intelligent machines, and equips machines with the functions of sensing, reasoning, and decision-making.

[0034] Artificial intelligence technology is a comprehensive academic field that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence usually include sensors, inherent artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interactive systems, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, voice processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0035] In the embodiments of the present application, the artificial intelligence software technology mainly involved includes the above-mentioned speech processing technology and machine learning, etc. For example, it may relate to automatic speech recognition (ASR) technology in speech technology, including speech signal preprocessing, speech signal frequency analyzing, speech signal feature extraction, speech signal feature matching / recognition, speech training, etc.

[0036] For example, it may relate to machine learning (ML), which is a multidisciplinary field that involves multiple fields such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. Machine learning is specialized in studying how computers simulate or realize human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continue improving their own performance. Machine learning is the core of artificial intelligence and is a basic method for making computers intelligent, and is applied to various fields of artificial intelligence. Machine learning usually includes techniques such as deep learning, which includes artificial neural networks such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and deep neural networks (DNNs).

[0037] The structure of the image processing device of the embodiment of the present application will be described in detail below, and the image processing device can be implemented in various forms, for example, it can be a dedicated terminal having the processing function of the image processing device, or it can be a server having the processing function of the image processing device, for example, it can be the server 200 in the above Fig. 1. Fig. 2 is an exemplary structural diagram of the configuration of an electronic device according to the embodiment of the present application, and it can be understood that Fig. 2 shows only an exemplary structure, not the entire structure of the image processing device and the training device for the image processing model, and the partial structure or the entire structure shown in Fig. 2 can be implemented as needed.

[0038] The image processing device and image processing model training device according to the embodiment of the present application includes at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. Each component in the image processing device is coupled via a bus system 205. It can be understood that the bus system 205 is used to realize the connection communication between these components. In addition to a data bus, the bus system 205 includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are represented as the bus system 205 in FIG. 2.

[0039] Here, the user interface 203 may include a display, a keyboard, a mouse, a trackpad, a click wheel, keys, buttons, a haptic pad or a touch screen, or the like.

[0040] It can be understood that the memory 202 may be a volatile memory or a non-volatile memory, and may include both volatile and non-volatile memory. The memory 202 of the embodiment of the present application can store data to support the operation of a terminal (e.g., 10-1). Examples of these data include any computer program operated on the terminal (e.g., 10-1), such as an operating system and application programs. Here, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, and is used to realize various basic services and process hardware-based tasks. The application programs may include various application programs.

[0041] In some embodiments, the image processing device and the training device for the image processing model according to the embodiments of the present application can be realized by combining software and hardware. For example, the image processing device according to the embodiments of the present application can be a processor adopting the form of a hardware decoding processor, which is programmed to execute the image processing method according to the embodiments of the present application, and the training device for the image processing model can be a processor adopting the form of a hardware decoding processor, which is programmed to execute the training method for the image processing model according to the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements.

[0042] As an example of an image processing device and an image processing model training device according to an embodiment of the present application being implemented in a manner combining software and hardware, the image processing device and the image processing model training device according to an embodiment of the present application may be a combination of software modules executed directly by the processor 201, or the software modules may be located in a storage medium, the storage medium being located in the memory 202, and the processor 201 may read executable instructions contained in the software modules in the memory 202 and combine them with the required hardware (e.g., including the processor 201 and other components connected to the bus 205) to realize the image processing method and the image processing model training method according to the embodiment of the present application.

[0043] As an example, the processor 201 may be an integrated circuit chip having signal processing capabilities, such as a general purpose processor, a digital signal processor (DSP), or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., where the general purpose processor may be a microprocessor or any conventional processor, etc.

[0044] As an example of the image processing device and the image processing model training device according to the embodiments of the present application being implemented using hardware, the device according to the embodiments of the present application can be implemented directly using a processor 201 in the form of a hardware decoding processor, and for example, the image processing method and the image processing model training method according to the embodiments of the present application can be executed by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic elements.

[0045] The memory 202 of the embodiment of the present application stores various types of data to support the operation of the image processing device and the image processing model training device. Examples of these data include any executable instructions operated on the image processing device, such as executable instructions, and programs implementing the image processing method and the image processing model training method of the embodiment of the present application can be included in the executable instructions.

[0046] In some other embodiments, the image processing device and the training device for the image processing model according to the embodiments of the present application can be realized in the form of software, and Fig. 2 shows the image processing device and the training device for the image processing model stored in the memory 202, which may be software in the form of programs, plug-ins, etc., and includes a series of modules, and an example of the program stored in the memory 202 includes the image processing device and the training device for the image processing model, where the image processing device includes a first information transmission module 2081 and a training module 2082, and the training device for the image processing model includes a second information transmission module 2083 and an information processing module 2084. When the software modules in the image processing device and the training device for the image processing model are read and executed by the processor 201, the image processing method and the training method for the image processing model according to the embodiments of the present application are realized.

[0047] Referring to Fig. 3, Fig. 3 is an exemplary flow chart of the training method of the image processing model according to the present application, it can be understood that the steps shown in Fig. 3 can be performed by various electronic devices that execute the training device of the image processing model, for example, a dedicated terminal with eye information detection function or video conference function, a server or server cluster with the training function of the image processing model, and realize the training and deployment of the image processing model that is suitable for different video conference scenarios. The steps shown in Fig. 3 are described below.

[0048] In step 301, a training device for an image processing model obtains a set of correction target eye images that match the usage environment of the image processing model.

[0049] Here, the correction target eye image set includes target face images in different regions, where the target may be a real person, an animal, or a virtual person, an animal, such as an anime character, etc.

[0050] In the embodiment of the present application, the step of obtaining a correction target eye image set that matches the use environment of the image processing model can be realized in the following manner: obtain a face image collected by a collection device (such as a camera, a terminal with a shooting function, etc.) in the use environment of the image processing model, perform image enhancement processing on the face image, obtain a processing result of the image enhancement, determine face position coordinates in the processing result of the image enhancement, crop the face image including the complete eye image according to the face position coordinates, perform depth processing on the face image including the complete eye image by a depth processing network included in the image processing model, obtain a depth map of the face image, use the depth map as a correction target eye image that matches the use environment of the image processing model, and use the set of multiple correction target eye images as a correction target eye image set.

[0051] Here, referring to FIG. 4, FIG. 4 is a schematic diagram of the acquisition of face images in the embodiment of the present application, taking the target as an example of a real user, after acquiring a user image collected by a collection device, first, frame the area 401 where the user's face is located by face detection technology, and enlarge this area to different multiples to obtain more background content 402, and crop the face image containing the background content to obtain a cropped face image 403, which can be cropped in the following manner, for example: adopt an image processing algorithm (e.g., a five-sense positioning algorithm) to mark the position coordinates of facial feature points such as the eyes, mouth, and nose of the face, and cut out the face image of the background content according to the detected face position coordinates; and then calculate the cropped real person's face by a depth estimation network to obtain a depth map 404 corresponding to the face, where the real person photo in the embodiment of the present application has a depth map, and the depth map corresponding to the attack photo is a black bottom map. Image information based class depth learning network techniques include, but are not limited to, LeNet, AlexNet, VGG, Inception series of networks, ResNet and DenseNet, and extract traditional features from an image or ROI, including, but not limited to, grayscale level based features such as mean and variance, distribution histogram based features, relational matrix based features such as GLCM and GLRLM, or Fourier transformed image based signal features.

[0052] In step 302, the training device of the image processing model performs style transfer processing on the correction target eye image in the correction target eye image set through a style transfer network included in the image processing model to obtain a target eye image.

[0053] In an embodiment of the present application, the step of processing the correction target eye images in the correction target eye image set by the style transfer network included in the image processing model to form a target eye image can be realized in the following manner.

[0054] Based on the usage environment of the image processing model, model parameters of a reference style transfer network corresponding to the usage environment of the image processing model are determined, and based on the model parameters of the reference style transfer network, model parameters of the style transfer network included in the image processing model are adjusted, and style transfer processing is performed on the correction target eye image in the correction target eye image set by the adjusted style transfer network to obtain a target eye image, where the style features of the target eye image match the gaze position of the target object. Here, the 3D modeling-based eye data collection utilizes 3D modeling to construct a synthetic eye, collects eye images by changing the gaze position of the synthetic eye, and generates a correction target and corrected synthetic eye data pair (S i , S t ), where S i is the synthesized correction target eye image, and S t is the synthesized corrected eye image.

[0055] Here, referring to FIG. 5, FIG. 5 is a schematic diagram of the style transfer effect of the image processing model according to the embodiment of the present application. When applied to other entertainment function apps or mini programs, the model parameters of the style transfer network can be freely adjusted based on the user's usage habits, for example, the model parameters of the style transfer network can be the display effect parameters of the eye image effect of the animation entertainment. In some embodiments, the initialization process includes, in the data collection stage, for a set of sample images (a set composed of sample images) including the eye images to be processed, cutting out face images including the eye images to be processed from each sample image, and matching each cut-out face image based on the feature points of the eye images to be processed. This allows the encoder to accurately learn the face style features and gaze angle, and inputs the collected synthetic eye data (synthetic eyes shown in FIG. 5) into the style transfer neural network F to obtain a large number of eye images with real styles (real style eyes shown in FIG. 5), the calculation formula of which is as shown in Equation (1) to Equation (2):

number

number

number

number

[0056] In some embodiments, the initialization process may include the steps of performing a random enhancement process on the facial image, performing a random distortion process on the random enhancement processed facial image to obtain a distorted facial image, cropping a sensory image of a processed eye image of the distorted facial image, and performing a scaling process on the cropped sensory image to obtain any one of the correction target eye images in the correction target eye image set, where any one of the correction target eye images conforms to an input resolution supported by an encoder in the image processing model.

[0057] As an example, the step of performing random augmentation processing on the face image may adopt the following scheme: randomly rotate the face image around its center, randomly scale the randomly rotated face image, and randomly translate the randomly scaled face image.

[0058] As an example, the step of performing random distortion processing on the random enhancement processed face image can adopt the following manner: inserting noise into the random enhancement processed face image, the noise including at least one of fixed noise (i.e., fixed color value) and dynamic noise (i.e., changing color value);

[0059] Regarding inserting dynamic noise into a randomly enhanced facial image, the randomly enhanced facial image is gridded, coordinates are assigned to at least some of the nodes, noise is added to the coordinates based on a Gaussian distribution, image interpolation (e.g., linear or nonlinear interpolation) is performed in areas other than the nodes, and the size of the facial image is enlarged; when linear interpolation is used, the value of the interpolated pixel is positively correlated with the value of the coordinate.

[0060] In step 303, the training device for the image processing model obtains a training sample matching the usage environment of the image processing model according to the correction target eye image and the target eye image.

[0061] Here, the training samples contain target eye images matching different gaze positions.

[0062] In step 304, the image processing model training device trains a correction network included in the image processing model using training samples that match the usage environment of the image processing model, obtains model update parameters that are compatible with the correction network, and generates a trained image processing model based on the model update parameters.

[0063] In this way, the trained image processing model can correct the eye image in the environment of use.

[0064] As shown in Figure 5, the eye image to be corrected (i.e., the real eye to be corrected) and the target eye image (the real-style eye to be corrected) are used as training samples to match the usage environment of the image processing model, and the correction network (the gaze correction network shown in Figure 5) is trained by the real eye to be corrected and the real-style eye to be corrected.

[0065] Continuing to refer to Fig. 6, Fig. 6 is an exemplary flow chart of a method for training an image processing model according to an embodiment of the present application, it can be understood that the steps shown in Fig. 6 can be performed by various electronic devices that execute the training device for the image processing model, such as a dedicated terminal with eye information detection function, a server or server cluster with image processing model training function, and train and deploy image processing models that are suitable for different video conference image processing scenarios. The steps shown in Fig. 6 are described below.

[0066] In step 601, a training sample set is processed by an encoder and a decoder of a rectification network included in an image processing model to obtain initial parameters of the encoder and the decoder of the rectification network.

[0067] In step 602, based on the initial parameters of the encoder and decoder of the rectification network, a new training sample set is processed to obtain updated parameters of the encoder and decoder of the rectification network.

[0068] Here, when determining the update parameters, different training samples in the new training sample set can be substituted into the loss function corresponding to the encoder and decoder of the correction network, and if the loss function satisfies the corresponding convergence condition, it is determined that the encoder and decoder of the correction network correspond to the update parameters.

[0069] In step 603, based on the updated parameters of the encoder and decoder of the correction network, the parameters of the encoder and decoder of the correction network are iteratively updated by the training sample set.

[0070] In step 604, it is determined whether a convergence condition has been reached, and the iterative updates are stopped when the convergence condition has been reached.

[0071] Here, different training samples in the new training sample set can be substituted into the loss functions corresponding to the encoder and decoder of the correction network, and if the loss function meets the corresponding convergence condition, the encoder and decoder of the correction network are determined to correspond to the updated parameters, and the real-style eye image obtained by the conversion is used to train the gaze correction network G, and the training process can be seen in equation (3).

[0072]

number

number

number

number

[0073] When optimizing the parameters of the correction network, a small amount of real-world data pairs (I i ,I t ) and the correction network obtained in the above embodiment can be used to perform optimization training, see equation (4) for optimization.

[0074]

number

[0075] In practical application, the technical solution of the embodiments of the present application can be realized by an APP with image processing function, and at the same time, the technical solution of the embodiments of the present application can be realized by an instant application mini-program, so that the image processing model result can be called up in other video conferencing programs, and thus, when a user changes terminals through the video conferencing cloud server network, the faces of different eye objects can be quickly detected through the image processing model in the cloud server network.

[0076] Specifically, the target user identifier, the model parameters of the image processing model, and the video conference scenario identifier are sent to the cloud server network, and when the corresponding image processing process is triggered, the target user identifier, the model parameters of the image processing model, and the video conference scenario identifier are obtained by the cloud server network and used in the corresponding image processing process to detect and correct the target target eyes in the video conference application scenario, and facilitate the eye detection model to identify the target target's eyes.

[0077] Here, the embodiments of the present application can be realized in combination with cloud technology, and cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize computing, storage, processing, and sharing of data, and can also be understood as a collective term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the application of cloud computing business mode. The back-end services of technical network systems require a large amount of computing and storage resources, such as video sites, photo sites, and more portal sites, and therefore cloud technology needs to be supported by cloud computing.

[0078] It should be explained that cloud computing is a kind of computing mode, which distributes computing tasks to a resource pool composed of a large number of computers, and allows various application systems to obtain computing power, storage space and information services as needed. The network that provides resources is called "cloud". From the user's perspective, the resources in the "cloud" can be expanded infinitely, can be obtained at any time, can be used on demand, can be expanded at any time, and can be paid for according to usage. As the basic function provider of cloud computing, a cloud computing resource pool platform is established, which is abbreviated as cloud platform, and is usually called infrastructure as a service (IaaS: Infrastructure as a Service), and various types of virtual resources are placed in the resource pool for external customers to select and use. The cloud computing resource pool mainly includes computing equipment (which can be a virtualization device, including an operating system), memory equipment and network equipment.

[0079] In combination with the above Fig. 1, the training method of the image processing model according to the embodiment of the present application can be realized by a corresponding cloud device, for example, a terminal (including a terminal 10-1 and a terminal 10-2) connects to a server 200 located in the cloud through a network 300, and the network 300 can be a wide area network or a local area network, or a combination of both. It should be explained that the server 200 can be a physical device or a virtualized device.

[0080] Continuing with reference to Fig. 7, Fig. 7 is an exemplary flow chart of an image processing method according to an embodiment of the present application. It should be understood that the steps shown in Fig. 7 can be performed by various electronic devices equipped with image processing devices, such as mini-programs with image processing capabilities, communication terminals with image processing capabilities, or video conference terminals, to enable training and deployment of image processing models adapted to different video conference image processing scenarios. The steps shown in Fig. 7 are described below.

[0081] In step 701, an eye image correction function option is indicated in a human-computer interaction interface, and a trigger operation for the eye image correction function option is received.

[0082] Now, referring to FIG. 8, FIG. 8 is a schematic diagram of a usage scenario of an image processing method according to an embodiment of the present application, in which a correction target eye image including a target object is obtained and presented in response to a trigger operation for an eye image correction function option 801 in the execution interface of a video conferencing client (or mini program).

[0083] and generating and presenting a target eye image of a target eye image template in response to a conversion decision operation triggered according to the correction target eye image, where the target eye image of the target eye image template is an image generated by replacing the eye features in the correction target eye image with the eye features in the target eye image template. Because the image collection angles at which the user of the video conference process is located are different, the model parameters of the image processing model adapted to different image collection angles are all different, in order to reduce the model adjustment time, a fixed model parameter can be set at a fixed image collection angle, for example, the model parameters of the image processing model when the image collection angle is 90 degrees or 75 degrees can be stored in a corresponding storage medium, and when the target eye image of the target eye image template is presented, a template selection command is received, and the model parameters of the image processing model can be quickly adjusted, thereby reducing the waiting time.

[0084] At step 702, a facial image of a target subject collected by a collection device in response to an eye correction request is obtained.

[0085] In step 703, a corresponding correction target eye image is determined based on the target face image.

[0086] Now, referring to Figure 9, Figure 9 is a schematic diagram of a usage scenario of the image processing method of an embodiment of the present application, where the correction target eye image (correction target eye 901 shown in Figure 9) may include the left eye image and the right eye image of the target object, or may include only the left eye image or the right eye image. Specifically, since the image collection angle at which the object is located is different and the ambient light intensity of the eye image is different, when the target image collection angle is 90 degrees or 75 degrees, only the left eye image or the right eye image is collected and adaptively adjusted to realize gaze correction, so as to reduce the calculation amount of the image processing model of the video conferencing mini program and shorten the waiting time.

[0087] In step 704, the correction target eye image is processed by an image processing model to obtain a corrected eye image.

[0088] In an embodiment of the present application, the human-computer interaction interface further presents an image sharing function option for sharing a facial image, and in response to a trigger operation on the image sharing function option, the adjusted facial image of the target subject can be shared, where the image sharing function option can be associated with a default sharing path, such as a function option for sharing to instant messaging software or social software, and in response to a trigger on the facial image sharing function option, a sharing interface including at least two sharing path selection options can be presented, and in response to a sharing path selection operation based on the trigger on the sharing interface, the facial image can be shared to a different social application process or an image capture (or screen recording) application process via the selected sharing path to share or crop the generated new image.

[0089] As shown in Fig. 9, a gaze correction network included in the image processing model performs a correction process on a correction target eye 901 to obtain a corrected eye 902 (i.e., a corrected eye image). It should be noted that, after triggering an eye image correction function option shown in Fig. 8, the embodiment of the present application first processes the correction target eye image by a style transfer network of the image processing model to form a target eye image, where the style features of the target eye image match the gaze position of the target object, and then performs a correction process on the target eye image by the gaze correction network to obtain a corrected eye.

[0090] In step 705, the eye image to be corrected in the face image is replaced with the corrected eye image to obtain an adjusted face image.

[0091] Referring to Fig. 10, Fig. 10 is a comparative schematic diagram of the image processing effect of the embodiment of the present application, Fig. 1001 is the processing target eye image (i.e., input image), Fig. 1002 is the eye image synthesized based on 3D modeling, Fig. 1003 is the eye image generated by the gaze correction network after training learning with limited real eye data (i.e., eye image generated without transfer learning), Fig. 1004 is the eye image generated based on the transfer learning and gaze correction network of the embodiment of the present application. As can be seen, compared with the 3D modeling technical solution, the embodiment of the present application can generate eye images that are more realistic and natural and consistent with the style of the input image, and compared with the deep learning technical solution without transfer learning, the embodiment of the present application has a better gaze correction effect.

[0092] Therefore, the embodiments of the present application have the following beneficial effects:

[0093] 1. A correction network for the image processing model is trained using a training sample set that matches the usage environment of the image processing model, model update parameters that are suitable for the correction network are obtained, and a trained image processing model is generated based on the model update parameters, so that the eye image in the usage environment can be corrected by the image processing model, and the training efficiency of the image processing model can be improved while ensuring the identification accuracy of the image processing model, and the complexity of image processing can be reduced, and the generalization function and data processing function of the image processing model can be improved while taking into account the training accuracy.

[0094] 2. By deploying the trained image processing model, it can be applied to different image processing environments, which can enhance the robustness of the image processing model, reduce the usage cost, and improve the user experience.

[0095] The following describes in detail the functions of each software module in the training device for the image processing model.

[0096] The first information transmission module 2081 is configured to obtain a set of correction target eye images matching a usage environment of an image processing model, and the training module 2082 is configured to process the correction target eye images in the correction target eye image set by a style transfer network included in the image processing model to obtain a target eye image, and the training module 2082 is configured to obtain training samples matching the usage environment of the image processing model based on the correction target eye images and the target eye images, where the training samples include target eye images matching different gaze positions, and the training module 2082 is configured to train a correction network included in the image processing model using the training samples matching the usage environment of the image processing model, obtain model update parameters compatible with the correction network, and generate a trained image processing model based on the model update parameters, where the eye images in the usage environment are corrected by the trained image processing model.

[0097] In some embodiments, the first information transmission module 2081 is further configured to obtain a facial image collected by a collection device, which is located in an environment in which the image processing model is used, perform image enhancement processing on the facial image, obtain a processing result of image enhancement, determine facial position coordinates in the processing result of image enhancement, crop a facial image including a complete eye image based on the facial position coordinates, perform depth processing on the facial image including the complete eye image by a depth processing network included in the image processing model, obtain a depth map of the facial image, use the depth map as a correction target eye image matching the environment in which the image processing model is used, and use a set of a plurality of the correction target eye images as the correction target eye image set.

[0098] In some embodiments, the training module 2082 is further configured to determine model parameters of a reference style transfer network corresponding to the usage environment of the image processing model based on the usage environment of the image processing model, adjust model parameters of a style transfer network included in the image processing model based on the model parameters of the reference style transfer network, and perform style transfer processing on the correction target eye image in the correction target eye image set by the adjusted style transfer network to obtain a target eye image, where the style features of the target eye image match the gaze position of the target target.

[0099] In some embodiments, the training module 2082 is further configured to process the training samples using an encoder and a decoder of a rectification network included in the image processing model, obtain initial parameters of the encoder and the decoder of the rectification network, process the training samples based on the initial parameters of the encoder and the decoder of the rectification network, obtain updated parameters of the encoder and the decoder of the rectification network, iteratively update the parameters of the encoder and the decoder of the rectification network using the training sample set based on the updated parameters of the encoder and the decoder of the rectification network, and obtain model parameters of the encoder and the decoder adapted to the rectification network.

[0100] In some embodiments, the training module 2082 is further configured to assign the training samples to a loss function corresponding to the encoder and decoder of the rectification network, and determine that the encoder and decoder of the rectification network correspond to the updated parameters if the loss function satisfies a corresponding convergence condition.

[0101] In some embodiments, the first information transmission module 2081 is further configured to acquire a facial image collected by a collection device, the collection device being located in an environment of use of the image processing model, perform a random enhancement process on the facial image, perform a random distortion process on the random enhancement processed facial image to obtain a distorted facial image, crop an eye image of the distorted facial image, perform a scaling process on the cropped eye image, and obtain any one of the correction target eye images in the set of correction target eye images, where the any one of the correction target eye images conforms to an input resolution supported by an encoder in the image processing model.

[0102] In some embodiments, the first information transmission module 2081 is further configured to randomly rotate the facial image based on a center thereof, randomly scale the randomly rotated facial image, and perform a random translation on the randomly scaled facial image.

[0103] In some embodiments, the first information transmission module 2081 is further configured to insert noise into the random enhancement processed face image, where the noise includes at least one of fixed noise and dynamic noise.

[0104] The functions of each software module in the image processing apparatus will be specifically described below.

[0105] The second information transmission module 2083 is configured to obtain a facial image of a target object, and the information processing module 2084 is configured to determine a corresponding correction target eye image based on the facial image of the target object, where the correction target eye image includes a left eye image and a right eye image of the target object, and the information processing module 2084 is configured to perform a correction process on the correction target eye image by an image processing model to obtain a corrected eye image, and the information processing module 2084 is configured to replace the correction target eye image in the facial image with the corrected eye image, where the image processing model is obtained by training based on a training method for an image processing model.

[0106] In some embodiments, the second information transmission module 2083 is further configured to present an eye image correction function option in a human-computer interaction interface, and in response to a trigger operation on the eye image correction function option, obtain and present a correction target eye image including the target subject, and in response to a conversion decision operation triggered based on the correction target eye image, generate and present a target eye image of a target eye image template, where the target eye image of the target eye image template is an image generated by replacing eye features in the correction target eye image with eye features in the target eye image template.

[0107] In some embodiments, the second information transmission module 2083 is further configured to present an image sharing function option for sharing the facial image in a human-computer interaction interface, and share the adjusted facial image in response to a trigger operation on the image sharing function option.

[0108] Based on the image processing model training device and image processing device implemented in an electronic device as shown in Figure 2, the embodiments of the present application further provide a computer program product or computer program, including computer instructions stored in a computer-readable storage medium. A processor of the electronic device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions by the processor, causing the electronic device to perform different embodiments and combinations of embodiments according to various alternative embodiments of the above image processing model training method and image processing device.

[0109] The above is merely an embodiment of the present application, and does not limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method for training an image processing model executed by an electronic device, comprising: Obtaining a set of correction target eye images that match the environment in which the image processing model is used; performing a style transfer process on the correction target eye image in the correction target eye image set by a style transfer network included in the image processing model to obtain a target eye image; constructing a synthetic eye using 3D modeling and obtaining a synthetic eye data pair from the synthetic eye, the synthetic eye data pair comprising a synthesized corrected eye image and a synthesized corrected eye image; obtaining a real-style synthetic eye data pair including a real-style correction target eye image and a real-style correction eye image from the synthetic eye data pair by the style transfer network included in the image processing model; obtaining a training sample matching a usage environment of the image processing model based on the correction target eye image and the target eye image, the training sample including target eye images matching different gaze positions and synthetic eye data pairs of the real style; training a correction network included in the image processing model using a training sample that matches a usage environment of the image processing model, obtaining model update parameters adapted to the correction network, and generating a trained image processing model based on the model update parameters; A method for training an image processing model, wherein an eye image in a usage environment is corrected by the trained image processing model.

2. The step of acquiring a correction target eye image set that matches the usage environment of the image processing model includes: acquiring face images collected by a collection device, the collection device being in a usage environment of the image processing model; performing an image enhancement process on the face image to obtain an image enhancement process result; determining face position coordinates in the processing result of the image enhancement, and cropping the face image including the complete eye image based on the face position coordinates; 2. The method for training an image processing model of claim 1, comprising the steps of: performing depth processing on a face image including the complete eye image by a depth processing network included in the image processing model to obtain a depth map of the face image; using the depth map as a correction target eye image matching the usage environment of the image processing model; and using a set of a plurality of the correction target eye images as the correction target eye image set.

3. The step of performing style transfer processing on the correction target eye image in the correction target eye image set by a style transfer network included in the image processing model to obtain a target eye image, determining model parameters of a reference style transfer network corresponding to the use environment of the image processing model based on the use environment of the image processing model; adjusting model parameters of a style transfer network included in the image processing model based on model parameters of the reference style transfer network; The method for training an image processing model of claim 1 , further comprising: performing a style transfer process on the correction target eye image in the correction target eye image set by the adjusted style transfer network to obtain a target eye image, wherein style features of the target eye image match a gaze position of the target object.

4. A step of training a correction network included in the image processing model using a training sample that matches the usage environment of the image processing model and obtaining a model update parameter suitable for the correction network, processing the training samples by an encoder and a decoder of a rectification network included in the image processing model to obtain initial parameters of the encoder and the decoder of the rectification network; processing the training samples based on initial parameters of the encoder and the decoder of the rectification network to obtain updated parameters of the encoder and the decoder of the rectification network; The method for training an image processing model according to claim 1, further comprising: iteratively updating the encoder and decoder parameters of the correction network by the training samples based on the encoder and decoder update parameters of the correction network, thereby obtaining encoder and decoder model update parameters adapted to the correction network.

5. The step of processing the training samples based on the initial parameters of the encoder and the decoder of the rectification network to obtain updated parameters of the encoder and the decoder of the rectification network includes: Substituting the training samples into loss functions corresponding to the encoder and decoder of the rectification network; and determining that an encoder and a decoder of the rectification network correspond to the updated parameters if the loss function satisfies a corresponding convergence condition.

6. The method for training the image processing model includes: acquiring face images collected by a collection device in a usage environment of the image processing model, and performing random enhancement processing on the face images; performing a random distortion process on the random enhancement processed face image to obtain a distorted face image; 2. The method for training an image processing model of claim 1, further comprising the steps of: cropping an eye image of the distorted face image; and performing a scaling process on the cropped eye image to obtain any one of the correction target eye images in the set of correction target eye images, wherein the any one of the correction target eye images is compatible with an input resolution supported by an encoder in the image processing model.

7. The step of performing random enhancement processing on the face image includes: randomly rotating the face image based on a center of gravity; randomly scaling the randomly rotated face image; and randomly translating the randomly scaled face images.

8. The step of performing random distortion processing on the face image that has been subjected to the random enhancement processing includes: The method for training an image processing model of claim 6 , further comprising the step of injecting noise into the randomly enhanced face image, the noise comprising at least one of a fixed noise and a dynamic noise.

9. An image processing method executed by an electronic device, comprising: acquiring a facial image of a target subject; determining corresponding correction target eye images based on the facial images of the target subject, the correction target eye images including a left eye image and a right eye image of the target subject; performing a correction process on the correction target eye image using an image processing model to obtain a corrected eye image; replacing a correction target eye image in the face image with the corrected eye image; 9. A method for image processing, wherein the image processing model is obtained by training according to the method for training an image processing model according to any one of claims 1 to 8.

10. The image processing method includes: presenting an eye image correction feature option in a human-computer interaction interface; in response to a trigger operation on the eye image correction function option, acquiring and presenting a correction target eye image including the target object; 10. The image processing method of claim 9, further comprising the step of generating and presenting a target eye image of a target eye image template in response to a transformation decision operation triggered based on the correction target eye image, the target eye image of the target eye image template being an image generated by replacing eye features in the correction target eye image with eye features in the target eye image template.

11. The image processing method includes: presenting an image sharing option in a human-computer interaction interface for sharing the facial image; The image processing method of claim 9 , further comprising the step of: sharing the adjusted face image in response to a trigger operation on the image sharing function option.

12. An image processing model training device that executes the image processing model training method described in any one of claims 1 to 8.

13. An image processing device, A second information transmission module configured to acquire a facial image of a target subject; and an information processing module configured to determine a corresponding correction target eye image based on the face image of the target subject, the correction target eye image including a left eye image and a right eye image of the target subject; The information processing module is configured to perform a correction process on the correction target eye image by an image processing model to obtain a corrected eye image; the information processing module is configured to replace a correction target eye image in the face image with the corrected eye image; An image processing device, wherein the image processing model is obtained by training based on the method for training an image processing model according to any one of claims 1 to 8.

14. A memory storing executable instructions; 12. An electronic device comprising: a processor which, when executing executable instructions stored in the memory, performs the method for training an image processing model according to any one of claims 1 to 8 or the image processing method according to any one of claims 9 to 11.

15. A computer program product that causes a computer to execute the method for training an image processing model according to any one of claims 1 to 8 or the image processing method according to any one of claims 9 to 11.

Citation Information

Patent Citations

  • Image conversion apparatus and image conversion method

    JP2019148980A

  • Solid-state imaging device and electronic device

    JP2020025261A

  • Generating gaze corrected images using bidirectionally trained network

    US20190266701A1