Image processing method, device, storage medium and electronic device

Abstract

The embodiments of the present application disclose an image processing method, device, storage medium and electronic device, which can be applied to the field of artificial intelligence or Internet of Vehicles. The above method includes obtaining a first face image; performing parameterized face reconstruction based on the first face image to obtain face model parameters; performing full-image reconstruction for the first face image based on the face model parameters to obtain a depth information map corresponding to the first face image; performing multi-stage intrinsic decomposition on the first face image based on the depth information map to obtain face texture parameters and ambient lighting parameters. The present application can obtain a depth information map by performing face reconstruction and full-image reconstruction on a face image, and performing multi-stage intrinsic decomposition based on the depth information map to obtain face texture parameters and ambient lighting parameters, thereby decoupling face texture information and lighting information, achieving an information stripping effect, and performing more accurate face image re-rendering based on this face texture information.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to image processing methods, devices, storage media, and electronic devices. Background Art

[0002] Related technologies can achieve image re-rendering by analyzing the original face image and outputting the corresponding face image under other environmental conditions based on the analysis results. However, with the rapid development of computer technology, higher requirements are put forward for the re-rendering accuracy of face images. However, related technologies are difficult to achieve the decoupling of the illumination information in the original face image and are difficult to support high-precision face image re-rendering, resulting in rough or distorted face image re-rendering. Summary of the invention

[0003] In order to solve at least one of the above technical problems, the embodiments of the present application provide an image processing method, an apparatus, a storage medium and an electronic device.

[0004] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0005] Obtaining a first face image;

[0006] Performing parametric face reconstruction based on the first face image to obtain face model parameters;

[0007] Performing full image reconstruction on the first face image based on the face model parameters to obtain a depth information map corresponding to the first face image;

[0008] Based on the depth information map, multi-stage intrinsic decomposition is performed on the first face image to obtain face texture parameters and ambient lighting parameters.

[0009] On the other hand, an embodiment of the present application provides an image processing device, the device comprising:

[0010] An image acquisition module, used to acquire a first face image;

[0011] A first reconstruction module, used for performing parameterized face reconstruction based on the first face image to obtain face model parameters;

[0012] A second reconstruction module, configured to perform full image reconstruction on the first face image based on the face model parameters to obtain a depth information map corresponding to the first face image;

[0013] The intrinsic decomposition module is used to perform multi-stage intrinsic decomposition on the first face image based on the depth information map to obtain face texture parameters and ambient lighting parameters.

[0014] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned image processing method.

[0015] On the other hand, an embodiment of the present application provides an electronic device, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the above-mentioned image processing method by executing the instructions stored in the memory.

[0016] On the other hand, an embodiment of the present application provides a computer program product, including a computer program or instructions, which implement the above-mentioned image processing method when executed by a processor.

[0017] The image processing method provided by the present application performs face reconstruction and full-image reconstruction on a face image to obtain a depth information map containing rich information, and performs multi-stage intrinsic decomposition based on the depth information map to obtain face texture parameters and ambient lighting parameters, thereby decoupling face texture information and lighting information to achieve the effect of information stripping. Based on this face texture information, more accurate face image re-rendering can be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a schematic diagram of a feasible implementation framework of the image processing method provided in the embodiments of this specification;

[0020] Figure 2 It is a flowchart of an image processing method provided in an embodiment of this specification;

[0021] Figure 3 is a schematic diagram of parametric face reconstruction provided by an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of full-image reconstruction provided by an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of a stage of intrinsic decomposition provided by an embodiment of the present application;

[0024] Figure 6 It is a schematic diagram of two-stage intrinsic decomposition provided by an embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of processing a picture provided by an embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of the image rendering effect of the generator provided in an embodiment of the present application;

[0027] Fig. 9 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0028] Fig.10 is a block diagram of an image processing device provided in an embodiment of the present application;

[0029] Fig.11 It is a schematic diagram of the hardware structure of a device provided in an embodiment of the present application for implementing the method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0032] In order to make the purpose, technical solution and advantages disclosed in the embodiments of the present application more clearly understood, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present application and are not used to limit the embodiments of the present application.

[0033] In the following, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "multiple" means two or more. In order to facilitate the understanding of the above-mentioned technical solutions and the technical effects produced by the embodiments of the present application, the embodiments of the present application first explain the relevant professional terms:

[0034] Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0035] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0036] Convolutional Neural Networks: CNN. It is a type of feedforward neural network with deep structure and convolutional calculation, and is one of the representative algorithms of deep learning. Convolutional neural network has the ability of representation learning and can classify input information in a translation-invariant manner according to its hierarchical structure, so it is also called translation-invariant artificial neural network.

[0037] Intelligent Traffic System (ITS), also known as Intelligent Transportation System, is an effective and comprehensive application of advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthening the connection between vehicles, roads and users, thus forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy. Or;

[0038] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), referred to as IVICS, is a development direction of Intelligent Transportation Systems (ITS). IVICS uses advanced wireless communications and new generation Internet technologies to implement all-round dynamic real-time information interaction between vehicles and roads, and conducts active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thus forming a safe, efficient and environmentally friendly road traffic system.

[0039] RGB: The RGB color model is a color standard in the industry. It obtains various colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. RGB represents the colors of the three channels of red, green, and blue. This standard includes almost all colors that can be perceived by human vision and is one of the most widely used color systems.

[0040] UV: Texture coordinates usually have two coordinate axes, U and V, so they are called UV coordinates. U represents the distribution on the horizontal coordinate and V represents the distribution on the vertical coordinate.

[0041] 3DDFA: 3D Dense Face Alignment, an end-to-end single-view face 3D reconstruction network that outputs the parameters of the face model.

[0042] ARAP: As Rigid As Possible, an algorithm that stretches the parameterized face toward the 2D key points in the face image while keeping the local shape unchanged.

[0043] PBR: Physical Based Rendering, physically based rendering method.

[0044] 3DMM: 3D Morphable Model, a three-dimensional morphological model, refers to a parametric face model.

[0045] Cook-Torrance: The Cook-Torrance illumination model considers the rough surface of an object as composed of many micro-planes. Each micro-plane can be regarded as an ideal mirror reflector. The roughness of the object surface is represented by the change in the slope of the micro-plane. The rougher the surface, the greater the change in slope, and vice versa.

[0046] BRDF: Bidirectional Reflectance Distribution Function, is used to define how the radiant illumination in a given incident direction affects the radiance in a given outgoing direction.

[0047] Spherical harmonic lighting: Spherical harmonic lighting is actually a simplification of lighting. For a point in space, the lighting received is different in all directions, that is, anisotropy. Therefore, if a point in space wants to completely restore the lighting situation, it is necessary to record the lighting in all directions on the surrounding sphere. For a large scene, the ambient light at each location may be different. If the ambient light map of each point is stored and the corresponding map is sampled each time the lighting is obtained, it is conceivable that such a method is expensive. This problem can be solved well by using spherical harmonics. The main function of spherical harmonics is to express complex spherical functions with simple coefficients.

[0048] Augmented Reality: AR. AR technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, intelligent interaction, and sensing. It simulates computer-generated text, images, three-dimensional models, music, videos and other virtual information and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0049] There are mainly two schemes for re-rendering facial images in related technologies. One is portrait re-rendering based on RGB texture map, and the other is portrait re-rendering based on physical intrinsic components. The portrait re-rendering algorithm based on texture map aligns and fuses a single or multiple color images based on a three-dimensional human body model, and uses numerical optimization methods to minimize the fusion loss energy function to obtain the texture of a single portrait. The portrait re-rendering based on physical intrinsic components first uses some prior constraints or light field data to estimate the physical intrinsic components of the portrait, and then uses light association or other realistic rendering modules to obtain a re-rendered photo of the portrait.

[0050] For example, related technologies can first obtain the two-dimensional key points of the face through the key point detection network, then obtain the initial texture map based on the geometric model of the face, and obtain the motion field through the key points and the initial texture map, thereby achieving the purpose of refining the texture. However, since there is no physical intrinsic decomposition, the texture contains lighting, and the task of lighting editing cannot be performed. The error caused by the key point detection network is inevitable, which reduces the accuracy of re-rendering.

[0051] For another example, related technologies can first reconstruct a 3D face model through a 3D reconstruction module, and then use the prior knowledge of the image to perform the intrinsic decomposition task of the face without supervision, so as to achieve re-rendering based on physical intrinsic components. However, this solution cannot decouple the ambient lighting in the image well, and the properties of the face material are relatively simple, which cannot represent the roughness of the skin, so the re-rendering lacks realism.

[0052] Moreover, the above two solutions are only limited to the face area, and cannot achieve intrinsic decomposition based on the whole image, and cannot decouple the lighting information well, so it is impossible to achieve portrait re-rendering based on the whole image. From the application perspective, they are not complete and it is difficult to ensure the accuracy of re-rendering.

[0053] In order to better decouple the illumination in the image, strip out the texture information in the image, and improve the information richness of the texture information, so as to support more accurate face image re-rendering based on the texture information, the embodiments of the present application provide an image processing method, device, storage medium and electronic device. By performing face reconstruction and full image reconstruction on the face image, a depth information map containing rich information is obtained, and multi-stage intrinsic decomposition is performed based on the depth information map to obtain face texture parameters and ambient lighting parameters, so that the face texture information and lighting information can be decoupled to achieve the effect of information stripping, and more accurate face image re-rendering can be performed based on this face texture information.

[0054] The embodiments of the present application may involve cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application. It can form a resource pool and be used on demand, which is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites, and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0055] The method provided in the embodiment of the present application may also involve a blockchain, that is, the method provided in the embodiment of the present application may be implemented based on a blockchain, or the data involved in the method provided in the embodiment of the present application may be stored based on a blockchain, or the execution subject of the method provided in the embodiment of the present application may be located in a blockchain. Blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain may include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0056] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operation management. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining the generation of public and private keys (account management), key management, and the maintenance of the correspondence between the user's real identity and the blockchain address (authority management), etc., and, under authorization, supervises and audits the transactions of certain real identities and provides risk control rule configuration (risk control audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests, and records valid requests to storage after consensus is reached. For a new business request, the basic service first adapts the interface for parsing and authentication (interface adaptation), and then encrypts the business information through the consensus algorithm (consensus management). The smart contract module is responsible for the registration and issuance of contracts, as well as contract triggering and contract execution. Developers can define the contract logic in a programming language and publish it to the blockchain (contract registration). According to the logic of the contract terms, the key or other events are called to trigger the execution and complete the contract logic. It also provides the function of contract upgrade and cancellation. The operation management module is mainly responsible for the deployment, configuration modification, contract setting, cloud adaptation and real-time status visualization output of the product during the product release process, such as alarm, network management, and node equipment health status management.

[0057] The platform product service layer provides the basic capabilities and implementation framework of typical applications. Developers can superimpose business features based on these basic capabilities to complete the blockchain implementation of business logic. The application service layer provides application services based on blockchain solutions for business participants to use.

[0058] See also Figure 1 , Figure 1 is a schematic diagram of a feasible implementation framework of the image processing method provided in the embodiments of this specification, such as Figure 1As shown, the implementation framework may include at least a terminal device 01 and a data processing server 02. Among them, the terminal device 01 may be a device located on the Internet, which may provide users with various optional Internet-based services, which are provided by the client in the embodiment of the present application. The terminal device 01 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. The client in the embodiment of the present invention may provide applications in various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, video media, smart communities, instant messaging, etc.

[0059] The data processing server 02 provides services to users by interacting with the terminal device 01. Specifically, the terminal device 01 can issue an image processing request to the data processing server 02, and in response to the image processing request, the data processing server 02 can obtain a face image. Perform face reconstruction based on the face image to obtain face model parameters. Perform full image reconstruction on the face image based on the face model parameters to obtain a depth information map corresponding to the face image. Perform multi-stage intrinsic decomposition on the face image based on the depth information map to obtain face texture parameters and ambient lighting parameters. After stripping off the ambient lighting parameters, the data processing server 02 can also re-render the face image based on the above-mentioned face texture parameters to obtain face images under various lighting conditions. Since the face texture parameters contain rich information and do not contain lighting information, the face images rendered under various lighting conditions based on the face texture parameters have higher precision.

[0060] The following introduces an image processing method of an embodiment of the present application. The information involved in the present disclosure and the facial images involved are all information authorized by the user or fully authorized by all parties.

[0061] Figure 2 A flow chart of an image processing method provided by an embodiment of the present application is shown. The embodiment of the present application provides method operation steps as described in the embodiment or flow chart, but more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system, terminal device or server product is executed, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the method shown in the embodiment or the accompanying drawings. The above method may include:

[0062] S101. Obtain a first face image.

[0063] In the embodiment of the present application, the first facial image can be a facial image obtained by photographing the face, a video frame image captured from a video containing the face, etc., and the embodiment of the present application is not limited to this.

[0064] S102. Perform parametric face reconstruction based on the first face image to obtain face model parameters.

[0065] In the embodiment of the present application, the face model obtained by face reconstruction can be used as a topologically consistent mesh model controlled by shape and expression coefficients, and controlled by expression parameters and shape parameters. The face model parameters include shape parameters, expression parameters and an initial depth map. The first face image can be subjected to end-to-end single-view face 3D reconstruction to obtain the shape information and the expression information; based on the shape information and the expression information, the initial depth map is obtained.

[0066] Specifically, the above end-to-end single-view face 3D reconstruction can be implemented based on the 3DDFA face reconstruction network, please refer to Figure 3 After performing face reconstruction on the first face image, three pieces of information, exp, id, and R / t, can be obtained. These three pieces of information represent expression information, shape information, and rotation offset information, respectively. Through these three pieces of information, a parameterized face texture model (Albedo) and an initial depth map (Depth) can be finally obtained. The parameterized face texture model (Albedo) and the initial depth map (Depth) are the reconstruction results of step S102. The reconstruction results can be obtained by F=M+A id α id +A exp α exp Representation, where F represents the reconstruction result, A id and A exp Both represent weight coefficients, ∝ id and ∝ exp They represent shape parameters and expression parameters respectively, and M represents the average face image.

[0067] Of course, in other embodiments, parametric face reconstruction may be performed based on other networks. For example, other similar networks may be used to achieve the same effect.

[0068] S103. Perform full image reconstruction on the first face image based on the face model parameters to obtain a depth information map corresponding to the first face image.

[0069] In S103, the facial model parameters obtained after reconstruction can be refined based on the whole image. Specifically, the first facial image can be subjected to key point extraction processing to obtain key point information; and the whole face image can be reconstructed based on the initial depth map and the key point information to obtain the depth information map.

[0070] The embodiment of the present application does not limit the method used for full-image face reconstruction. For example, the face model can be aligned with the above key point information by the ARAP algorithm, and the depth information of the non-face area can be filled using the Poisson diffusion equation to obtain the depth information map. The core formula of the ARAP algorithm is argmin {R,p′} ((∑ i ∑ j∈N(i) w ij ||(p′ i -p′ j )-R i (p i -p j )||)+∑ k∈K ||(p k -p′ k )||), i represents the i-th point; K represents the key point index set; N(i) represents the index set of the first-level topological neighborhood points of the i-th point; p represents the vertex coordinates before deformation; p' represents the vertex coordinates after deformation; R i represents the rotation matrix corresponding to vertex i; w ij Represents the weight, which is set to the inverse of the number of elements in the N(i) set. Figure 4 Based on the initial depth map (Depth), the depth information map (Full Depth) can be obtained by using the ARAP algorithm. The inner dotted frame represents the image conversion process of performing the ARAP operation on the initial depth map. ARAP is Figure 4 The specific operation method of Warp (bending deformation).

[0071] S104. Perform multi-stage intrinsic decomposition on the first face image based on the depth information map to obtain face texture parameters and ambient lighting parameters.

[0072] In the embodiment of the present application, two stages of intrinsic decomposition can be performed on the first face image. The first stage of intrinsic decomposition can decouple texture information and illumination information, thereby stripping off illumination information; in the second stage, more abundant texture information can be further obtained. In the embodiment of the present application, intrinsic decomposition can be understood as decomposing illumination information, shape information, and texture information of the image, wherein texture information includes roughness information, highlight information, and diffuse reflection information.

[0073] In the first stage of intrinsic decomposition, the facial texture parameters include a first diffuse reflection parameter and a first highlight parameter, the ambient lighting parameters include a first lighting parameter, and the multi-stage intrinsic decomposition of the first facial image based on the depth information map to obtain facial texture parameters and ambient lighting parameters includes: based on the depth information map and the first facial image, using a parameterized facial texture model and a spherical harmonic lighting model, and obtaining the first diffuse reflection parameter, the first highlight parameter and the first lighting parameter through a differentiable renderer.

[0074] In order to avoid the ambiguity problem of unsupervised intrinsic decomposition, the first-stage intrinsic decomposition uses a parameterized face texture model (AlbedoMM: albedo deformation model) and a spherical harmonic illumination model to roughly reconstruct the face texture. The parameterized face texture model divides the face texture model into diffuse and highlight parts, and each part is represented by a linear combination of topologically consistent bases: Where vec(a)∈R 3N Represents the diffuse reflection model of the face, N represents the number of vertices; vec(b)∈R 3N A highlight model representing a human face; Represent the average diffuse reflection and average highlight of the face respectively; P a ,P b ∈R M*3N Respectively represent the diffuse reflection base and specular reflection base of the face; α, β represent the first diffuse reflection parameter and the first specular parameter of the face respectively. In order to solve the texture parameters of the face, given the geometric shape of the face (Full Depth), the differentiable renderer is used to solve the face texture and spherical harmonic light parameters through the gradient descent algorithm, where the face texture includes the first diffuse reflection parameter and the first illumination parameter, and the spherical harmonic light parameters include the above-mentioned first illumination parameter.

[0075] Please refer to Figure 5 Based on the first face image and the depth information map, Diffuse (first diffuse reflection parameter), Specular (highlight parameter), and first illumination parameter can be obtained. This information can be used for geometric reconstruction to obtain a rough rendering result.

[0076] In the second stage of intrinsic decomposition, the facial texture parameters also include a second diffuse reflection parameter, a second highlight parameter and a roughness parameter, and the ambient lighting parameters also include a second lighting parameter. The multi-stage intrinsic decomposition of the first facial image based on the depth information map to obtain facial texture parameters and ambient lighting parameters also includes: obtaining roughness prior information and lighting hypothesis information; based on the first diffuse reflection parameter, the first highlight parameter, the roughness prior information and the lighting hypothesis information, using differentiable light association to obtain the second diffuse reflection parameter, the second highlight parameter, the roughness parameter and the second lighting parameter.

[0077] In order to solve the problem of detail loss and lack of realism in the first stage, the second stage intrinsic decomposition uses differentiable ray association to finely reconstruct the physically consistent human face texture and ambient lighting, thereby obtaining the second diffuse reflection parameter, the second highlight parameter, the roughness parameter and the second illumination parameter. The second diffuse reflection parameter, the second highlight parameter, the roughness parameter and the second illumination parameter are reconstructed through the physically based differentiable ray association technology, combined with the initial value constraints of diffuse reflection and highlight, roughness prior and illumination assumptions.

[0078] The core formula of the second stage eigendecomposition dependency is Where Render() represents a differentiable renderer function; Rst() represents a rasterization function; A∈R H×w×3 ,B∈R H×w×3 ,C∈R H×w ,D∈R H×w Respectively represent the diffuse map, specular map, roughness map and depth information map of the face; M∈R H×w Represents the mask of the face; I∈R H×w×3 Represents the first face image; l∈R 9 ×3 represents the spherical harmonic lighting parameters, represents the color average of l. The roughness prior information and illumination assumption information relied on by the second stage of intrinsic decomposition are that the roughness of the face area is as close to 1 as possible, and the distribution of the ambient light map satisfies the spherical harmonic light assumption as much as possible. Please refer to Figure 6 It can be seen that in the second stage of intrinsic decomposition, the second diffuse reflection parameter (Diffuse), the second highlight parameter (Specular), the roughness parameter (Roughness) and the second illumination parameter (irradiance map) are obtained by using differentiable light association.

[0079] Please refer to Figure 7, which shows a schematic diagram of the processing of an image in an embodiment of the present application. The ultimate goal of image processing is to achieve unsupervised intrinsic decomposition. Specifically, the depth of the portrait and the whole image can be estimated first; the intrinsic components of the portrait can be roughly estimated through the depth information and the texture prior of the portrait; finally, the intrinsic components of the whole image can be finely estimated using differentiable ray associations. Among them, the input image (first face image) can be reconstructed based on the whole image, which can obtain richer information compared to the related art of only reconstructing the face. Multi-stage intrinsic decomposition is performed on the basis of reconstruction based on the whole image, and facial texture parameters and ambient lighting parameters can be obtained, that is, facial texture information and lighting information are stripped. Among them, facial texture information includes two types of information: shape and texture. In the depth information map obtained in the previous article, the shape information has actually been stripped out.

[0080] In one embodiment, a sample triplet can be constructed based on the above method. Specifically, the sample triplet can be constructed by the following method:

[0081] S201. Obtain at least one second facial image and facial texture parameters of each of the second facial images.

[0082] The method for obtaining the facial texture parameters of the first facial image has been described above. The method for obtaining the facial texture parameters of the second facial image is the same as that of the first facial image, and will not be repeated here. The method for obtaining the second facial image is also the same as that of the first facial image, and will not be repeated here.

[0083] S202. Obtain at least one preset lighting parameter.

[0084] The embodiment of the present application aims to render a facial image corresponding to the facial texture parameters under the given preset lighting parameters based on the given preset lighting parameters and the known facial texture parameters, and the facial texture parameters have been stripped of the influence of lighting in the original image, so that the facial image generated based on the facial texture parameters under the given preset lighting parameters can be more accurate. The preset lighting parameters can be understood as given lighting parameters, and the embodiment of the present application does not limit the number of preset lighting parameters, that is, multiple preset lighting parameters can be given to obtain facial images under each given preset lighting parameter.

[0085] S203. Render a fourth facial image based on facial texture parameters of a third facial image and a third lighting parameter, wherein the third facial image is the first facial image or any one of the at least one second facial images, and the third lighting parameter is any one of the at least one preset lighting parameter.

[0086] After determining the facial texture parameters (facial texture parameters of the third facial image) and determining the preset illumination parameters (third illumination parameters), rendering can be performed to obtain an image (fourth facial image) corresponding to the facial texture parameters of the third facial image under the third illumination parameters. The present application embodiment does not limit the rendering method. For example, UnrealEngine can be used for rendering. Unreal Engine is one of the world's most well-known and widely licensed game engines with powerful image rendering capabilities. Of course, other game engines can also be used for rendering.

[0087] S204. Generate a sample triplet according to the third facial image, the third illumination parameter and the fourth facial image.

[0088] The sample triplet includes three elements, the first element can be expressed as image 1, the second element is the lighting parameter A, and the third element can be expressed as image 2, where image 2 is the rendering result of image 1 under lighting parameter A.

[0089] In one embodiment, the following operations may further be included:

[0090] S301. Obtain a sample set, wherein the sample set includes a plurality of sample triplets.

[0091] Each sample triplet in the sample set in S301 can be obtained based on steps S201-204, which will not be described in detail here.

[0092] S302. Train a generator and a discriminator in a generative adversarial network based on the sample set, wherein the generator is used to output the face image under the arbitrary illumination parameter condition according to the input face image and arbitrary illumination parameter.

[0093] Generative Adversarial Networks (GAN) is a deep learning model and one of the most promising methods for unsupervised learning on complex distributions in recent years. The model produces fairly good outputs through the mutual game learning of (at least) two modules in the framework: the generator (Generative Model) and the discriminator (Discriminative Model). In the original GAN ​​theory, it is not required that the generator and the discriminator are both neural networks, but only that they can fit the corresponding generation and discrimination functions. However, in practice, deep neural networks are generally used as generators and discriminators.

[0094] By training GAN based on the sample set, the generator can generate a third element corresponding to the first element under the condition of the second element, and the discriminator has difficulty in distinguishing whether the third element is generated by the generator or exists in reality. In other words, the generator can produce an image rendering effect that is realistic enough to be mistaken for a fake. The images rendered by the generator not only surpass traditional game rendering engines in terms of realism, but also surpass various mainstream engines in terms of performance, and can achieve real-time rendering goals on mobile terminals.

[0095] Please refer to Figure 8 , which shows the image rendering effect of the generator. Taking the face images in the FFHQ high-definition outdoor portrait dataset as input images, the image rendering results under various lighting conditions can be obtained. Obviously, this rendering result is natural and realistic. Among them, FFHQ is a high-quality open source face dataset.

[0096] Please refer to Fig. 9 , which shows a flow chart of an embodiment of the present application. The input face image can be subjected to parameterized face reconstruction and full-image depth reconstruction in sequence, and multi-stage intrinsic decomposition can be performed based on the reconstruction result. The intrinsic decomposition includes two stages, the first stage is parameterized intrinsic decomposition, and the second stage is refined intrinsic decomposition. Based on the result of the intrinsic decomposition, an image under specified illumination can be rendered, thereby realizing image re-rendering. Rendering can use the generator described above, which belongs to a neural network.

[0097] In the embodiment of the present application, the portrait and scene in the first face image, which is a real picture, can be subjected to multi-stage intrinsic decomposition to obtain an intrinsic decomposition result including diffuse reflection information, roughness information and highlight information. This intrinsic decomposition result can be re-rendered by computer graphics methods, which is crucial for virtual reality, augmented reality applications, pan-entertainment functional components and face recognition technology. This intrinsic decomposition does not rely on light field data. With a single picture as input, the 3D face in the picture can be reconstructed through 3DMM first; secondly, the full face image is constructed to reconstruct the 3D shape of the face, head, and background; then, the diffuse reflection information, highlight reflection information, roughness information and ambient lighting of the portrait and background are reconstructed using differentiable ray association and PBR rendering technology. Finally, a realistic picture can be rendered through the generator based on the result of the intrinsic decomposition to achieve the effect of re-rendering the portrait. This application not only proposes a complete full-image portrait 3D reconstruction algorithm that can finely reconstruct portraits, but also proposes an unsupervised intrinsic decomposition method to achieve high-fidelity intrinsic decomposition of portraits, and also proposes a high-quality, realistic neural network rendering module (generator) for simulating the generation of portraits under real lighting.

[0098] The texture reconstruction in the embodiment of the present application is based on the differentiable ray association technology, which far exceeds the algorithm based on the feature point motion field in terms of texture fidelity and realism, and decouples texture information and lighting information, which can ensure the editability of lighting. The BRDF modeling of the portrait material in the embodiment of the present application is based on Cook-Torrance, which can represent the roughness of the facial skin and the redness of the face caused by subsurface reflection; the modeling of ambient light is also based on the ambient light map, rather than spherical harmonic light, which can better estimate the ambient lighting in the picture. And an end-to-end algorithm for intrinsic decomposition and re-rendering of the entire portrait is realized. From an application perspective, the present invention can be directly used to realize complete AR applications such as virtual backgrounds.

[0099] The embodiments of the present application can be mainly used for the application of AR-related products such as portrait light editing and texture editing. For example, in the application of short videos, the special effects of some 3D face map pendants often require texture reconstruction and intrinsic decomposition of the face in the video, so as to complete the editing of the face material on this basis, so as to realize the above application. The embodiments of the present application provide a complete algorithm for intrinsic decomposition and re-rendering of the whole picture face, which does not rely on light field data and realizes low-cost and high-fidelity intrinsic decomposition tasks of the whole picture. Since the algorithm outputs the diffuse reflectance, high light reflectance, roughness and ambient light map of the portrait, the rendering engines in the related technology such as Maya and Unity can use the output results of the present invention to realize the re-rendering of the portrait. On the one hand, the application of traditional intrinsic decomposition is based on indoor scenes. The embodiments of the present application are free from the constraints of light field data, starting from the portrait picture of any scene, and performing multi-stage intrinsic decomposition on the portrait. On the other hand, the embodiments of the present application can meet some real-time AR applications, and complete the intrinsic decomposition and re-rendering of the face more accurately on the mobile terminal, because the algorithm of the embodiments of the present application is lightweight.

[0100] Since the 3D reconstruction task is more dependent on high-cost 3D scanners and strict acquisition environments, the embodiment of the present application is separated from the 3D acquisition process and proposes a complete full-image portrait 3D reconstruction algorithm starting from RGB images for fine-grained reconstruction of portraits. In order to achieve physically consistent and realistic intrinsic decomposition, the embodiment of the present application proposes an unsupervised multi-stage intrinsic decomposition module for high-fidelity intrinsic decomposition of portraits, which reconstructs portrait textures from coarse to fine. The embodiment of the present application also proposes a high-quality, realistic neural network rendering module for simulating the generation of portraits under real lighting, which surpasses mainstream game rendering engines in terms of performance and effects, and can achieve real-time rendering on mobile terminals.

[0101] Please refer to Fig.10 , which shows a block diagram of an image processing device in this embodiment, the device includes:

[0102] An image acquisition module 101 is used to acquire a first face image;

[0103] A first reconstruction module 102, configured to perform parametric face reconstruction based on the first face image to obtain face model parameters;

[0104] A second reconstruction module 103, configured to perform full image reconstruction on the first face image based on the face model parameters to obtain a depth information map corresponding to the first face image;

[0105] The intrinsic decomposition module 104 is used to perform multi-stage intrinsic decomposition on the first face image based on the depth information map to obtain face texture parameters and ambient lighting parameters.

[0106] In one embodiment, the face model parameters include shape parameters, expression parameters and an initial depth map, and the first reconstruction module 102 is used to perform the following operations: perform end-to-end single-view face 3D reconstruction on the first face image to obtain the shape information and the expression information; obtain the initial depth map based on the shape information and the expression information;

[0107] The second reconstruction module 103 is used to perform the following operations: perform key point extraction processing on the first face image to obtain key point information; perform full-image face reconstruction based on the initial depth map and the key point information to obtain the depth information map.

[0108] In one embodiment, the face texture parameter includes a first diffuse reflection parameter and a first highlight parameter, the ambient illumination parameter includes a first illumination parameter, and the intrinsic decomposition module 104 is configured to perform the following operations:

[0109] Based on the depth information map and the first face image, using a parameterized face texture model and a spherical harmonic illumination model, the first diffuse reflection parameter, the first highlight parameter and the first illumination parameter are obtained through a differentiable renderer analysis.

[0110] In one embodiment, the face texture parameter further includes a second diffuse reflection parameter, a second highlight parameter and a roughness parameter, the ambient illumination parameter further includes a second illumination parameter, and the intrinsic decomposition module 104 is further configured to perform the following operations:

[0111] Obtain roughness prior information and lighting assumption information;

[0112] Based on the first diffuse reflection parameter, the first highlight parameter, the roughness prior information and the illumination hypothesis information, the second diffuse reflection parameter, the second highlight parameter, the roughness parameter and the second illumination parameter are obtained by using differentiable light association.

[0113] In one embodiment, the apparatus further comprises a triple generation module, configured to perform the following operations:

[0114] Acquire at least one second facial image and facial texture parameters of each of the second facial images;

[0115] Obtaining at least one preset lighting parameter;

[0116] Rendering a fourth facial image based on a facial texture parameter of a third facial image and a third lighting parameter, wherein the third facial image is any one of the first facial image or the at least one second facial image, and the third lighting parameter is any one of the at least one preset lighting parameter;

[0117] A sample triplet is generated according to the third facial image, the third illumination parameter and the fourth facial image.

[0118] In one embodiment, the apparatus further comprises a training module, configured to perform the following operations:

[0119] Acquire a sample set, wherein the sample set includes a plurality of the sample triplets;

[0120] Training a generator and a discriminator in a generative adversarial network based on the sample set;

[0121] The generator is used to output the face image under the arbitrary illumination parameter conditions based on the input face image and arbitrary illumination parameters.

[0122] The device embodiment and method embodiment of the present application are based on the same inventive concept and will not be described in detail here.

[0123] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the above-mentioned image processing method.

[0124] The embodiment of the present application further provides a computer-readable storage medium, which can store a plurality of instructions. The instructions can be suitable for a processor to load and execute the image processing method described above in the embodiment of the present application.

[0125] In one embodiment, the above-mentioned image processing method includes:

[0126] Obtaining a first face image;

[0127] Performing parametric face reconstruction based on the first face image to obtain face model parameters;

[0128] Performing full image reconstruction on the first face image based on the face model parameters to obtain a depth information map corresponding to the first face image;

[0129] Based on the depth information map, multi-stage intrinsic decomposition is performed on the first face image to obtain face texture parameters and ambient lighting parameters.

[0130] In one embodiment, performing parameterized face reconstruction based on the first face image to obtain face model parameters includes: performing end-to-end single-view face 3D reconstruction on the first face image to obtain the shape information and the expression information; obtaining the initial depth map based on the shape information and the expression information;

[0131] The full-image reconstruction of the first face image based on the face model parameters to obtain a depth information map corresponding to the first face image includes: performing key point extraction processing on the first face image to obtain key point information; and performing full-image face reconstruction based on the initial depth map and the key point information to obtain the depth information map.

[0132] In one embodiment, the facial texture parameter includes a first diffuse reflection parameter and a first highlight parameter, the ambient lighting parameter includes a first lighting parameter, and the multi-stage intrinsic decomposition of the first facial image based on the depth information map to obtain the facial texture parameter and the ambient lighting parameter includes:

[0133] Based on the depth information map and the first face image, using a parameterized face texture model and a spherical harmonic illumination model, the first diffuse reflection parameter, the first highlight parameter and the first illumination parameter are obtained through a differentiable renderer analysis.

[0134] In one embodiment, the facial texture parameter further includes a second diffuse reflection parameter, a second highlight parameter and a roughness parameter, the ambient lighting parameter further includes a second lighting parameter, and the multi-stage intrinsic decomposition of the first facial image based on the depth information map to obtain the facial texture parameter and the ambient lighting parameter further includes:

[0135] Obtain roughness prior information and lighting assumption information;

[0136] Based on the first diffuse reflection parameter, the first highlight parameter, the roughness prior information and the illumination hypothesis information, the second diffuse reflection parameter, the second highlight parameter, the roughness parameter and the second illumination parameter are obtained by using differentiable light association.

[0137] In one embodiment, the method further comprises:

[0138] Acquire at least one second facial image and facial texture parameters of each of the second facial images;

[0139] Obtaining at least one preset lighting parameter;

[0140] Rendering a fourth facial image based on a facial texture parameter of a third facial image and a third lighting parameter, wherein the third facial image is any one of the first facial image or the at least one second facial image, and the third lighting parameter is any one of the at least one preset lighting parameter;

[0141] A sample triplet is generated according to the third facial image, the third illumination parameter and the fourth facial image.

[0142] In one embodiment, the method further comprises:

[0143] Acquire a sample set, wherein the sample set includes a plurality of the sample triplets;

[0144] Training a generator and a discriminator in a generative adversarial network based on the sample set;

[0145] The generator is used to output the face image under the arbitrary illumination parameter conditions based on the input face image and arbitrary illumination parameters.

[0146] Further, Fig.11 A schematic diagram of the hardware structure of a device for implementing the method provided in the embodiment of the present application is shown. The above device may participate in constituting or include the apparatus or system provided in the embodiment of the present application. Fig.11 As shown, the device 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig.11 The structure shown is for illustration only and does not limit the structure of the electronic device. Fig.11 More or fewer components as shown, or with Fig.11 Different configurations shown.

[0147] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0148] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the above-mentioned method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned image processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the device 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0150] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of device 10 (or mobile device).

[0151] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of the embodiments of the present application are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0152] Each embodiment in the embodiments of the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0153] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program. The above program may be stored in a computer-readable storage medium, and the above storage medium may be a read-only memory, a disk or an optical disk, etc.

[0154] The above is only a preferred embodiment of the embodiment of the present application and is not intended to limit the embodiment of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiment of the present application should be included in the protection scope of the embodiment of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Obtaining a first face image; Performing end-to-end single-view face 3D reconstruction on the first face image to obtain shape information and expression information; and obtaining an initial depth map based on the shape information and the expression information; Performing key point extraction processing on the first face image to obtain key point information; performing full-image face reconstruction based on the initial depth map and the key point information to obtain a depth information map; Based on the depth information map and the first face image, a parameterized face texture model and a spherical harmonic illumination model are used to obtain a first diffuse reflection parameter, a first highlight parameter, and a first illumination parameter through a differentiable renderer analysis; Obtain roughness prior information and lighting assumption information; Based on the first diffuse reflection parameter, the first highlight parameter, the roughness prior information and the illumination hypothesis information, obtain a second diffuse reflection parameter, a second highlight parameter, a roughness parameter and a second illumination parameter by using differentiable light association; Among them, the first diffuse reflection parameter, the first highlight parameter, the second diffuse reflection parameter, the second highlight parameter and the roughness parameter are face texture parameters, and the first illumination parameter and the second illumination parameter are ambient illumination parameters.

2. The method according to claim 1, characterized in that The method further comprises: Acquire at least one second facial image and facial texture parameters of each of the second facial images; Obtaining at least one preset lighting parameter; Rendering a fourth facial image based on a facial texture parameter of a third facial image and a third lighting parameter, wherein the third facial image is any one of the first facial image or the at least one second facial image, and the third lighting parameter is any one of the at least one preset lighting parameter; A sample triplet is generated according to the third facial image, the third illumination parameter and the fourth facial image.

3. The method according to claim 2, characterized in that The method further comprises: Acquire a sample set, wherein the sample set includes a plurality of the sample triplets; Training a generator and a discriminator in a generative adversarial network based on the sample set; The generator is used to output the face image under the arbitrary illumination parameter conditions based on the input face image and arbitrary illumination parameters.

4. An image processing device, characterized in that: The device comprises: An image acquisition module, used to acquire a first face image; A first reconstruction module is used to perform end-to-end single-view face 3D reconstruction on the first face image to obtain shape information and expression information; and obtain an initial depth map based on the shape information and the expression information; A second reconstruction module is used to perform key point extraction processing on the first face image to obtain key point information; perform full-image face reconstruction based on the initial depth map and the key point information to obtain a depth information map; An intrinsic decomposition module is used to obtain a first diffuse reflection parameter, a first highlight parameter and a first illumination parameter through a differentiable renderer based on the depth information map and the first face image, using a parameterized face texture model and a spherical harmonic illumination model; obtain roughness prior information and illumination hypothesis information; based on the first diffuse reflection parameter, the first highlight parameter, the roughness prior information and the illumination hypothesis information, obtain a second diffuse reflection parameter, a second highlight parameter, a roughness parameter and a second illumination parameter through differentiable light association; wherein the first diffuse reflection parameter, the first highlight parameter, the second diffuse reflection parameter, the second highlight parameter and the roughness parameter are face texture parameters, and the first illumination parameter and the second illumination parameter are ambient illumination parameters.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement an image processing method as claimed in any one of claims 1 to 3.

6. An electronic device, characterized in that: It includes at least one processor and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor implements an image processing method as described in any one of claims 1 to 3 by executing the instructions stored in the memory.

7. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, an image processing method according to any one of claims 1 to 3 is implemented.

Images