Method and device for relighting target object, storage medium and background replacement method

Abstract

The method and device for relighting a target object, the electronic device, the computer readable storage medium and the background replacement method relate to computer vision and artificial intelligence. The method comprises: acquiring a first image containing a target object and a second image containing a target environment map; inputting the first image and the second image into a pre-trained neural network illumination map subnetwork to obtain a predicted value of an illumination map of a third image for depicting a display state of the target object in a light environment of the target environment map; and obtaining the third image through a pre-trained neural network rendering subnetwork, and generating a relighting image of the target object according to the third image. The neural network illumination map subnetwork is trained to perform eigen-decomposition by means of a first decomposition strategy, and the neural network rendering subnetwork is trained to perform eigen-decomposition by means of the first decomposition strategy and to perform image synthesis according to an eigen-image by means of a first synthesis strategy corresponding to a common eigenvalue decomposition mode of the first decomposition strategy. The relighting according to a single picture is realized.

Description

Technical Field

[0001] The present application relates to the field of computer vision and artificial intelligence technology. Specifically, the present application relates to a method, device, electronic device, computer-readable storage medium, computer program product and a method for replacing the background of a target object. Background Art

[0002] Target object relighting involves taking an image containing a target object (e.g., a portrait) and a target lighting scenario (e.g., a specified background or light source), and applying computer vision techniques to image processing and light synthesis to produce an image of the target object under the target lighting scenario. This technology is crucial for virtual reality, augmented reality applications, and entertainment scenarios such as filmmaking. It also has widespread applications in photo processing, video conferencing, and other fields.

[0003] The methods of re-lighting in related technologies are mainly divided into two categories: one is the light synthesis technology based on the physical rendering method, and the other is the light synthesis technology based on the neural network.

[0004] Physically based rendering techniques aim to extract the physical reality of the input photo. Specifically, they precisely decompose the corresponding physical reality, separating the intrinsic characteristics of the target object from those of the environment. Based on these separated characteristics, they then perform relighting synthesis in the target lighting scenario. Because this decomposition of physical reality typically requires combining depth information from the target object, achieved through sampling from depth camera photos, synthesis results in suboptimal results when the relevant source data for the target object is incomplete, and is susceptible to depth noise.

[0005] Neural network-based lighting synthesis relies primarily on style transformation generation techniques. These techniques utilize implicit encoding to generate images that retain the original attributes of the target object in the input image, such as the geometry and identity of a person, while modifying the lighting properties of the scene. These methods are generally unsupervised and suffer from poor controllability. Current neural network-based lighting synthesis only achieves good results under specific constraints. However, it performs poorly in complex lighting environments, and the realism of the resulting synthesis needs to be improved. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to solve at least one aspect of the above technical problems to a certain extent, and to provide a solution for relighting in a complex lighting environment based on an ordinary picture of the target object.

[0007] According to a first aspect of an embodiment of the present application, a method for re-illumination of a target object is provided, the method comprising:

[0008] Acquire a first image containing a target object and a second image containing a target environment map;

[0009] Estimating the illumination map specifically includes: inputting the first image and the second image into a pre-trained neural network illumination map subnet and obtaining an estimated value of the illumination map of a third image outputted by the pre-trained neural network, wherein the third image is used to depict the display state of the target object in the illumination environment of the target environment map, the neural network illumination map subnet is trained to perform eigendecomposition on the first image using a first decomposition strategy, and generating an estimated value of the illumination map of the third image based on a result of the eigendecomposition;

[0010] Performing illumination map-guided rendering, specifically comprising inputting estimated illumination maps of the first image, the second image, and the third image into a pre-trained neural network rendering subnet and obtaining a third image as output, wherein the neural network rendering subnet is trained to perform eigendecomposition on the first image using a first decomposition strategy to obtain an albedo of the target object, and generate the third image based on the albedo of the target object and the illumination map of the third image using a preset first synthesis strategy, wherein the first synthesis strategy and the first decomposition strategy are conjugate strategies corresponding to a common eigenvalue decomposition method; and

[0011] A re-illuminated image of the target object is generated and output according to the third image output by the neural network rendering subnet.

[0012] According to a second aspect of the present application, a method for replacing a background of a target object is provided, the method comprising:

[0013] Acquire a first image containing a target object;

[0014] Obtain a second image containing the target background image;

[0015] Get the target environment image in the target background image;

[0016] Using the target object relighting method according to the first aspect of the present application, based on the first image and the target environment map, obtaining a relighting image of the target object under the target environment map;

[0017] According to the re-illuminated image and the second image, an image is obtained in which the background of the target object is replaced with the target background image.

[0018] According to a third aspect of the present application, another method for replacing the background of a target object is provided, characterized by comprising:

[0019] Obtaining a first video containing a target object;

[0020] Obtain a second image containing the target background image;

[0021] Get the target environment image in the target background image;

[0022] Acquire a first image set consisting of images containing a target object according to the first video;

[0023] Using the target object relighting method according to the first aspect of the present application, obtaining, based on the first image set and the target environment map, a relighting image set of the target object corresponding to each image in the first image set under the target environment map; and

[0024] Generating, based on the re-illuminated image set, the first video, and the second image, a third video in which a background corresponding to a target object in the first video is replaced with a target background image;

[0025] Alternatively, the background replacement method for the target object includes:

[0026] Acquire a first image containing a target object;

[0027] Get the second video containing the target background image;

[0028] Get the target environment image in the target background image;

[0029] Acquire a second image set consisting of images including a target environment map according to the second video;

[0030] Using the target object relighting method according to the first aspect of the present application, obtaining, based on the first image and the second image set, a relighting image set of the target object under the target environment map corresponding to each image in the second image set; and

[0031] A third video is generated according to the re-illuminated image set, the first image, and the second video, in which the corresponding target object is placed in the target background image of the second video.

[0032] According to a fourth aspect of an embodiment of the present application, a device for re-illuminating a target object is provided, the device comprising:

[0033] An input acquisition module is used to acquire a first image containing a target object and a second image containing a target environment map;

[0034] an irradiance map estimation module, configured to perform irradiance map estimation, specifically to input the first image and the second image into a pre-trained neural network irradiance map subnet and obtain an estimated irradiance map value of a third image outputted therefrom, wherein the third image is used to depict the display state of the target object in the illumination environment of the target environment map, the neural network irradiance map subnet is trained to perform eigendecomposition on the first image using a first decomposition strategy, and to generate an estimated irradiance map value of the third image based on the result of the eigendecomposition;

[0035] a rendering module, configured to perform illumination map-guided rendering, specifically configured to input estimated illumination maps of the first, second, and third images into a pretrained neural network rendering subnet and obtain a third image as output, wherein the neural network rendering subnet is trained to perform eigendecomposition of the first image using a first decomposition strategy to obtain an albedo of the target object, and to generate the third image from the albedo of the target object and the illumination map of the third image using a preset first synthesis strategy, wherein the first synthesis strategy and the first decomposition strategy are conjugate strategies corresponding to a common eigenvalue decomposition method; and

[0036] The image output module is used to generate a re-illuminated image of the target object based on the third image output by the neural network rendering subnet and output it.

[0037] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, the electronic device including:

[0038] A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the target object re-illumination method according to the embodiment of the first aspect of the present application, the steps of the target object background replacement method according to the second aspect of the present application, or the steps of the background replacement method according to the third aspect of the present application.

[0039] According to the sixth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the steps of the target object re-illumination method according to the embodiment of the first aspect of the present application, implements the steps of the target object background replacement method according to the second aspect of the application, or implements the steps of the background replacement method according to the third aspect of the present application.

[0040] According to the seventh aspect of the embodiments of the present application, a computer program product is provided, which, when executed by a processor, implements the steps of the target object re-illumination method according to the first aspect of the embodiment of the present application, implements the steps of the target object background replacement method according to the second aspect of the application, or implements the steps of the background replacement method according to the third aspect of the present application.

[0041] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0042] The target object can be re-illuminated based on the RGB image containing the target object, without relying on depth information and / or light field data;

[0043] Furthermore, the output of the re-illuminated composite image is based on neural network rendering rather than physical rendering, which avoids the loss of light consistency caused by error accumulation and produces a more realistic composite image.

[0044] Supports any ambient light map as input, and has strong generalization for portraits in outdoor scenes;

[0045] After the neural network training is completed, the computational complexity is small during the use phase, the algorithm is lightweight, and can be deployed on mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0047] Figure 1 Schematic diagram of the method architecture of the target object re-illumination method according to an embodiment of the present application;

[0048] Figure 2 is a schematic flow chart of a method for re-illuminating a target object according to an embodiment of the present invention;

[0049] Figure 3 A schematic flow chart of a method for generating a training data set for a neural network illumination map subnet according to an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of a training process of a neural network rendering subnet according to an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of a flow chart of a method for replacing a background of a target object according to an embodiment of the present invention;

[0052] Figure 6 A schematic flow chart of a background replacement method for another target object according to an embodiment of the present invention;

[0053] Figure 7 A schematic flow chart of a background replacement method for a target object according to another embodiment of the present invention;

[0054] Figure 8 is a schematic structural diagram of a target object re-illumination device according to an embodiment of the present invention;

[0055] Figure 9 FIG. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0057] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0058] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0059] First, to facilitate description and understanding, a brief introduction to some technical terms used in this disclosure is provided. It should be noted that the following descriptions of these terms are provided solely for the purpose of facilitating understanding of this disclosure and do not constitute restrictive definitions of their meanings. Unless otherwise specified, the meanings of technical terms commonly known in the art shall conform to generally accepted understandings in the art.

[0060] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0061] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0062] Computer vision (CV) is the science of making machines "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, tracking, and measuring objects. Further image processing is performed, allowing the computer to generate images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0063] Relighting is the process of re-lighting the target object in the first image according to specified lighting conditions, such as the lighting environment of the second image. The goal is to ensure that the resulting light composition is consistent with the target object under the specified lighting environment.

[0064] Ambient map is a way to specify lighting conditions during the relighting process. Ambient map expresses the lighting environment of the target scene in the form of an image.

[0065] Intrinsic decomposition is the process of calculating the intrinsic image of an original image containing a target, an object, and an environmental scene. Intrinsic images generally include an albedo image and a shading image. The shading image (also referred to as shading) represents the portion of the image intrinsic decomposition result that primarily characterizes the effects of ambient light. It is generally composed of various lighting and the geometric shapes of objects in the scene, reflecting the lighting information at various locations in the scene. The albedo image (also referred to as albedo) represents the portion of the image intrinsic decomposition result that primarily characterizes the characteristics of the target object itself, namely, the color and surface material properties of the target object under colorless uniform illumination.

[0066] Differentiable rendering is a rendering method for image synthesis in related technologies. It is used to perform inverse rendering based on a given 2D image to obtain the 3D scene information required to generate this image, such as one or more of 3D geometry, lighting, materials, and perspective.

[0067] 3DDFA: 3D Dense Face Alignment, a three-dimensional dense face alignment, an end-to-end single-view face 3D reconstruction model in related technologies, and a parameterized face model.

[0068] D3DFR, Accurate 3D Face Reconstruction, is a parameterized face model in related technologies.

[0069] AlbedoMM, Morphable Face Albedo Model, a deformable face albedo model, a face albedo model in related technologies.

[0070] Next, the inventive concept of the present application is briefly described. The idea of ​​eigendecomposition can be traced back to the view of the 10th-century Egyptian physicist Alhazen: "The human eye can only perceive the intensity and color of light, and other information is obtained through insight, reasoning, recognition and other methods." The concept of eigendecomposition was formally proposed in 1978. In modern digital image processing and computer vision methods, the basic idea of ​​relighting is to place the albedo corresponding to the physical reality of the object in the target environment, and perform physical rendering to synthesize the relighting image. Based on this, the common understanding of technical personnel in related fields has always been that the closer the result of eigendecomposition is to physical reality, the better the quality of the rendered relighting image. And the physical renderers in related technologies are also designed to obtain good rendering effects only based on the albedo obtained by eigendecomposition close to physical reality.

[0071] However, current technology demands very stringent raw data requirements for obtaining a near-realistic eigendecomposition. For example, it requires a depth image of the target object, or information fusion from multiple images of the target object under a given reference lighting environment. In the most common case where the raw data consists solely of a single 2D RGB image of the target object, it's difficult to calculate a near-realistic eigendecomposition, resulting in suboptimal re-illumination results.

[0072] The neural network-based relighting methods in related technologies are also based on ideas similar to physical rendering. They attempt to use neural networks to learn a method of intrinsic decomposition that is close to physical reality, or to learn the light field information of the target environment through neural networks. Based on the decomposition results, rendering is then performed by learning a method similar to that of a physical renderer. Currently, the only method that can achieve good results is relighting under specific constraints, such as when the ambient light is a light source at a given angle and the background is a simple empty space. However, due to the limitations of modeling ideas and the high requirements for training and input data, current neural network-based relighting methods are unable to relight according to complex real-world lighting environments. In addition, due to the problems in obtaining training data sets, such as the lack of relevant groundtruth about the target object, most existing neural network-based relighting methods use completely unsupervised training models, which also makes them less controllable.

[0073] The inventors of the present application have discovered that the reliance of the related art on re-illumination of the intrinsic decomposition consistent with physical reality is actually unnecessary. The human eye's perception of an image is essentially the combined result of the visual cells' response to incident light and the brain's comprehensive processing of the stress signal, rather than physical reality; the image presentation of various electronic image display devices is also a combination of color and light intensity emitted by the display device, which is also not physical reality. If the intrinsic decomposition of the original image and the synthesis of the light of the target environment can be achieved based on the decomposition of the same decomposition strategy and the corresponding synthesis strategy, the synthesized re-illumination image can also achieve a better synthesis effect.

[0074] Based on the above findings, the inventors creatively proposed a technical concept of performing re-illumination through other forms of intrinsic decomposition, and making the process of image synthesis containing the intrinsic decomposition of the original image and the target image follow the same decomposition method. At the same time, they designed a neural network model to implement this concept and specially designed training data to train the neural network model, so that the intrinsic decomposition process is freed from the dependence on the input of precise depth information and texture information of the original image of the target object, and instead uses it through neural network model recognition, and achieved good results.

[0075] It should be noted that in this application, the term "eigendecomposition" can be interpreted in a broad sense, that is, the method of performing intrinsic decomposition on the original image to obtain the intrinsic image is not unique. Any combination of albedo and shading that is decomposed from the original image according to a certain strategy and can be used to reversely synthesize the original image can be regarded as an intrinsic image, and is not necessarily limited to the decomposition method that strictly corresponds to physical reality.

[0076] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0077] The embodiment of the first aspect of the present application provides a method for re-illumination of a target object. Figure 1 、 Figure 2 , Figure 1 Schematic diagram of the method architecture of the target object re-illumination method according to an embodiment of the present application; Figure 2 4 is a flow chart of a method for re-illuminating a target object according to an embodiment of the present invention.

[0078] The target object relighting method according to the embodiment of the present application includes two aspects: pre-training of the neural network and using the trained neural network. Among them, in most application scenarios, the pre-training of the neural network can be completed in advance on the server side, or by a supplier of related services or software products. The client only performs the step of using the neural network to calculate the relighted image, and the calculation of the relighted image can communicate with the server side or be completed offline.

[0079] According to an embodiment of the present application, the target object relighting method comprises inputting a first image containing the target object and a second image containing a target environment map into a pre-trained neural network illumination map subnet to obtain an estimated illumination map of a third image used to depict the display state of the target object in the target environment map illumination environment; then, the pre-trained neural network rendering subnet obtains the third image, and a relighted image of the target object is generated based on the third image. Specifically, the method comprises the following steps S110 to S140.

[0080] In step S110 , a first image containing a target object and a second image containing a target environment map are acquired.

[0081] Among them, the original image containing the target object can be an image that only contains the area where the target object is located, or it can be an image that contains the target object and a certain range around the target object. In this case, the image that only contains the area where the target object is located can be obtained through image processing to simplify the operation. The acquisition of the area where the target object is located can be automatically implemented based on the semantic segmentation algorithm in the relevant technology, or it can be manually input as needed. For the convenience of description, in this application, the first image can be the original image, or it can be an image that only contains the area where the target object is located after preprocessing. In both cases, the re-illumination method of this application can be implemented through corresponding neural network training.

[0082] The second image can be a simple lighting environment image or a photograph of a real environment. The target environment image can be the second image itself, or a partial image of a corresponding scale captured from the second image as the target environment image, depending on the usage requirements. This embodiment does not limit the order in which the first and second images are acquired; they can be acquired in any order.

[0083] In step S120, an illumination map is estimated, specifically including: inputting the first image and the second image into a pre-trained neural network illumination map subnet and obtaining an estimated value of the illumination map of the third image outputted therefrom, wherein the third image is used to depict the display state of the target object in the lighting environment of the target environment map, and the neural network illumination map subnet is trained to perform eigendecomposition on the first image with the aid of a first decomposition strategy, and generate an estimated value of the illumination map of the third image according to the result of the eigendecomposition.

[0084] Among them, in terms of network structure, various neural network structures used for image processing in related technologies, such as various deep neural network models of commonly used CNN (convolutional neural network) structures, can be used in the neural network illumination map subnet of the embodiment of the present application as long as the calculation accuracy is sufficient. This application does not impose any restrictions on this.

[0085] In order to train the neural network illumination map subnet to perform eigendecomposition of the first image using the first decomposition strategy, one way to achieve this is to complete it through supervised training based on a dataset that satisfies the first decomposition strategy. A high-quality training dataset is particularly important for achieving better synthesis results. The construction of the training dataset will be discussed later in conjunction with Figure 3 Further explanation.

[0086] In step S130, rendering is performed under the guidance of the illumination map, specifically including inputting the estimated values ​​of the illumination maps of the first image, the second image, and the third image into a pre-trained neural network rendering subnet and obtaining the third image outputted therefrom, wherein the neural network rendering subnet is trained to perform eigendecomposition on the first image with the help of a first decomposition strategy to obtain an albedo of the target object, and to generate a third image based on the albedo of the target object and the illumination map of the third image with the help of a preset first synthesis strategy, wherein the first synthesis strategy and the first decomposition strategy are conjugate strategies corresponding to a common eigenvalue decomposition method.

[0087] Among them, in terms of network structure, various neural network structures used for image processing in related technologies, for example, various deep neural network models of commonly used CNN (convolutional neural network) structures, as long as the calculation accuracy is sufficient, can be used in the neural network rendering subnet of the embodiment of the present application, such as U-net, ResNet, etc., and the present application does not impose any restrictions on this. In order to train the neural network rendering subnet to be used for performing intrinsic decomposition of the first image with the help of a first decomposition strategy to obtain the albedo of the target object, and to generate a third image based on the albedo of the target object and the illumination map of the third image with the help of a preset first synthesis strategy, a training method combining supervised training and unsupervised training can be used, so as to improve the generalization ability of the model while ensuring the accuracy of the model. The training data set and specific training method of the neural network rendering subnet will be discussed later in conjunction with Figure 4 Further explanation.

[0088] In step S140, a re-illuminated image of the target object is generated and output according to the third image output by the neural network rendering subnet.

[0089] When the target environment map covers the entire second image, the third image corresponds to the entire range of the re-illuminated image and can be output as the re-illuminated image of the target object. When the target environment map does not cover the entire second image, the third image, generated based on the target object's area and the target environment map, can be spliced ​​with the portion of the second image outside the target environment map area to generate the re-illuminated image of the target object.

[0090] It should be understood that at the current stage of scientific and technological development, artificial neural networks work as a whole to implement the data relationship between input and output. The data in the intermediate layers of the neural network does not necessarily correspond to a clear and describable physical meaning. Furthermore, in the functional description of the neural network, in order to facilitate the reader's understanding of the overall function of the neural network, the function is sometimes described in a step-by-step manner. This step-by-step description does not mean that the data results corresponding to each step will necessarily be explicitly presented in the intermediate variables or outputs of the neural network. In other words, the execution of the "steps" by the neural network is implicit, and the order and execution method of the steps do not necessarily correspond one-to-one with the specific calculations of the neural network.

[0091] According to the target object relighting method of the embodiment of the present application, the target object can be relighted based on the RGB image containing the target object, without relying on depth information and / or light field data. And the image synthesized after relighting is output based on neural network rendering rather than physical rendering, which avoids the weakening of light consistency caused by the accumulation of depth errors, and the output synthesized image has good realism. It supports any ambient light map as input and has strong generalization for portraits in outdoor scenes. The method provided in the present application can support light synthesis under complex lighting, and for any given ambient light HDR map, a stable, consistent and realistic synthesis result can be obtained. Of course, the input of directional light can also be converted into an ambient light HDR map, so that the corresponding output can be obtained using the method provided in the present application.

[0092] Moreover, it can be seen that after the neural network training is completed, the method of the present application only needs to perform a very small number of neural network calculations during the use phase, and the calculation amount is small. The algorithm is lightweight and does not require strong server-level computing power support. It can be deployed on mobile terminals or personal computers, and can support offline applications. This makes it easier to apply the method of the present application to a variety of mobile APPs or related application software for personal computers. For example, it can be applied to image processing APPs or software to achieve light synthesis and background replacement of photos and other images; it can be applied to social or office APPs or software to replace the background of video conversations, video conferences, etc.; it can be applied to AR and VR APPs or software to insert target scenes, etc.

[0093] Although the method of the present application does not rely on physically realistic eigendecomposition, the choice of different eigendecomposition strategies will also have a certain degree of impact on the training process of the neural network, as well as the accuracy and realism of the output re-illuminated image.

[0094] In some embodiments, in the illumination map estimation step and the illumination map guided rendering step, performing eigendecomposition on the first image with the help of a first decomposition strategy may include: obtaining a three-dimensional parametric model of the main body of the target object; and performing eigendecomposition on the first image based on the three-dimensional parametric model of the main body of the target object to obtain an albedo map of the target object.

[0095] The 3D parametric model of the target object can be obtained by identifying the type of the target object or based on input information about the type of the target object. For example, for the most common portrait relighting situation, 3D parametric models of the face based on various related technologies can be obtained and used. For common types of target objects, the type of the target object can be obtained based on the first image through methods such as semantic analysis.

[0096] Performing eigendecomposition based on a 3D parametric model makes it relatively easy to obtain, which is more beneficial for generating training datasets. Furthermore, because the 3D parametric model draws on prior knowledge about the target object, it significantly improves the accuracy of the 3D reconstruction. This ultimately manifests itself in better synthesis results in the re-illuminated synthetic images.

[0097] For the neural network illumination map subnet, the first decomposition strategy can be implemented by training using a dataset corresponding to the first decomposition strategy. It should be noted that during data processing, the trained neural network illumination map subnet does not require the explicit intrinsic decomposition results presented in its intermediate variables; instead, it only estimates the illumination map of the third image in its output.

[0098] See also Figure 3 , Figure 3 Schematic diagram of a flow chart of a method for generating a training data set for a neural network illumination map subnetwork according to an embodiment of the present invention.

[0099] Specifically, each record of the first preset data set for the neural network illumination map subnet may include: a first reference image containing a target object, a second reference image containing an environment map with known parameters, a third reference image depicting the display state of the target object within the environment map with known parameters, and an illumination map of the third reference image.

[0100] The method for generating each record in the first data set may include steps S201 to S205.

[0101] In step S121, a parametric reconstruction of the main body of the target object is performed based on the first reference image to obtain a 3D parametric model of the main body of the target object. The parametric reconstruction of the main body of the target object can utilize prior knowledge about the target object to improve the accuracy of the parametric model. For example, if the target object is a face, a vehicle, or a specific identifiable object, the 3D parametric reconstruction can be performed based on existing 3D morphological information about the target object or an existing 3D model.

[0102] In step S122, eigendecomposition is performed on the first reference image based on the 3D parameterized model to obtain a rough estimate of the albedo of the first reference image. Compared to performing eigendecomposition directly on a 2D image, the 3D parameterized model incorporates knowledge of the 3D geometry of the target object. Under this constraint, eigendecomposition can more easily achieve good results.

[0103] For example, the first reference image may be subjected to eigendecomposition to obtain a corresponding depth map, material and normal vector map, and thus a rough estimate of the albedo of the first reference image may be obtained.

[0104] In step S123, the albedo of the first reference image is iteratively optimized by the differentiable renderer to obtain the corrected albedo of the first reference image.

[0105] In step S124 , offline rendering is performed based on the albedo accurate value of the first reference image and the second reference image to obtain a third reference image.

[0106] The second reference image in the training dataset can be obtained by randomly sampling from the ambient light map dataset to achieve diversity and randomness of the training data, so that the trained model has better generalization ability.

[0107] In step S125 , an illumination map of the third reference image is calculated based on the third reference image and the precise values ​​of the albedo maps of the first reference image.

[0108] The training process of the neural network illumination map subnet may include: performing residual calculation according to a preset first loss function to iteratively update the model parameters of the neural network illumination map subnet until the model converges. shading As shown below:

[0109]

[0110] It consists of two parts, one is the pixel-by-pixel L1 supervision loss, denoted as L s_pix , the other part is the L2 supervision loss of high-dimensional features, denoted as L s_per , high-dimensional features in L2 supervision loss can be provided by the pre-trained vgg network.

[0111] As one of the most common implementation scenarios, in some embodiments, the target object is a portrait, and the main body of the target object is a human face. In this case, step S121, performing parametric reconstruction of the main body of the target object based on the first reference image to obtain a three-dimensional parametric model of the main body of the target object, and step S122, performing eigendecomposition on the first reference image based on the three-dimensional parametric model to obtain a coarse estimate of the albedo of the first reference image, can be implemented as follows:

[0112] Obtain a 3D parametric model of a human face, and obtain a 3D reconstruction result of the target object based on the 3D parametric model of the human face. The 3D parametric model of the human face includes shape parameters and expression parameters of the human face, and the shape parameters and expression parameters correspond to a topologically consistent mesh model. In some embodiments, the 3D parametric model of the human face may include a 3DDFA or D3DFR model. The reconstructed parametric face model is transformed into a corresponding depth map by changing the camera coordinate system.

[0113] Based on the target object's 3D reconstruction, the first image is subjected to eigendecomposition to obtain the target object's albedo. The AlbedoMM model can be used to perform eigendecomposition on the first image to obtain the albedo of the first image. The AlbedoMM model can be used to obtain the diffuse and specular components of the face, and the roughness component can be further calculated using the residuals. This is then refined through pixel-by-pixel iterative optimization using a differentiable renderer to obtain the final eigendecomposition result.

[0114] The first training dataset thus generated can produce high-fidelity synthetic portrait image data under complex lighting conditions. Since the third reference image is generated using an Albedo image generated according to a predetermined first decomposition strategy, when using the first training dataset for reverse intrinsic decomposition training, the trained neural network will also learn how to use the first decomposition strategy to perform intrinsic decomposition to obtain a shading map. Furthermore, since the shading map reflects light and shadow information and is independent of the target object's material, the dataset entries containing the first, second, and third reference images and their illumination map information can implement supervised training of the neural network's illumination map subnetwork.

[0115] After training the neural network's illumination map subnetwork, the neural network's rendering subnetwork can be trained. This subnetwork can be trained using a combination of supervised training using the same training data source as the illumination map subnetwork and unsupervised training using wild-field data from a separate dataset. This approach allows supervised training to help learn the first synthesis strategy corresponding to the first decomposition strategy, while the unsupervised training portion can be used to provide better generalization capabilities.

[0116] The training dataset of the neural network rendering subnet is referred to as the second preset dataset, wherein each record of the second preset dataset includes: a first reference image containing a target object, a second reference image containing a target environment map, a third reference image depicting a display state of the target object within the target environment map, and an estimated value of the illumination map of the third reference image obtained by the neural network illumination map subnet based on the first reference image and the second reference image; and the second dataset includes a first category of record entries generated based on the record entries of the first preset dataset, and a second category of record entries generated based on other datasets other than the first preset data.

[0117] See also Figure 4 , Figure 4A schematic diagram illustrating the training process for a neural network rendering subnet according to an embodiment of the present invention. Regardless of whether the first or second category of data is used, training begins by first reading the first and second reference images from the dataset entry, using them as the first and second images, and then using the third reference image as the third image.

[0118] First, the first and second images are fed into the neural network's illumination map subnetwork to obtain an estimated illumination map for the third image. The first image and the estimated illumination map are then fed into the neural rendering subnetwork to obtain an estimate of the re-illuminated image. Based on the estimated re-illuminated image, an estimate of the lighting environment is calculated. The model parameters of the neural rendering subnetwork are then updated by backpropagation of the error using a second loss function until the model converges.

[0119] The second loss function may include: pixel-by-pixel L1 supervision loss and high-dimensional feature L2 supervision loss, and may also include at least one of true-false adversarial loss and illumination consistency loss.

[0120] For example, the second loss function L relit Can be set to:

[0121]

[0122] Among them, the L1 supervision loss L r_pix and the L2 supervision loss L of high-dimensional features r_per Similar to the definition in the neural network illumination map subnet training, please refer to the relevant description above. The L1 supervision loss and the L2 supervision loss of high-dimensional features are mainly used to prevent abnormal results during model training.

[0123] Authenticity confrontation loss L g_adv The probability distribution of the output image of the characterization model being judged as the real result by the authenticity discriminator. For example, the authenticity discriminator can be made to accept the output re-illuminated image estimation value and generate a label. If the label is 1, it is real, and if it is 0, it is fake. The purpose of the authenticity adversarial loss is set to make the label 1, that is, as close to the real distribution as possible. Among them, the authenticity discriminator determines whether the output image is real based on the output image or the grayscale image of the output image. Experimental results show that when the authenticity is judged based on the grayscale image, accurate judgment results can be obtained. The authenticity discriminator can be implemented by a separate GAN (generative adversarial neural network). For example, WGAN (WassersteinGAN) can be used to implement it. And the authenticity discriminator can be trained synchronously with the neural network rendering subnet.

[0124] Lighting consistency loss L lcThe loss is used to characterize the degree of consistency between the ambient lighting of the second image and the ambient lighting estimated based on the output image of the neural network rendering subnetwork. This loss ensures that the output of the lighting estimation network is consistent with the input lighting of the neural rendering subnetwork, allowing the rendering network to preserve the input lighting information.

[0125] Parameters u2 and u3 are the true and false adversarial losses L g_adv and lighting consistency loss L lc The corresponding coefficients can be selected as needed.

[0126] The construction of the second loss function of this embodiment is specially designed for the specific training data set composition and usage purpose of the rendering subnet, and can combine the characteristics of supervised and unsupervised training to achieve better training results.

[0127] Among the four components of the second loss function, L1 supervision loss L r_pix and the L2 supervision loss L of high-dimensional features r_per This can be calculated based on the estimated illumination map and the estimated re-illuminated image. The authenticity loss can be implemented by distinguishing authenticity from re-illuminated images. The illumination consistency loss can be calculated by comparing the estimated illumination environment with the second image.

[0128] Among them, for the data in the first category, the second loss function can be calculated to calculate all four items. For the data in the second category, since there is no supervised data as the true value (groundtruth) to refer to, the L1 supervised loss L r_pix and the L2 supervision loss L of high-dimensional features r_per No calculation is required and the coefficient is considered to be 0. Of course, those skilled in the art can also use other unsupervised training ideas and design other expressions of the loss function as needed.

[0129] The target object relighting method of the present application can be applied to image processing apps or software to achieve light synthesis and background replacement of photos and other images; it can be applied to social or office apps or software to replace the background of video conversations and video conferences; it can be applied to AR and VR apps or software to insert target scenes, and it can also be used in the production of movies, animation games, etc. According to the embodiments of the present invention, specific implementation methods for various scenarios of background replacement based on the target object relighting method are also provided.

[0130] For a single image, the relighting result of the target subject can be used in scenarios such as changing the background of photos and portraits, and can also be further used to replace the background of a single video frame in a video.

[0131] The embodiment of the second aspect of the present application provides a method for background replacement based on target object relighting. Figure 5 , Figure 5 FIG. 4 is a flow chart of a method for replacing a background of a target object according to an embodiment of the present invention.

[0132] The background replacement method according to an embodiment of the present application includes steps S210 to S250.

[0133] In step S210 , a first image containing a target object is acquired.

[0134] In step S220, a second image including a target background image is acquired.

[0135] There is no restriction on the order of steps S210 and S220. The first image and the second image can be provided or specified based on input during the use of the APP or software. For example, in the GUI (graphical user interface) of the image processing software or APP, a first image input button icon and / or a second image input button icon can be set. When clicked, a window pops up for the user to select or upload the first image and / or the second image.

[0136] In step S230, a target environment image within the target background image is obtained. In some embodiments, the original images owned and used by software or app users are often in the form of photographs, etc., that include a larger background area. The scale of the target object will correspond to a portion of the target background, so the specific placement position of the target object must be determined. For example, if a portrait is placed within a landscape painting, different positions of the portrait within the landscape will correspond to different lighting conditions, so the specific placement position, i.e., the target environment image within the target background image, must be determined.

[0137] Obviously, for the method of this embodiment, the order of acquiring the first image and the second image has no effect on the implementation of the method, and the two can be interchanged at will.

[0138] In step S240 , a target object relighting method is used to obtain a relighting image of the target object under the target environment map based on the first image and the target environment map.

[0139] In step S250, an image is obtained in which the background of the target object is replaced with the target background image based on the re-illuminated image and the second image.

[0140] In addition to the relevant beneficial effects of the re-illumination method of the present application, the background replacement method of this embodiment can obtain a composite picture of the target object under any given target background environment based on the first image and target object image information from any source, such as an ordinary portrait photo. In related similar applications, most of them only support software-given backgrounds (meaning certain known lighting parameters) or simple pure-color plane direct light backgrounds (a very simple special case of certain known lighting parameters). Therefore, the background replacement method according to the embodiment of the present application can provide greater background replacement freedom and can generate good composite images under any background.

[0141] In addition to background replacement for static images, the third aspect of the present application also provides a method for background replacement based on video. Figure 6 and Figure 7 , Figure 6 The method of replacing the background of a dynamic video containing a target object with a target background according to the dynamic video of the target object is shown. Figure 7 The invention shows a method for placing a target object into an environmental dynamic video according to the environmental dynamic video of the captured environmental background.

[0142] Figure 6 FIG. 1 is a flow chart of another method for replacing the background of a target object according to an embodiment of the present invention, wherein the method for replacing the background of the target object includes steps S310 to S360.

[0143] In step S310, a first video containing a target object is acquired.

[0144] In step S320, a second image including a target background image is acquired.

[0145] There is no restriction on the order of steps S310 and S320. The first video and the second image can be provided or specified based on the input when the APP or software is used. For example, in the GUI (graphical user interface) of the image processing software or APP, a first video input button icon and / or a second image input button icon can be set. When clicked, a window pops up for the user to select or upload the first video and / or the second image.

[0146] In step S330, a target environment image within the target background image is obtained. In some embodiments, the original images owned and used by software or app users are often in the form of photographs, etc., that include a larger background area. The scale of the target object will correspond to a portion of the target background, so the specific placement of the target object must be determined. For example, if a portrait is placed within a landscape painting, different positions of the portrait within the landscape will correspond to different lighting conditions, so the specific placement position, i.e., the target environment image within the target background image, must be determined.

[0147] In step S340, a first image set consisting of images of the target object is obtained based on the first video. A video is generally composed of multiple video frames. Background replacement in a video is typically performed by replacing the background of each video frame and then generating a new composite video based on the replaced video frames. Specifically, the first video is deframed, and the deframed video frames form the first image set.

[0148] In step S350, using the method according to the embodiment of the present application, based on the first image set and the target environment map, a set of re-illuminated images of the target object corresponding to each image in the first image set under the target environment map is obtained.

[0149] In step S360, a third video is generated based on the re-illuminated image set, the first video, and the second image, in which the background of the target object in the first video is replaced with the target background image.

[0150] The method of this embodiment can be implemented by a personal computer or mobile terminal, etc., or it can be implemented together with the server-side interaction. For applications such as video conferencing, AR, VR, etc., if it involves information interaction between client software and the server, the video frame splitting and background replacement of the video frame image can be placed on the server side to speed up the processing speed. For example, in a video conference, the user can use a personal computer or mobile terminal to capture a video as the target object, select a specified second image, and upload the two to the server. After the server completes the synthesis of the third video, it sends it to the corresponding designated recipient and / or feeds it back to the user.

[0151] In addition to the relevant beneficial effects of the re-illumination method of the present application, the background replacement method of this embodiment can obtain a synthetic video of the target object in any given target background environment based on the first video from any source and the image information of the target object, which can meet the needs of background enrichment in applications such as live broadcast and video conferencing, and the calculation amount of background replacement is small, so that the overall background replacement of the video is fast and effective, which can meet various real-time application needs.

[0152] Figure 7 4 is a flow chart of another method for replacing the background of a target object according to an embodiment of the present invention, wherein the method for replacing the background of the target object includes steps S410 to S460.

[0153] In step S410 , a first image containing a target object is acquired.

[0154] In step S420, a second video including the target background image is obtained.

[0155] There is no restriction on the order of steps S410 and S420. The first image and the second video can be provided or specified based on the input when the APP or software is used. For example, in the GUI (graphical user interface) of the image processing software or APP, a first image input button icon and / or a second video input button icon can be set. When clicked, a window pops up for the user to select or upload the first image and / or the second video.

[0156] In step S430, a target environment image within a target background image is obtained. One application scenario of this embodiment is to place a target object within a dynamic background video, such as for augmented reality. The background video often includes a relatively large background area, and the scale of the target object corresponds to a portion of the target background. Therefore, the specific placement position of the target object must be determined.

[0157] In step S440, a second image set consisting of images of the target environment map is obtained based on the second video. A video is generally composed of multiple video frames. Background replacement is typically performed by replacing the background of each video frame and then generating a new composite video based on the replaced video frames. Specifically, the second video is deframed, and the deframed video frames form the second image set.

[0158] In step S450, using the method according to the embodiment of the present invention, a re-illuminated image set of the target object under the target environment map corresponding to each image in the second image set is obtained based on the first image and the second image set.

[0159] In step S460, a third video is generated based on the re-illuminated image set, the first image, and the second video, with the corresponding target object being placed in the target background image of the second video.

[0160] The method of this embodiment can also replace the first image with a first image set containing the target object, where the first image set has the same number of images as the second image set, and sequentially synthesize frame images of the target object at different time points in the second video in the order of the first image set and the second image set. This achieves the placement of a dynamic target subject within a dynamic background video.

[0161] In addition to the relevant beneficial effects of the re-illumination method of the present application, the background replacement method of this embodiment can also obtain a synthetic video of the target object placed in a dynamic environment of the target background based on a first image and / or a first video containing the target object from any source and a second video as the background. It can meet the needs of background enrichment in applications such as live broadcast, video conferencing, VR, and AR, and the calculation amount of background replacement is small, so that the overall background replacement speed of the video is fast and the effect is good, which can meet various real-time application needs.

[0162] It should be noted that the background replacement method of the target object implemented based on the target object re-illumination method of the present application also has similar beneficial effects as the target object re-illumination method of the present application, which will not be repeated here.

[0163] According to an embodiment of the present application, a device for re-illuminating a target object is also provided. Figure 8 2 is a schematic structural diagram of a target object re-illumination device according to an embodiment of the present invention.

[0164] The target object re-illumination device 100 includes:

[0165] An input acquisition module 110 is configured to acquire a first image containing a target object and a second image containing a target environment map;

[0166] an irradiance map estimation module 120 for performing irradiance map estimation, specifically for inputting the first image and the second image into a pre-trained neural network irradiance map subnet and obtaining an estimated irradiance map value of a third image outputted therefrom, wherein the third image is used to depict the display state of the target object in the illumination environment of the target environment map, the neural network irradiance map subnet is trained to perform eigendecomposition on the first image using a first decomposition strategy, and to generate an estimated irradiance map value of the third image based on the result of the eigendecomposition;

[0167] a rendering module 130 for performing illumination map-guided rendering, specifically for inputting estimated illumination maps of the first, second, and third images into a pre-trained neural network rendering subnet and obtaining a third image as output, wherein the neural network rendering subnet is trained to perform eigendecomposition of the first image using a first decomposition strategy to obtain an albedo of the target object, and to generate the third image from the albedo of the target object and the illumination map of the third image using a preset first synthesis strategy, wherein the first synthesis strategy and the first decomposition strategy are conjugate strategies corresponding to a common eigenvalue decomposition method; and

[0168] The image output module 140 is configured to generate and output a re-illuminated image of the target object based on the third image output by the neural network rendering subnet.

[0169] In some embodiments, in the step of estimating the illumination map and the step of rendering guided by the illumination map, performing intrinsic decomposition on the first image using a first decomposition strategy includes:

[0170] Obtaining a three-dimensional parametric model of a body of the target object; and

[0171] The first image is subjected to eigendecomposition according to a three-dimensional parameterized model of a main body of the target object to obtain an albedo of the target object.

[0172] In some embodiments, in the step of performing irradiance map estimation, inputting the first image and the second image into a pre-trained neural network irradiance map subnet and obtaining an estimated irradiance map value of a third image outputted by the pre-trained neural network irradiance map subnet, the neural network irradiance map subnet is supervisedly trained based on a first preset data set, wherein:

[0173] Each record of the first preset data set includes: a first reference image containing a target object, a second reference image containing an environment map with known parameters, a third reference image depicting a display state of the target object within the environment map with known parameters, and an illumination map of the third reference image;

[0174] The training process of the neural network illumination map subnet specifically includes: performing residual calculation according to a preset first loss function to iteratively update the model parameters of the neural network illumination map subnet until the model converges.

[0175] Furthermore, each record in the first dataset is generated as follows:

[0176] Performing parametric reconstruction of the main body of the target object based on the first reference image to obtain a three-dimensional parametric model of the main body of the target object;

[0177] performing eigendecomposition on the first reference image according to the three-dimensional parameterized model to obtain a rough estimate of the albedo of the first reference image;

[0178] Iteratively optimizing the albedo of the first reference image through a differentiable renderer to obtain an accurate value of the albedo of the modified first reference image;

[0179] Perform offline rendering based on the altimetry precise value of the first reference image and the second reference image to obtain a third reference image; and

[0180] An illumination map of the third reference image is calculated based on the third reference image and the precise values ​​of the albedo of the first reference image.

[0181] In some embodiments, in the step of performing illumination map-guided rendering and inputting estimated illumination maps of the first image, the second image, and the third image into a pre-trained neural network rendering subnet and obtaining the third image outputted therefrom, the neural network rendering subnet is trained based on a second preset data set, wherein:

[0182] Each record of the second preset data set includes: a first reference image containing a target object, a second reference image containing a target environment map, a third reference image depicting a display state of the target object within the target environment map, and an estimated value of an irradiance map of the third reference image obtained by the irradiance map subnet of the neural network based on the first reference image and the second reference image; and the second data set includes a first category of record entries generated based on record entries of the first preset data set, and a second category of record entries generated based on data sets other than the first preset data set;

[0183] The training process of the neural network illumination map subnet specifically includes: supervised training based on the first category of record entries, and unsupervised training based on the second category of entries, and residual calculation based on the preset second loss function to iteratively update the model parameters of the neural network rendering subnet until the model converges.

[0184] In some embodiments, residual calculation is performed according to a preset second loss function to iteratively update the model parameters of the neural network rendering subnet until the model converges.

[0185] The second loss function includes: pixel-by-pixel L1 supervision loss and high-dimensional feature L2 supervision loss, and also includes at least one of the true-false adversarial loss and illumination consistency loss, wherein,

[0186] The authenticity adversarial loss is used to represent the probability distribution of the model's output image being judged as a real result by the authenticity discriminator;

[0187] The lighting consistency loss is used to characterize the degree of consistency between the ambient lighting of the second image and the ambient lighting estimated based on the output image of the neural network rendering subnetwork.

[0188] In some embodiments, the output image of the authenticity adversarial loss representation model is judged by the authenticity discriminator as a probability distribution of the true result, wherein the authenticity discriminator can determine whether the output image is true based on the grayscale image of the output image.

[0189] In some embodiments, the target object is a portrait, and the main body of the target object is a human face.

[0190] In some embodiments, in the illumination map estimation step and the illumination map guided rendering step, performing eigendecomposition on the first image using a first decomposition strategy includes:

[0191] Obtaining a three-dimensional parametric model of a human face, and obtaining a three-dimensional reconstruction result of a target object based on the three-dimensional parametric model of the human face, wherein the three-dimensional parametric model of the human face includes shape parameters and expression parameters of the human face, and the shape parameters and the expression parameters correspond to a topologically consistent mesh model;

[0192] The first image is subjected to eigendecomposition according to the three-dimensional reconstruction result of the target object to obtain an albedo of the target object.

[0193] In some embodiments, a three-dimensional parametric model of a human face is obtained, and a three-dimensional reconstruction result of a target object is obtained based on the three-dimensional parametric model of the human face, wherein the three-dimensional parametric model of the human face includes a three-dimensional dense face alignment model or a precise three-dimensional face reconstruction model; and / or

[0194] Performing eigendecomposition on the first image according to the three-dimensional reconstruction result of the target object to obtain the albedo of the target object, including: performing eigendecomposition on the first image according to the three-dimensional reconstruction result of the target object using the AlbedoMM model (deformable face albedo model) to obtain the albedo of the first image.

[0195] The target object re-illumination device according to the embodiment of the present application can achieve similar beneficial effects as the corresponding target object re-illumination method, which will not be described in detail here. Specific details can be found in the description of the relevant embodiments in the method section.

[0196] An embodiment of the present application also provides an electronic device (computer device / equipment / system), including a memory, a processor and a computer program stored on the memory, and the processor executes the above-mentioned computer program to implement the steps of the target object re-illumination method according to the embodiment of the first aspect of the present application, implement the steps of the target object background replacement method according to the second aspect of the application, or implement the steps of the background replacement method according to the third aspect of the present application.

[0197] The electronic device according to the embodiment of the present application can achieve similar beneficial effects as the corresponding target object re-illumination method, which will not be described in detail here. For specific details, please refer to the description of the relevant embodiments in the method section.

[0198] In an alternative embodiment, an electronic device is provided, such as Figure 9 As shown, Figure 9 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0199] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0200] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0201] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0202] The memory 4003 is used to store computer programs for executing the embodiments of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps of the target object relighting method according to the embodiment of the first aspect of the present application, the steps of the target object background replacement method according to the second aspect of the present application, or the steps of the background replacement method according to the third aspect of the present application.

[0203] Among them, electronic devices include but are not limited to: personal computers, mobile communication terminals, servers, wearable devices, virtual reality devices, game consoles, etc.

[0204] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of the target object re-illumination method according to the embodiment of the first aspect of the present application, the steps of the target object background replacement method according to the second aspect of the present application, or the steps of the background replacement method according to the third aspect of the present application.

[0205] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of the target object re-illumination method according to the embodiment of the first aspect of the present application, implements the steps of the target object background replacement method according to the second aspect of the application, or implements the steps of the background replacement method according to the third aspect of the present application.

[0206] The terms "first," "second," "third," "fourth," "1," "2," and the like (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0207] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0208] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A method for re-illuminating a target object, characterized in that: include: Acquire a first image containing a target object and a second image containing a target environment map; Performing an illumination map estimation specifically includes: inputting the first image and the second image into a pre-trained neural network illumination map subnet and obtaining an estimated value of the illumination map of a third image outputted therefrom, wherein the third image is used to depict the display state of the target object in the illumination environment of the target environment map, the neural network illumination map subnet is trained to perform eigendecomposition on the first image using a first decomposition strategy, and generating an estimated value of the illumination map of the third image based on a result of the eigendecomposition; Performing illumination map-guided rendering, specifically comprising inputting estimated illumination maps of the first image, the second image, and the third image into a pre-trained neural network rendering subnet and obtaining a third image outputted therefrom, wherein the neural network rendering subnet is trained to perform eigendecomposition on the first image using a first decomposition strategy to obtain an albedo of a target object, and generating the third image from the albedo of the target object and the illumination map of the third image using a preset first synthesis strategy, wherein the first synthesis strategy and the first decomposition strategy are conjugate strategies corresponding to a common eigenvalue decomposition method; and A re-illuminated image of the target object is generated and outputted based on the third image output by the neural network rendering subnet.

2. The method for re-illuminating a target object according to claim 1, wherein: In the step of estimating the illumination map and the step of rendering under the guidance of the illumination map, performing eigendecomposition on the first image by using the first decomposition strategy includes: Acquire a three-dimensional parametric model of the main body of the target object; and According to the three-dimensional parameterized model of the main body of the target object, the first image is subjected to eigendecomposition to obtain an albedo of the target object.

3. The method for re-illuminating a target object according to claim 1, wherein: In the step of estimating the irradiance map, inputting the first image and the second image into a pre-trained neural network irradiance map subnet and obtaining an estimated value of the irradiance map of the third image outputted therefrom, the neural network irradiance map subnet is supervisedly trained based on a first preset data set, wherein: Each record of the first preset data set includes: a first reference image containing a target object, a second reference image containing an environment map with known parameters, a third reference image depicting a display state of the target object in the environment map with known parameters, and an illumination map of the third reference image; The training process of the neural network illumination map subnetwork specifically includes: performing residual calculation according to a preset first loss function to iteratively update the model parameters of the neural network illumination map subnetwork until the model converges; Furthermore, each record in the first preset data set is generated in the following manner: performing parametric reconstruction of the main body of the target object according to the first reference image to obtain a three-dimensional parametric model of the main body of the target object; performing eigendecomposition on the first reference image according to the three-dimensional parameterized model to obtain a rough estimate of the albedo of the first reference image; Iteratively optimizing the albedo of the first reference image using a differentiable renderer to obtain a corrected albedo value of the first reference image; Performing offline rendering based on the altimetry precise value of the first reference image and the second reference image to obtain the third reference image; and An illumination map of the third reference image is calculated based on the third reference image and the precise values ​​of the albedo of the first reference image.

4. The method for re-illuminating a target object according to claim 1, wherein: In the step of performing illumination map-guided rendering, inputting estimated illumination map values ​​of the first image, the second image, and the third image into a pre-trained neural network rendering subnet and obtaining the third image outputted therefrom, the neural network rendering subnet is trained based on a second preset data set, wherein: Each record of the second preset data set includes: a first reference image containing a target object, a second reference image containing a target environment map, a third reference image depicting a display state of the target object within the target environment map, and an estimated value of the illumination map of the third reference image obtained by the neural network illumination map subnet based on the first reference image and the second reference image; and the second preset data set includes a first category of record entries generated based on record entries of the first preset data set, and a second category of record entries generated based on data sets other than the first preset data set; The training process of the neural network rendering subnet specifically includes: performing supervised training based on the first category of record entries, and performing unsupervised training based on the second category of record entries, and performing residual calculation based on a preset second loss function to iteratively update the model parameters of the neural network rendering subnet until the model converges.

5. The method for re-illuminating a target object according to claim 4, wherein: The residual calculation is performed according to the preset second loss function to iteratively update the model parameters of the neural network rendering subnet until the model converges. The second loss function includes: pixel-by-pixel L1 supervision loss and high-dimensional feature L2 supervision loss, and also includes at least one of true-false adversarial loss and illumination consistency loss, wherein, The probability distribution of the output image of the authenticity adversarial loss representation model being judged as a true result by the authenticity discriminator; The lighting consistency loss represents a degree of consistency between the ambient lighting of the second image and the ambient lighting estimated based on the output image of the neural network rendering subnetwork.

6. The method for re-illuminating a target object according to claim 5, wherein: The output image of the authenticity adversarial loss representation model is judged by the authenticity discriminator as the probability distribution of the true result, wherein the authenticity discriminator determines whether the output image is true based on the grayscale image of the output image.

7. The method for re-illuminating a target object according to any one of claims 1 to 6, wherein: The target object is a portrait, and the main body of the target object is a human face.

8. The method for re-illuminating a target object according to claim 7, wherein: In the step of estimating the illumination map and the step of rendering under the guidance of the illumination map, performing intrinsic decomposition on the first image by using the first decomposition strategy includes: Obtaining a three-dimensional parametric model of a human face, and obtaining a three-dimensional reconstruction result of the target object based on the three-dimensional parametric model of the human face, wherein the three-dimensional parametric model of the human face includes shape parameters and expression parameters of the human face, and the shape parameters and expression parameters correspond to a topologically consistent mesh model; Performing eigendecomposition on the first image according to a three-dimensional reconstruction result of the target object to obtain an albedo of the target object.

9. The method for re-illuminating a target object according to claim 8, wherein: Obtaining a three-dimensional parametric model of a human face, and obtaining a three-dimensional reconstruction result of the target object based on the three-dimensional parametric model of the human face, wherein the three-dimensional parametric model of the human face includes a three-dimensional dense face alignment model or a precise three-dimensional face reconstruction model; and / or The step of performing eigendecomposition on the first image according to the three-dimensional reconstruction result of the target object to obtain the albedo of the target object includes: performing eigendecomposition on the first image according to the three-dimensional reconstruction result of the target object using a deformable face albedo model to obtain the albedo of the first image.

10. A method for replacing the background of a target object, characterized in that: include: Acquire a first image containing a target object; Obtain a second image containing the target background image; Acquire a target environment image from the target background image; Using the method according to any one of claims 1 to 9, based on the first image and the target environment map, obtaining a re-illuminated image of the target object under the target environment map; An image in which the background of the target object is replaced with the target background image is obtained according to the re-illuminated image and the second image.

11. A method for replacing the background of a target object, characterized in that: include: Obtaining a first video containing a target object; Obtain a second image containing the target background image; Acquire a target environment image from the target background image; Acquire, based on the first video, a first image set consisting of images of the target object; Using the method according to any one of claims 1 to 9, based on the first image set and the target environment map, obtaining a set of re-illuminated images of the target object corresponding to each image in the first image set under the target environment map; as well as generating, based on the re-illuminated image set, the first video, and the second image, a third video in which a background corresponding to the target object in the first video is replaced with the target background image; Alternatively, the background replacement method of the target object includes: Acquire a first image containing a target object; Get the second video containing the target background image; Acquire a target environment image from the target background image; Acquire, based on the second video, a second image set consisting of images including the target environment map; Using the method according to any one of claims 1 to 9, obtaining, based on the first image and the second image set, a set of re-illuminated images of the target object in the target environment map corresponding to each image in the second image set; and A third video is generated based on the re-illuminated image set, the first image and the second video, in which the target object is placed in the target background image of the second video.

12. A device for re-illuminating a target object, characterized in that: include: An input acquisition module is used to acquire a first image containing a target object and a second image containing a target environment map; an irradiance map estimation module, configured to perform irradiance map estimation, specifically to input the first image and the second image into a pre-trained neural network irradiance map subnet and obtain an estimated irradiance map value of a third image outputted therefrom, wherein the third image is used to depict the display state of the target object in the illumination environment of the target environment map, the neural network irradiance map subnet is trained to perform eigendecomposition on the first image using a first decomposition strategy, and to generate an estimated irradiance map value of the third image based on a result of the eigendecomposition; a rendering module, configured to perform illumination map-guided rendering, specifically configured to input estimated illumination maps of the first image, the second image, and the third image into a pretrained neural network rendering subnet and obtain a third image as output, wherein the neural network rendering subnet is trained to perform eigendecomposition on the first image using the first decomposition strategy to obtain an albedo of a target object, and to generate the third image from the albedo of the target object and the illumination map of the third image using a preset first synthesis strategy, wherein the first synthesis strategy and the first decomposition strategy are conjugate strategies corresponding to a common eigenvalue decomposition method; and An image output module is used to generate and output a re-illuminated image of the target object based on the third image output by the neural network rendering subnet.

13. The device for re-illuminating a target object according to claim 12, wherein: The illumination map estimation module is used to perform illumination map estimation and the rendering module is used to perform rendering guided by the illumination map. When performing intrinsic decomposition of the first image by means of the first decomposition strategy, the module is specifically used to: Acquire a three-dimensional parametric model of the main body of the target object; and According to the three-dimensional parameterized model of the main body of the target object, the first image is subjected to eigendecomposition to obtain an albedo of the target object.

14. The device for re-illuminating a target object according to claim 12, wherein: The irradiance map estimation module is configured to perform irradiance map estimation by inputting the first image and the second image into a pre-trained neural network irradiance map subnet and obtaining an estimated irradiance map value of a third image outputted therefrom, wherein the neural network irradiance map subnet is supervisedly trained based on a first preset data set, wherein: Each record of the first preset data set includes: a first reference image containing a target object, a second reference image containing an environment map with known parameters, a third reference image depicting a display state of the target object in the environment map with known parameters, and an illumination map of the third reference image; The training process of the neural network illumination map subnetwork specifically includes: performing residual calculation according to a preset first loss function to iteratively update the model parameters of the neural network illumination map subnetwork until the model converges; Furthermore, each record in the first preset data set is generated in the following manner: performing parametric reconstruction of the main body of the target object according to the first reference image to obtain a three-dimensional parametric model of the main body of the target object; performing eigendecomposition on the first reference image according to the three-dimensional parameterized model to obtain a rough estimate of the albedo of the first reference image; Iteratively optimizing the albedo of the first reference image using a differentiable renderer to obtain a corrected albedo value of the first reference image; Performing offline rendering based on the altimetry precise value of the first reference image and the second reference image to obtain the third reference image; and An illumination map of the third reference image is calculated based on the third reference image and the precise values ​​of the albedo of the first reference image.

15. The object re-illumination device according to claim 12, characterized in that: The rendering module is configured to perform illumination map-guided rendering, inputting estimated illumination maps of the first image, the second image, and the third image into a pre-trained neural network rendering subnet and obtaining a third image outputted therefrom, wherein the neural network rendering subnet is trained based on a second preset data set, wherein: Each record of the second preset data set includes: a first reference image containing a target object, a second reference image containing a target environment map, a third reference image depicting a display state of the target object within the target environment map, and an estimated value of the illumination map of the third reference image obtained by the neural network illumination map subnet based on the first reference image and the second reference image; and the second preset data set includes a first category of record entries generated based on record entries of the first preset data set, and a second category of record entries generated based on data sets other than the first preset data set; The training process of the neural network rendering subnet specifically includes: performing supervised training based on the first category of record entries, and performing unsupervised training based on the second category of record entries, and performing residual calculation based on a preset second loss function to iteratively update the model parameters of the neural network rendering subnet until the model converges.

16. The device for re-illuminating a target object according to claim 15, wherein: The residual calculation is performed according to the preset second loss function to iteratively update the model parameters of the neural network rendering subnet until the model converges. The second loss function includes: pixel-by-pixel L1 supervision loss and high-dimensional feature L2 supervision loss, and also includes at least one of true-false adversarial loss and illumination consistency loss, wherein, The probability distribution of the output image of the authenticity adversarial loss representation model being judged as a true result by the authenticity discriminator; The lighting consistency loss represents a degree of consistency between the ambient lighting of the second image and the ambient lighting estimated based on the output image of the neural network rendering subnetwork.

17. The device for re-illuminating a target object according to claim 16, wherein: The output image of the authenticity adversarial loss representation model is judged by the authenticity discriminator as the probability distribution of the true result, wherein the authenticity discriminator determines whether the output image is true based on the grayscale image of the output image.

18. The object re-illumination device according to any one of claims 12 to 17, characterized in that: The target object is a portrait, and the main body of the target object is a human face.

19. The object re-illumination device according to claim 18, characterized in that: When the illumination map estimation module is used to perform illumination map estimation and the rendering module is used to perform illumination map guided rendering, the module is used to perform intrinsic decomposition of the first image by means of the first decomposition strategy, specifically to: Obtaining a three-dimensional parametric model of a human face, and obtaining a three-dimensional reconstruction result of the target object based on the three-dimensional parametric model of the human face, wherein the three-dimensional parametric model of the human face includes shape parameters and expression parameters of the human face, and the shape parameters and expression parameters correspond to a topologically consistent mesh model; Performing eigendecomposition on the first image according to a three-dimensional reconstruction result of the target object to obtain an albedo of the target object.

20. The object re-illumination device according to claim 19, characterized in that: Obtaining a three-dimensional parametric model of a human face, and obtaining a three-dimensional reconstruction result of the target object based on the three-dimensional parametric model of the human face, wherein the three-dimensional parametric model of the human face includes a three-dimensional dense face alignment model or a precise three-dimensional face reconstruction model; and / or The step of performing eigendecomposition on the first image according to the three-dimensional reconstruction result of the target object to obtain the albedo of the target object includes: performing eigendecomposition on the first image according to the three-dimensional reconstruction result of the target object using a deformable face albedo model to obtain the albedo of the first image.

21. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the target object re-illumination method according to any one of claims 1 to 9.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target object re-illumination method according to any one of claims 1 to 9 are implemented.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the target object re-illumination method according to any one of claims 1 to 9 are implemented.

Images